forked from openshift/enhancements
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
USHIFT-2188: introduce microshift API Custom certs (openshift#1541)
* USHIFT-2188: introduce microshift API Custom certs Signed-off-by: Evgeny Slutsky <[email protected]> * added kubeconfig for custom Certs section Signed-off-by: Evgeny Slutsky <[email protected]> * added more details regarding kubeconfig generation Signed-off-by: Evgeny Slutsky <[email protected]> * Update according to latest template Signed-off-by: Evgeny Slutsky <[email protected]> * tls-sni-cert-key flag apiserver clearification Signed-off-by: Evgeny Slutsky <[email protected]> * round of review and fixes Signed-off-by: Evgeny Slutsky <[email protected]> * added remark about name not included in the cert Signed-off-by: Evgeny Slutsky <[email protected]> * add Kubeconfig Generation details Signed-off-by: Evgeny Slutsky <[email protected]> * another round of reviews Signed-off-by: Evgeny Slutsky <[email protected]> * added comment about wild-card only certs Signed-off-by: Evgeny Slutsky <[email protected]> * ignore invalid certificate upon service start Signed-off-by: Evgeny Slutsky <[email protected]> * few corrections and clearifications Signed-off-by: Evgeny Slutsky <[email protected]> * added cert san forbidden ip/dns values and few nits Signed-off-by: Evgeny Slutsky <[email protected]> * added reserved missing address Signed-off-by: Evgeny Slutsky <[email protected]> --------- Signed-off-by: Evgeny Slutsky <[email protected]>
- Loading branch information
Showing
1 changed file
with
236 additions
and
0 deletions.
There are no files selected for viewing
236 changes: 236 additions & 0 deletions
236
enhancements/microshift/microshift-apiserver-custom-certs.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,236 @@ | ||
| --- | ||
| title: microshift-apiserver-custom-certs | ||
| authors: | ||
| - "@eslutsky" | ||
| reviewers: | ||
| - "@jerpeter, MicroShift contributor" | ||
| - "@pacevedom, MicroShift contributor" | ||
| - "@ggiguash, MicroShift contributor" | ||
| - "@benluddy, OpenShift API server team" | ||
| - "@stlaz, OpenShift Auth Team" | ||
|
|
||
| approvers: | ||
| - "@jerpeter" | ||
| api-approvers: | ||
| - N/A | ||
| creation-date: 2021-01-18 | ||
| last-updated: 2023-01-18 | ||
| tracking-link: | ||
| - https://issues.redhat.com/browse/USHIFT-2101 | ||
| see-also: | ||
| - microshift-apiserver-certs.md | ||
| replaces: | ||
| - N/A | ||
| superseded-by: | ||
| - N/A | ||
| --- | ||
|
|
||
| # MicroShift API server custom external certificates | ||
| ## Summary | ||
|
|
||
| This enhancement extends the Microshift apiserver Certs to allow the user to | ||
| provide additional custom certificates (e.g. signed by a 3rd party CA) for API server SSL handshake and external authentication. | ||
|
|
||
|
|
||
| > Anytime the document mentions API server, it refers to kube API server. | ||
|
|
||
| ## Motivation | ||
| Currently, users of Microshift cannot provide their own generated certificates, | ||
| Some customers have very strict requirements regarding TLS certs. There are frequently intermediate proxies which allow only TLS traffic with certs that are signed by organization owned CAs. | ||
|
|
||
| ### User Stories | ||
| * As a MicroShift administrator, I want to be able to add organization generated certificates. | ||
| - each certificate may contain: | ||
| - Single Common Name containing The API server DNS/IPAddress . | ||
| - Multiple Subject Alternative Names (SAN) containing the API server DNS/IPAddress or a wildcard entry (wildcard certificate). | ||
|
|
||
| * As a MicroShift administrator, I want to provide additional DNS names for each certificate (in the Microshift config). | ||
|
|
||
| ### Goals | ||
| * Allow MicroShift administrator to configure Microshift with externally generated certificates and | ||
| domain names. | ||
|
|
||
| ### Non-Goals | ||
| Automatic certificate rotation and integration with 3rd party cert issuing services. | ||
|
|
||
| ## Proposal | ||
| Proposal suggests to provide Administrator means adding their own Certificates using microshift configuration file (/etc/microshift/config.yaml). | ||
|
|
||
| A new `apiServer` level section will be added to the configuration called `namedCertificates`: | ||
|
|
||
| ```yaml | ||
| apiServer: | ||
| namedCertificates: | ||
| - certPath: ~/certs/api_fqdn_1.crt | ||
| keyPath: ~/certs/api_fqdn_1.key | ||
| - certPath: ~/certs/api_fqdn_2.crt | ||
| keyPath: ~/certs/api_fqdn_2.key | ||
| names: | ||
| - api_fqdn_1 | ||
| - *.apps.external.com | ||
|
|
||
| ``` | ||
| For each provided certificate, the following configuration is proposed: | ||
| 1. `names` - optional list of explicit DNS names (leading wildcards allowed). | ||
| If no names are provided, the implicit names will be extracted from the certificates. | ||
| 1. `certPath` - certificate full path. | ||
| 1. `keyPath` - certificate key full path. | ||
|
|
||
| Certificate files will be read from their configured location by the Microshift service. | ||
| External custom certificates will extend Microshift self-signed internal certificates. | ||
| Each certificate configured this way will be validated and treated according to [Failure Modes](#failure-modes). | ||
|
|
||
| ### Kubeconfig Generation | ||
| [Wildcard](https://www.rfc-editor.org/rfc/rfc6125#section-7.2) certificate is a single certificate with a wildcard character (*) in the left-most domain name , | ||
| This allows the certificate to secure multiple sub domain names (hosts) . | ||
|
|
||
| Each configured certificate can contain multiple FQDN and wildcards values in dNSName component of SubjectAlternativeName, for each unique FQDN address kubeconfig file will be generated on the filesystem. | ||
|
|
||
| Every generated `kubeconfig` for the custom certificates will omit the certificate-authority-data section, | ||
| therefore custom certificates will have to be validated against CAs in the RHEL Client trust store. | ||
|
|
||
| Each certificate gets examined in order to determine if it's purely wildcard. | ||
| A certificate is considered to be purely wildcard only if there are no FQDN Entries found. | ||
| Certificate will be considered as a wildcard only if there are no FQDN Entries found. | ||
| The FQDN Address is searched in: | ||
| - Certificate Subject Alternative Name (SAN) | ||
| - names configuration value. | ||
|
|
||
| when a Certificate is found to be a wildcard only, Microshift will not be able to find the real FQDN, node IP Address will be placed inside its kubeconfig's `server` section. | ||
| this kubeconfig should be treated as an example only and its `server` section value should be changed to the real FQDN. | ||
|
|
||
| Certain [reserved](#reserved-names-values) values that configured in `names` field can cause internal API communication issues therefore we must not allow them. | ||
|
|
||
| ### Workflow Description | ||
|
|
||
| #### Default API Certs | ||
| 1. By default, when there is no namedCertificates configuration the behaviour will remain the same. | ||
|
|
||
| #### when custom namedCertificates configured | ||
| 1. Device Administrator copies the certificates to MicroShift host, files should be accessible by root only. | ||
| 1. Device Administrator configures additonal CAs in the RHEL Client trust store on the client system. | ||
| 1. Device Administrator configures `namedCertificates` in the Microshift configuration yaml file (/etc/microshift/config.yaml). | ||
| 1. Device Administrator start/restarts MicroShift | ||
| 1. Check and validate the certificates see [Failure Modes](#failure-modes). | ||
| 1. Microshift passes the certificates to the tls-sni-cert-key as apiserver command line option preceding all the other certificates. | ||
| 1. kube-apiserver picks up the certificates and start serving them on the configured SNI. | ||
|
|
||
| ### Topology Considerations | ||
| #### Hypershift / Hosted Control Planes | ||
| N/A | ||
|
|
||
| #### Standalone Clusters | ||
| N/A | ||
|
|
||
| #### Single-node Deployments or MicroShift | ||
| Enhancement is solely intended for MicroShift. | ||
|
|
||
| ### Implementation Details/Notes/Constraints | ||
| The certs will be prepended to the `[]configv1.NamedCertificate` list before the api server is started. (it will be added to `-tls-sni-cert-key` flag) | ||
|
|
||
| This certificate paths configuration and names will be prepended into the kube-apiserver `tls-sni-cert-key` command line flag. | ||
| When the same SNI appears in the SAN part of the provided certs, this certificate will take precedence over the `default` external-signer. [ref](https://github.com/kubernetes/kubernetes/blob/98358b8ce11b0c1878ae7aa1482668cb7a0b0e23/staging/src/k8s.io/apiserver/pkg/server/dynamiccertificates/named_certificates.go#L38) | ||
|
|
||
|
|
||
| ### API Extensions | ||
| N/A | ||
|
|
||
| ### Risks and Mitigations | ||
| * User provide certificate and it expired after some time. | ||
| kube-api server will continue serving with an expired cert - similiar approach is taken by OpenShift. | ||
| > users can mitigate this by using --insecure-skip-tls-verify client mode | ||
| * User provided expired Certificate and Microshift service started/restarted | ||
| Microshift will start with a warning in the logs, kube-apiserver will continue serve with an expired cert - similiar approach is taken by OpenShift. | ||
| > users can mitigate this by using --insecure-skip-tls-verify client mode | ||
| * Users might configure certs with the wrong names that doesnt match the certificate SAN,but openShift allows it so we will too. | ||
| ### Drawbacks | ||
| * External certificate wont be automaticly renewed, so it requires manual Certificate rotation. | ||
| similiar approach is taken by OpenShift. | ||
|
|
||
|
|
||
| ## Design Details | ||
| N/A | ||
|
|
||
| ## Open Questions [optional] | ||
| N/A | ||
|
|
||
| ## Test Plan | ||
| add e2e test that will: | ||
| - generate and sign certificates with custom ca. | ||
| - change the default Microshift configuration to use the newly generated certs. | ||
| - make sure system is functional using generated external kubeconfig. | ||
| - intentionally configure invalid value in `names` (ie: 127.0.0.1,localhost) , Microshift should ignore this configuration and skip this certificate. | ||
|
|
||
| ## Graduation Criteria | ||
| ### Dev Preview -> Tech Preview | ||
| - Gather feedback from early adopters on possible issues. | ||
|
|
||
| ### Tech Preview -> GA | ||
| - Extensive testing with various certificates variations. | ||
| - User facing documentation created in [openshift-docs](https://github.com/openshift/openshift-docs/) | ||
|
|
||
| ### Removing a deprecated feature | ||
| N/A | ||
|
|
||
| ## Upgrade / Downgrade Strategy | ||
| N/A | ||
|
|
||
| ## Version Skew Strategy | ||
| N/A | ||
|
|
||
| ## Operational Aspects of API Extensions | ||
| N/A | ||
|
|
||
| ### Failure Modes | ||
| The provided certs value will be validated before is it passed to the api-server flag | ||
|
|
||
| Those conditions will cause Microshift to ignore certificates and log the error: | ||
| 1. certificates files do not exist in the disk or are not readable by microshift process. | ||
| 1. certificate is not parseable by [x509.ParseCertificate](https://pkg.go.dev/crypto/x509#ParseCertificate). | ||
| 1. certificate override the internal certificates IPAddress/DNSNames in the SubjectAlternativeNames . see [reserved](#reserved-names-values) names below. | ||
|
|
||
| This condition will cause Microshift to accept the certificate with a warning in the logs. | ||
| 1. certificate is expired. | ||
|
|
||
|
|
||
| ### Reserved Names values | ||
| | Address | Type | Comment | | ||
| |:--------------------------|:---------|:----------------| | ||
| | localhost |DNS | | | ||
| | 127.0.0.1 |IPAddress | | | ||
| | 10.42.0.0 |IPAddress | Cluster Network | | ||
| | 10.43.0.0/16,10.44.0.0/16 |IPAddress | Service Network | | ||
| | 169.254.169.2/29 |IPAddress | br-ex Network | | ||
| | kubernetes.default.svc |DNS | | | ||
| | openshift.default.svc |DNS | | | ||
| | svc.cluster.local |DNS | | | ||
|
|
||
|
|
||
| ## Support Procedures | ||
| - Make sure that certificate is served by the kube-apiserver, verify that the certificate path is appended to the `--tls-sni-cert-key` FLAG | ||
|
|
||
| ```shell | ||
| > journalctl -u microshift -b0 | grep tls-sni-cert-key | ||
|
|
||
| Jan 24 14:53:00 localhost.localdomain microshift[45313]: kube-apiserver I0124 14:53:00.649099 45313 flags.go:64] FLAG: --tls-sni-cert-key="[/home/eslutsky/dev/certs/server.crt,/home/eslutsky/dev/certs/server.key;/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.key;/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.key;/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.key | ||
| ``` | ||
| - Make sure the kube-apiserver is serving the correct certificate. | ||
| ```shell | ||
| $ openssl s_client -connect <SNI_ADDRESS>:6443 -showcerts | openssl x509 -text -noout -in - | grep -C 1 "Alternative\|CN" | ||
| ``` | ||
| ## Implementation History | ||
| Prototype implentation details https://github.com/openshift/microshift/pull/3130 | ||
| ## Alternatives | ||
| - Use [cert-manager](https://cert-manager.io/docs/) for managing Microshift external custom certs | ||
| which allow MicroShift administrators to rotate certificates automatically (e.g. via ACME). | ||
| cert-manager is not supported on Microshift because it requires a lot of internal API changes. | ||
| ## Infrastructure Needed [optional] | ||
| N/A |