From 6f71e94580719805a301b9712c5227318c342f76 Mon Sep 17 00:00:00 2001 From: Mikael Kundert Date: Thu, 9 Mar 2023 16:16:17 +0200 Subject: [PATCH 01/20] Add possible cause of refued connection via SSH --- docs/troubleshooting.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 21fca58..80b8a26 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -54,6 +54,8 @@ Possible causes: - There is no `shell` access in frontend chart by default, You need to enable it (`shell.enabled: true`) and use customized base images from https://hub.docker.com/r/wunderio/silta-node +- The environment has been downscaled to standby and has not been yet re-launched + ## Q: Timeout without error when connecting via SSH Error: From c419f912f051762fc910ab887d47358af021a685 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C4=81nis=20Bebr=C4=ABtis?= Date: Thu, 10 Aug 2023 10:47:27 +0300 Subject: [PATCH 02/20] Removes some wunder specific information --- docs/creating_a_new_project.md | 2 +- docs/migrating_existing_project.md | 4 ---- 2 files changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/creating_a_new_project.md b/docs/creating_a_new_project.md index 891ea3b..155475a 100644 --- a/docs/creating_a_new_project.md +++ b/docs/creating_a_new_project.md @@ -16,5 +16,5 @@ title: Creating a new project ## Frontend project guidelines - Make a copy of Wunder's [frontend-project](https://github.com/wunderio/frontend-project), and push it as a new repository within the wunderio Github organisation. -- Log in to CircleCI with your Github credentials, select "wunderio" and [enable your project](https://circleci.com/add-projects/gh/wunderio). +- Log in to CircleCI with your Github credentials and [enable your project](https://circleci.com/docs/getting-started/). - Watch your project build, the CircleCI output has a link to your deployed environment. diff --git a/docs/migrating_existing_project.md b/docs/migrating_existing_project.md index 4d7d703..c56156e 100644 --- a/docs/migrating_existing_project.md +++ b/docs/migrating_existing_project.md @@ -4,10 +4,6 @@ title: Migrating an existing Drupal project --- ## General tips -- We automate as much as possible, but many projects have project-specific differences. -- When in doubt, ask for advice. -- Ask any questions in our #dev-silta slack channel. - ## Step by step instructions 1. Make sure you have a clean, up-to-date checkout of your repository. From 26ec88bf51b89adaff8b87f2a2ceb8ab098fc563 Mon Sep 17 00:00:00 2001 From: Rasmus Werling Date: Thu, 17 Aug 2023 10:31:01 +0300 Subject: [PATCH 03/20] Removed empty heading --- docs/migrating_existing_project.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/migrating_existing_project.md b/docs/migrating_existing_project.md index c56156e..6791d03 100644 --- a/docs/migrating_existing_project.md +++ b/docs/migrating_existing_project.md @@ -2,8 +2,6 @@ id: migrating-an-existing-drupal-project title: Migrating an existing Drupal project --- -## General tips - ## Step by step instructions 1. Make sure you have a clean, up-to-date checkout of your repository. From e1cc745607441038bd8878d43f9dd85d348d5ade Mon Sep 17 00:00:00 2001 From: Rasmus Werling Date: Thu, 17 Aug 2023 10:35:01 +0300 Subject: [PATCH 04/20] Update troubleshooting.md --- docs/troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 80b8a26..4ba0884 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -54,7 +54,7 @@ Possible causes: - There is no `shell` access in frontend chart by default, You need to enable it (`shell.enabled: true`) and use customized base images from https://hub.docker.com/r/wunderio/silta-node -- The environment has been downscaled to standby and has not been yet re-launched +- The environment has been downscaled to standby and has not been yet re-launched. Visit the environment URL and press the button to trigger upscaling. ## Q: Timeout without error when connecting via SSH From 21fbff62fe42c8b3b1dd52c8320496a4abc640c5 Mon Sep 17 00:00:00 2001 From: Artis Krumins Date: Thu, 17 Aug 2023 14:23:24 +0300 Subject: [PATCH 05/20] SSL certificates documentation --- docs/silta-examples.md | 41 +++++++++--------------- docs/ssl_certificates.md | 68 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+), 26 deletions(-) create mode 100644 docs/ssl_certificates.md diff --git a/docs/silta-examples.md b/docs/silta-examples.md index add5ae0..6790f23 100644 --- a/docs/silta-examples.md +++ b/docs/silta-examples.md @@ -465,15 +465,12 @@ If the `smtp` is configured and enabled, but it does not appear to send anything ## Domain names and SSL certificates -All environments are given a hostname by default. It is possible to attach a custom domain name to environment by configuring `exposeDomains` configuration parameter. All hostnames attached to environment are printed in release notes. +All environments are given a hostname by default. It is possible to attach a custom domain name to environment by configuring `exposeDomains` configuration parameter. All hostnames attached to environment are printed in release notes. +You can also use `letsencrypt-staging` issuer to avoid hitting `letsencrypt` [rate limits](https://letsencrypt.org/docs/rate-limits/). -Note: You can also use `letsencrypt-staging` issuer to avoid hitting `letsencrypt` [rate limits](https://letsencrypt.org/docs/rate-limits/). +!NB Deploy `exposeDomains` entries only when DNS entries are changed or are soon to be changed. Otherwise, Letsencrypt validation might eventually get stuck due to retries. -Note 2: For custom certificates it's advised to add CA root certificate to `exposeDomains[].ssl.crt` value. Having it under `exposeDomains[].ssl.ca` is not enough. - -Note 3: Deploy `exposeDomains` entries only when DNS entries are changed or are soon to be changed. Otherwise, Letsencrypt validation might eventually get stuck due to retries. - -Note 4: Put `exposeDomains` in a dedicated configuration yaml file, so only one environment (branch) would be assigned this hostname. Having multiple environments with the same domain will act as a round robin load balancer for all environments and unexpected responses might be returned. +!NB Put `exposeDomains` in a dedicated configuration yaml file, so only one environment (branch) would be assigned this hostname. Having multiple environments with the same domain will act as a round robin load balancer for all environments and unexpected responses might be returned. _Drupal chart and Frontend chart_: @@ -491,43 +488,35 @@ exposeDomains: enabled: true issuer: custom # Encrypt key and certificate. See: docs/encrypting_sensitive_configuration.md - ca: | - -----BEGIN CERTIFICATE----- - < CA CHAIN ROOT > - -----END CERTIFICATE----- - -----BEGIN CERTIFICATE----- - < CA CHAIN RCA > - -----END CERTIFICATE----- - -----BEGIN CERTIFICATE----- - < CA CERTIFICATE > - -----END CERTIFICATE----- key: | -----BEGIN RSA PRIVATE KEY----- -----END RSA PRIVATE KEY----- - crt: | -----BEGIN CERTIFICATE----- - < CERTIFICATE > + < DOMAIN CERTIFICATE > -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- - < CA CHAIN ROOT > + < INTERMEDIATE CERTIFICATE > -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- - < CA CHAIN RCA > - -----END CERTIFICATE----- - -----BEGIN CERTIFICATE----- - < CA CERTIFICATE > + < ROOT CA CERTIFICATE > -----END CERTIFICATE----- ``` +`key` value is certificates private key. +`crt` value is full chain of certificate. +`ca` value is not required anymore for exposed domains. +[See more information on how to convert and prepare SSL certificate for exposed domains](ssl_certificates.md) If you have same SSL certificate for multiple domains You can reuse `ssl` block. ```yaml exposeDomains: - example-customcert: &shared-ssl + example-domain1: &shared-ssl ssl: [....] - example-anothercert: + example-domain2: + <<: *shared-ssl + example-domain3: <<: *shared-ssl ``` diff --git a/docs/ssl_certificates.md b/docs/ssl_certificates.md new file mode 100644 index 0000000..c6be8fd --- /dev/null +++ b/docs/ssl_certificates.md @@ -0,0 +1,68 @@ +## Basics + +Full chain consists of 3 parts. +`End-Entity (Server) Certificate:` This is your server's SSL/TLS certificate, also known as the end-entity certificate. It is the certificate that identifies your server's domain. +`Intermediate Certificates:` These are the certificates of intermediate Certificate Authorities (CAs) that form the chain between your end-entity certificate and the root CA certificate. Intermediate certificates help build the trust chain between your certificate and a root CA. They are necessary because root CA certificates are typically not distributed widely due to security reasons. +`Root CA Certificate:` This is the certificate of the root Certificate Authority. This certificate is the ultimate trust anchor in the chain. The root CA certificate establishes trust in the entire chain. + +You can have multiple Intermediate Certificates in chain. +```yaml +exposeDomains: + example-customcert: + hostname: ssl-custom.example.com + ssl: + enabled: true + issuer: custom + key: | + -----BEGIN RSA PRIVATE KEY----- + + -----END RSA PRIVATE KEY----- + crt: | + -----BEGIN CERTIFICATE----- + < DOMAIN CERTIFICATE > + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + < INTERMEDIATE CERTIFICATE 1 > + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + < INTERMEDIATE CERTIFICATE 2 > + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + < INTERMEDIATE CERTIFICATE N > + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + < ROOT CA CERTIFICATE > + -----END CERTIFICATE----- +``` + +## PFX to PEM +Extraction (legacy flag is required if older version of PKCS#12 was used to create PFX file): +`openssl pkcs12 -legacy -in custom_cert.pfx -nocerts -nodes | sed -ne '/-BEGIN PRIVATE KEY-/,/-END PRIVATE KEY-/p' > private.key` +`openssl pkcs12 -legacy -in custom_cert.pfx -cacerts -nokeys -chain | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ca.crt` +`openssl pkcs12 -legacy -in custom_cert.pfx -clcerts -nokeys | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > domain.crt` + +Creating full chain: +`cat domain.crt ca.crt > fullchain.crt` + +You can also use [these scripts](https://github.com/wunderio/internal-ops-utils/blob/master/silta-scripts/pfx-readme.md). + +## SSL certificate verification + +You can verify full chain part: +`openssl verify -CAfile fullchain.crt domain.crt` +And then matching with private key +`openssl x509 -noout -modulus -in fullchain.crt | openssl md5` +`openssl rsa -noout -modulus -in private.key | openssl md5` +Output values should match. + +Testing certificate on live server can be done only on different cluster/environment. In this case `/etc/hosts` +should be changed accordingly. +!NB Do not try to test it on Production cluster/environment where production hostname is in use already. + + +## Tips + +PEM strings can be encoded in different formats. Both cases are valid +`-----BEGIN RSA PRIVATE KEY-----` +`-----BEGIN PRIVATE KEY-----` +`openssl` will take care of correct decoding. [List of all supported formats](https://git.openssl.org/?p=openssl.git;a=blob;f=include/openssl/pem.h;hb=HEAD#l35). \ No newline at end of file From 38e4689ba4f116a1887708b9ba795739d9f8d7a3 Mon Sep 17 00:00:00 2001 From: Artis Krumins Date: Thu, 17 Aug 2023 15:05:44 +0300 Subject: [PATCH 06/20] PFX script location moved to public repo --- docs/ssl_certificates.md | 2 +- scripts/pfx-ready.sh | 47 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 1 deletion(-) create mode 100755 scripts/pfx-ready.sh diff --git a/docs/ssl_certificates.md b/docs/ssl_certificates.md index c6be8fd..29646ed 100644 --- a/docs/ssl_certificates.md +++ b/docs/ssl_certificates.md @@ -44,7 +44,7 @@ Extraction (legacy flag is required if older version of PKCS#12 was used to crea Creating full chain: `cat domain.crt ca.crt > fullchain.crt` -You can also use [these scripts](https://github.com/wunderio/internal-ops-utils/blob/master/silta-scripts/pfx-readme.md). +You can also use [this script](../scripts/pfx-ready.sh). ## SSL certificate verification diff --git a/scripts/pfx-ready.sh b/scripts/pfx-ready.sh new file mode 100755 index 0000000..d3705e8 --- /dev/null +++ b/scripts/pfx-ready.sh @@ -0,0 +1,47 @@ +#!/bin/bash + +if [ "$#" -ne 2 ]; then + echo -e "Usage: $0 filename prefix" + exit 1 +fi + +filename="$1" +prefix="$2" + +# Check if the specified file exists +if [ ! -f "$filename" ]; then + echo "Error: File '$filename' does not exist." + exit 1 +fi + +# Check if the corresponding .pass file exists +passfile="${prefix}.pass" +if [ ! -f "$passfile" ]; then + echo "Error: Pass file '$passfile' does not exists. Create a password file [prefix].pass and put there password for decoding PFX file (f.ex. mysite.pass)" + exit 1 +fi + +openssl pkcs12 -legacy -in "$filename" -nocerts -nodes -password "file:$prefix.pass" | sed -ne '/-BEGIN PRIVATE KEY-/,/-END PRIVATE KEY-/p' > "${prefix}_private.key" +openssl pkcs12 -legacy -in "$filename" -cacerts -nokeys -password "file:$prefix.pass" | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > "${prefix}_ca.crt" +openssl pkcs12 -legacy -in "$filename" -clcerts -nokeys -password "file:$prefix.pass" | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > "${prefix}_root.crt" + +openssl x509 -in "${prefix}_root.crt" -noout -startdate -enddate + +sed -ne 's/^\( *\)[Ss]ubject[=:] */ \1/p;/X509v3 Subj.*Alt.*Name/{ + N;s/^.*\n//;:a;s/^\( *\)\(.*\), /\1\2\n\1/;ta;p;q; }' < <( + openssl x509 -in "${prefix}_root.crt" -noout -subject -ext subjectAltName) + +echo -e "\n--------------------------\n"; +echo -e "Put this under ssl:\n" +echo -e " key: |" +key_content=$(<"${prefix}_private.key") +key_indented_content=$(echo "$key_content" | sed "s/^/ /") +echo "$key_indented_content" + +echo -e " crt: |" +root_content=$(<"${prefix}_root.crt") +root_indented_content=$(echo "$root_content" | sed "s/^/ /") +echo "$root_indented_content" +ca_content=$(<"${prefix}_ca.crt") +ca_indented_content=$(echo "$ca_content" | sed "s/^/ /") +echo "$ca_indented_content" From 415abdc4de849418dc2327ad69d70794d2fa3f98 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Artis=20Kr=C5=ABmi=C5=86=C5=A1?= Date: Fri, 25 Aug 2023 13:19:54 +0300 Subject: [PATCH 07/20] Update ssl_certificates.md Adding steps to verify SSL certificate on live server --- docs/ssl_certificates.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/docs/ssl_certificates.md b/docs/ssl_certificates.md index 29646ed..b6972a2 100644 --- a/docs/ssl_certificates.md +++ b/docs/ssl_certificates.md @@ -55,9 +55,19 @@ And then matching with private key `openssl rsa -noout -modulus -in private.key | openssl md5` Output values should match. -Testing certificate on live server can be done only on different cluster/environment. In this case `/etc/hosts` -should be changed accordingly. -!NB Do not try to test it on Production cluster/environment where production hostname is in use already. +Testing certificate on live server can be done only on different cluster/environment. +*!NB Do not try to test it on Production cluster/environment where production hostname is in use already.* +#### Steps to test SSL certificate on Development cluster + * Make a new Git branch + * Add SSL certificates domain to Exposed domains in `stila.yml` + * Create secrets file, put relevant structure and encrypt it with cluster's secret key + * Modify `.circleci/config.yml` to decrypt secret and use it in `silta_config` part + * Push branch to trigger deployment + * Verify SSL certificate with `openssl s_client -connect [IP]:443 -servername [hostname]`. Expected result + `SSL handshake has read 7583 bytes and written 408 bytes Verification: OK`. If something is wrong You'll get + `Verification error: unable to verify the first certificate` and/or `Verify return code: 21 (unable to verify the first certificate)` + * You can also change `/etc/hosts` to resolve hostname and verify SSL certificate via browser + * When everything looks good delete the testing branch and proceed with production release. ## Tips @@ -65,4 +75,4 @@ should be changed accordingly. PEM strings can be encoded in different formats. Both cases are valid `-----BEGIN RSA PRIVATE KEY-----` `-----BEGIN PRIVATE KEY-----` -`openssl` will take care of correct decoding. [List of all supported formats](https://git.openssl.org/?p=openssl.git;a=blob;f=include/openssl/pem.h;hb=HEAD#l35). \ No newline at end of file +`openssl` will take care of correct decoding. [List of all supported formats](https://git.openssl.org/?p=openssl.git;a=blob;f=include/openssl/pem.h;hb=HEAD#l35). From a80b14334fad044c61f9ec452f5e0fb897ad0c24 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C4=81nis=20Bebr=C4=ABtis?= Date: Mon, 11 Sep 2023 13:28:15 +0300 Subject: [PATCH 08/20] Update vendor-gcs.md minor configuration example fix --- docs/vendor-gcs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/vendor-gcs.md b/docs/vendor-gcs.md index c56ec28..3767f1f 100644 --- a/docs/vendor-gcs.md +++ b/docs/vendor-gcs.md @@ -47,7 +47,7 @@ nginx: # Reverse proxy IP's to trust with contents of X-Forwarded-For header realipfrom: # Load Balancer IP (static ip you were given) - gce-lb-ip: 1.2.3.4/32; + gce-lb-ip: 1.2.3.4/32 ``` ### Cloud Armor @@ -82,4 +82,4 @@ mounts: storageClassName: nfs-shared ``` -Full example on using the provisioned storageclass in **new and existing** projects [here](gcp_filestore_migration.md) \ No newline at end of file +Full example on using the provisioned storageclass in **new and existing** projects [here](gcp_filestore_migration.md) From f284ca722642c096b3c2defc2f3b4e71b6ff51d1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C4=81nis=20Bebr=C4=ABtis?= Date: Wed, 27 Sep 2023 16:37:04 +0300 Subject: [PATCH 09/20] mariadb version change --- docs/silta-examples.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/silta-examples.md b/docs/silta-examples.md index 6790f23..07f2345 100644 --- a/docs/silta-examples.md +++ b/docs/silta-examples.md @@ -31,6 +31,25 @@ Note that storage can only be increased, not decreased. Note 2: If you change it for existing deployment, You'll need to run special comands in cluster to expand the storage or deployment will fail (see [Mariadb or Elasticsearch running out of disk space](troubleshooting.md#mariadb-or-elasticsearch-running-out-of-disk-space) in troubleshooting page). +## Using different version of MariaDB than provided in chart defaults. + +While it's normally not advised, it's possible to adjust MariaDB image version - + +_Drupal chart and Frontend chart_: + +```yaml +mariadb: + image: + # Available image tags listed at https://hub.docker.com/r/bitnami/mariadb/tags. Use debian images. + # tag: 10.10.6-debian-11-r25 + # tag: 10.11.5-debian-11-r24 + tag: 11.0.3-debian-11-r25 +``` + +It's highly suggested to create mysql data backup before image change. + +Note: Do not change image to an earlier version, it may break the data. + ## Mount Drupal public files to a different location _Drupal chart_: From c7d967ac4f23908e3fb2ab0902383abfb8ec43d1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C4=81nis=20Bebr=C4=ABtis?= Date: Thu, 28 Sep 2023 15:20:22 +0300 Subject: [PATCH 10/20] Update troubleshooting.md adjustment for nginx version example --- docs/troubleshooting.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 4ba0884..904cd01 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -20,13 +20,13 @@ Error: > nginx: [emerg] unknown directive "echo_sleep" in /etc/nginx/conf.d/drupal.conf:292 A: Ensure you are using an nginx version with this `echo` module compiled in. -In silta/nginx.Dockerfile, the FROM instructive should point to one of the newer versions, for example, latest available. +In silta/nginx.Dockerfile, the FROM instructive should point to one of the newer versions. Versions available are listed here: https://github.com/wunderio/silta-images/tree/master/silta-nginx Note: nginx:v0.1, wunderio/drupal Docker images do not have this module. -Example: `FROM wunderio/silta-nginx:latest` +Example: `FROM wunderio/silta-nginx:1.17-v1` # Issues with the deployed environments From c75cb538bcdd86cb37cdd37125655c149307a177 Mon Sep 17 00:00:00 2001 From: Janis Bebritis Date: Fri, 6 Oct 2023 14:04:49 +0300 Subject: [PATCH 11/20] Silta v1.0 components --- docs/anatomy_of_a_silta_project.md | 4 ++-- docs/gcp_filestore_migration.md | 2 +- docs/troubleshooting.md | 6 +++--- scripts/drupal7-migrate.sh | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/anatomy_of_a_silta_project.md b/docs/anatomy_of_a_silta_project.md index 3d549e8..7d1f6bb 100644 --- a/docs/anatomy_of_a_silta_project.md +++ b/docs/anatomy_of_a_silta_project.md @@ -16,7 +16,7 @@ This file is located in the project root at `.circleci/config.yml` and uses the version: 2.1 orbs: - silta: silta/silta@0.1 + silta: silta/silta@1 workflows: version: 2 @@ -58,7 +58,7 @@ We use version 2.1 of the CircleCI API. If your project configured to use an old ```yaml orbs: - silta: silta/silta@0.1 + silta: silta/silta@1 ``` CircleCI has a packaging system called [orbs](https://circleci.com/docs/2.0/orb-intro/#section=configuration). We have published our own orb called `silta/silta`, which enables you to use predefined jobs and commands. diff --git a/docs/gcp_filestore_migration.md b/docs/gcp_filestore_migration.md index 45d407e..100059b 100644 --- a/docs/gcp_filestore_migration.md +++ b/docs/gcp_filestore_migration.md @@ -87,7 +87,7 @@ If you run out of free space on volume, contact cluster administrator for its ex ``` Dockerfile example of a project ```dockerfile - FROM wunderio/silta-php-fpm:8.0-fpm-v0.1 + FROM wunderio/silta-php-fpm:8.0-fpm-v1 COPY --chown=www-data:www-data . /app diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 4ba0884..9171f3e 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -26,7 +26,7 @@ Versions available are listed here: https://github.com/wunderio/silta-images/tre Note: nginx:v0.1, wunderio/drupal Docker images do not have this module. -Example: `FROM wunderio/silta-nginx:latest` +Example: `FROM wunderio/silta-nginx:1.17-v1` # Issues with the deployed environments @@ -297,7 +297,7 @@ Content of [silta/shell.Dockerfile](https://github.com/wunderio/drupal-project/b ``` # Dockerfile for the Shell container. -FROM wunderio/silta-php-shell:php7.4-v0.1 +FROM wunderio/silta-php-shell:php7.4-v1 COPY --chown=www-data:www-data . /app ``` @@ -321,7 +321,7 @@ If You want to test docker images locally, You'd need to install docker or other Running a docker image: ```bash -docker run -it --entrypoint sh wunderio/silta-php-shell:php7.4-v0.1 +docker run -it --entrypoint sh wunderio/silta-php-shell:php7.4-v1 ``` This will download shell image and run a shell inside it. Typing `exit` will quit and stop the container. diff --git a/scripts/drupal7-migrate.sh b/scripts/drupal7-migrate.sh index cb6ad40..6ea35d9 100755 --- a/scripts/drupal7-migrate.sh +++ b/scripts/drupal7-migrate.sh @@ -15,7 +15,7 @@ cat > .circleci/config.yml << EOF version: 2.1 orbs: - silta: silta/silta@0.1 + silta: silta/silta@1 workflows: version: 2 From 312f6e9a6ee538017e641450b4198ddaacff826b Mon Sep 17 00:00:00 2001 From: Agnis Mateuss Date: Thu, 12 Oct 2023 11:28:38 +0300 Subject: [PATCH 12/20] EKS silta-cluster chart notes --- docs/vendor-eks.md | 55 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 52 insertions(+), 3 deletions(-) diff --git a/docs/vendor-eks.md b/docs/vendor-eks.md index a95ce13..d3fe830 100644 --- a/docs/vendor-eks.md +++ b/docs/vendor-eks.md @@ -4,14 +4,63 @@ Silta is mostly AWS compatible, there are some requirements for environments dep ## Cluster requirements -- Calico service which helps enforcing namespace (project) network separation; +- Ingress-nginx as the Ingress choice +- Amazon VPC CNI plugin for NetworkPolicy +- Amazon EBS CSI Driver plugin for default storage class (gp2) -There are few more requirements listed on [silta-cluster chart page](https://github.com/wunderio/charts/tree/master/silta-cluster#requirements), those are common for all silta-cluster installations +### Preparation steps + +On new, empty cluster, before installing silta-cluster chart: + +1. Install Amazon VPC CNI plugin (plugins are located in EKS -> cluster > Add-ons tab) +2. Install Amazon EBC CSI Driver plugin +3. Create and attach IAM role to worker nodes with these permissions: + - AmazonEC2ContainerRegistryFullAccess + - AmazonEC2FullAccess + - AmazonEKSWorkerNodePolicy + - AmazonElasticFileSystemFullAccess + - AmazonS3FullAccess + +### Silta-cluster chart requirements + +Enabling proxy protocol over ingress-nginx, for passing client IP to pods: +```yaml + ingress-nginx: + config: + use-proxy-protocol: true + service: + annotations: + "service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": "*" +``` +SSH uses NLB as ingress point. Apply these annotations: +```yaml +gitAuth: + enabled: true + scope: 'https://github.com/wunderio' + annotations: + "service.beta.kubernetes.io/aws-load-balancer-backend-protocol": "tcp" + "service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout": "60" + # "service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled": "true" + "service.beta.kubernetes.io/aws-load-balancer-type": "nlb" + # the length of the list must be equal to the number of subnets + "service.beta.kubernetes.io/aws-load-balancer-eip-allocations": "" + "service.beta.kubernetes.io/aws-load-balancer-subnets": " Elastic IPs + +Subnet names are in VPC Dashboard -> Virtual Private Cloud -> Subnets + +There are few more requirements listed on [silta-cluster chart page](https://github.com/wunderio/charts/tree/master/silta-cluster#requirements), those are common for all silta-cluster installations ## Missing functionality -- ALB ingress +- NLB for HTTP/HTTPS ingress ## Deployment specifics From 3e977b2881401dffe4e42aa0fb380bfed8469c61 Mon Sep 17 00:00:00 2001 From: Agnis Mateuss Date: Wed, 18 Oct 2023 11:35:45 +0300 Subject: [PATCH 13/20] added missing key --- docs/vendor-eks.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/vendor-eks.md b/docs/vendor-eks.md index d3fe830..56fa0fc 100644 --- a/docs/vendor-eks.md +++ b/docs/vendor-eks.md @@ -26,8 +26,9 @@ On new, empty cluster, before installing silta-cluster chart: Enabling proxy protocol over ingress-nginx, for passing client IP to pods: ```yaml ingress-nginx: - config: - use-proxy-protocol: true + controller: + config: + use-proxy-protocol: true service: annotations: "service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": "*" From 65cb520ce2fdd557fd38a842104ac48542c70e86 Mon Sep 17 00:00:00 2001 From: Rasmus Werling Date: Fri, 20 Oct 2023 11:04:18 +0300 Subject: [PATCH 14/20] Added vendor page about Upcloud UKS. --- docs/vendor-uks.md | 79 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 docs/vendor-uks.md diff --git a/docs/vendor-uks.md b/docs/vendor-uks.md new file mode 100644 index 0000000..f585126 --- /dev/null +++ b/docs/vendor-uks.md @@ -0,0 +1,79 @@ +# Upcloud compatibility + +Silta is mostly Upcloud compatible, there are some requirements for environments deployed to UKS cluster. + +## Cluster requirements + +- Load balancers are configured using json in annotations, see https://github.com/UpCloudLtd/uks-instructions/blob/main/ccm/README.md#customising-load-balancer-configuration + - By default, they are in HTTP mode, which needs to be changed to TCP + - Example configuration for ingress-nginx: + ```yaml + ingress-nginx: + controller: + admissionWebhooks: + enabled: true + autoscaling: + enabled: false + config: + use-forwarded-headers: "true" + compute-full-forwarded-for: "true" + use-proxy-protocol: "true" + real-ip-header: "proxy_protocol" + service: + type: LoadBalancer + annotations: + service.beta.kubernetes.io/upcloud-load-balancer-config: | + { + "name": "silta-ingress-1", + "plan": "production-small", + "frontends": [ + { + "name": "https", + "mode": "tcp", + "port": 443 + }, + { + "name": "http", + "mode": "tcp", + "port": 80 + } + ], + "backends": [ + { + "name": "https", + "properties": { "outbound_proxy_protocol": "v1"} + }, + { + "name": "http", + "properties": { "outbound_proxy_protocol": "v1"} + } + ] + } + ``` + +- Creating an object storage and configuring rclone is quite well explained at https://upcloud.com/resources/tutorials/migrate-object-storage-rclone + - Example configuration: + ```yaml + rclone: + params: + remote: s3 + remotePath: silta-shared + s3-access-key-id: + s3-acl: private + s3-endpoint: xyz.fi-hel2.upcloudobjects.com + s3-provider: Other + s3-region: fi-hel2 + s3-secret-access-key: + ``` + +There are few more requirements listed on [silta-cluster chart page](https://github.com/wunderio/charts/tree/master/silta-cluster#requirements), those are common for all silta-cluster installations + + +## Missing functionality + +- Managed Docker image registry +- Managed NFS storage + +## Deployment specifics + +There is no extra configuration required for basic deployments. The only change would be `cluster.type` but it's normally overridden in CI pipeline. From fdc36ef1e4cfd6048138a64f4ed0703fc8101a77 Mon Sep 17 00:00:00 2001 From: Agnis Mateuss Date: Tue, 14 Nov 2023 08:32:43 +0200 Subject: [PATCH 15/20] Added SSH annotation, extra notes --- docs/vendor-uks.md | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/docs/vendor-uks.md b/docs/vendor-uks.md index f585126..38b066b 100644 --- a/docs/vendor-uks.md +++ b/docs/vendor-uks.md @@ -50,6 +50,44 @@ Silta is mostly Upcloud compatible, there are some requirements for environments ] } ``` + - To enable whitelist for VPN, SSH service has to annotated with: + ``` + gitAuth: + annotations: + service.beta.kubernetes.io/upcloud-load-balancer-config: | + { + "name": "silta-ssh-1", + "plan": "development", + "frontends": [ + { + "name": "ssh", + "mode": "tcp", + "port": 22, + "rules": [ + { + "name": "allow-vpn", + "priority": 100, + "matchers": [ + { + "type": "src_ip", + "inverse": true, + "match_src_ip": { + "value": "" + } + } + ], + "actions": [ + { + "type": "tcp_reject", + "action_tcp_reject": {} + } + ] + } + ] + } + ] + } + ``` - Creating an object storage and configuring rclone is quite well explained at https://upcloud.com/resources/tutorials/migrate-object-storage-rclone - Example configuration: @@ -66,6 +104,10 @@ Silta is mostly Upcloud compatible, there are some requirements for environments s3-secret-access-key: ``` +- If using managed database, create a new database user and set authentication method to `mysql_native_password` + +- Smallest size for UKS storage volumes is 1Gi - set this for mariadb, elasticsearch pods + There are few more requirements listed on [silta-cluster chart page](https://github.com/wunderio/charts/tree/master/silta-cluster#requirements), those are common for all silta-cluster installations From c54135c373abe9353e8e26b1d8c64ae9e2746b0f Mon Sep 17 00:00:00 2001 From: Agnis Mateuss Date: Mon, 29 Jan 2024 08:51:08 +0200 Subject: [PATCH 16/20] compatibility table --- docs/compatibility_matrix.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 docs/compatibility_matrix.md diff --git a/docs/compatibility_matrix.md b/docs/compatibility_matrix.md new file mode 100644 index 0000000..d8ce66e --- /dev/null +++ b/docs/compatibility_matrix.md @@ -0,0 +1,23 @@ +Component \\ Vendor |GKE (Google) |AKS (Azure) |EKS (Amazon) |UKS (UpCloud) |microk8s (self-hosted) |minikube (local) +------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|----------------------------------------------------------------|---------------------------------------------------------------- +Container Image Registry |[Artifact Registry](https://cloud.google.com/artifact-registry) |[Azure Container Registry](https://azure.microsoft.com/en-us/products/container-registry/) |[Elastic Container Registry](https://aws.amazon.com/ecr/) |Missing** |[docker-registry](https://github.com/twuni/docker-registry.helm)*|[docker-registry](https://github.com/twuni/docker-registry.helm)* +Read-write many storage |[Filestore](https://cloud.google.com/filestore) |[Azure Files](https://azure.microsoft.com/en-us/products/storage/files) (azurefile-csi) |[Amazon S3 File Gateway (untested)](https://docs.aws.amazon.com/filegateway/latest/files3/what-is-file-s3.html)*** |Missing** |nfs-server* |nfs-server* +Silta-shared storage backend (rwx) |[Google Cloud Storage](https://cloud.google.com/storage/docs/buckets) |[Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) |[Amazon S3](https://aws.amazon.com/s3/) |Missing** |[MinIO](https://github.com/minio/minio)* |[MinIO](https://github.com/minio/minio)* +Load Balancer |[GKE Ingress](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress) |[Standard Load Balancer](https://learn.microsoft.com/en-us/azure/aks/load-balancer-standard) |[ELB\* & ALB](https://aws.amazon.com/elasticloadbalancing/features/) |[CCM](https://github.com/UpCloudLtd/uks-instructions/tree/main/ccm)|metallb* |metallb* +Static, reserved Ingress IP |Yes |Yes |Untested*** | |Yes |Yes +Static, reserved Egress IP |[CloudNAT](https://cloud.google.com/nat/docs/overview) (private clusters only), silta-proxy\* |Yes |Yes | |Yes |Yes +Network Policy |[Calico](https://cloud.google.com/kubernetes-engine/docs/how-to/network-policy#enabling_network_policy_enforcement)|[Calico or Network Policy Manager](https://learn.microsoft.com/en-us/azure/aks/use-network-policies) |Have to install* |[Cilium](https://cilium.io/) |Have to install* / Untested*** |Have to install* / Untested*** +Managed DBs |[CloudSQL](https://cloud.google.com/sql?hl=en) |[Azure Database for MySQL](https://azure.microsoft.com/en-us/products/mysql) |RDS*** |[Managed Databases](https://upcloud.com/products/managed-databases)| | +K8s versions |Multiple |Multiple |Multiple |1.26 |Multiple |Multiple +Web Application Firewall\* |[Cloud Armor (only for GKE ingress)](https://cloud.google.com/armor/) |[Application Gateway](https://azure.microsoft.com/en-us/products/application-gateway) (only for azure/application-gateway ingress) |AWS WAF*** | | | + +___ +Notes: +Load Balancing - all vendors support installing own ingress controller (Ingress-Nginx, Traefik) +Web Application Firewall - all vendors support [Signal Sciences WAF](https://www.signalsciences.com/) (in cluster agent) +ELB provides client ip via PROXY protocol +silta-proxy - requires separate nodepool and taints, does not work with all applications +___ +*Have to install +**Missing +***Untested From 1585ad59b2592308e2225bd423d9fff4ce270877 Mon Sep 17 00:00:00 2001 From: Rasmus Werling Date: Mon, 5 Feb 2024 11:58:15 +0200 Subject: [PATCH 17/20] Changed "notes" into a list --- docs/compatibility_matrix.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/compatibility_matrix.md b/docs/compatibility_matrix.md index d8ce66e..aa6b2e7 100644 --- a/docs/compatibility_matrix.md +++ b/docs/compatibility_matrix.md @@ -12,11 +12,11 @@ K8s versions Web Application Firewall\* |[Cloud Armor (only for GKE ingress)](https://cloud.google.com/armor/) |[Application Gateway](https://azure.microsoft.com/en-us/products/application-gateway) (only for azure/application-gateway ingress) |AWS WAF*** | | | ___ -Notes: -Load Balancing - all vendors support installing own ingress controller (Ingress-Nginx, Traefik) -Web Application Firewall - all vendors support [Signal Sciences WAF](https://www.signalsciences.com/) (in cluster agent) -ELB provides client ip via PROXY protocol -silta-proxy - requires separate nodepool and taints, does not work with all applications +Notes: +- Load Balancing - all vendors support installing own ingress controller (Ingress-Nginx, Traefik) +- Web Application Firewall - all vendors support [Signal Sciences WAF](https://www.signalsciences.com/) (in cluster agent) +- ELB provides client ip via PROXY protocol +- silta-proxy - requires separate nodepool and taints, does not work with all applications ___ *Have to install **Missing From 9c3b9ff5084414178e24c11276eba56aa661ce8b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C4=81nis=20Bebr=C4=ABtis?= Date: Fri, 8 Mar 2024 08:45:53 +0200 Subject: [PATCH 18/20] ingress-nginx on gke private clusters --- docs/vendor-gcs.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/vendor-gcs.md b/docs/vendor-gcs.md index 3767f1f..6104a21 100644 --- a/docs/vendor-gcs.md +++ b/docs/vendor-gcs.md @@ -83,3 +83,30 @@ mounts: ``` Full example on using the provisioned storageclass in **new and existing** projects [here](gcp_filestore_migration.md) + +### ingress-nginx load balancer on GKE private cluster + +When using GKE private cluster, [ingress-nginx](https://github.com/kubernetes/ingress-nginx) requires an additional firewall rule that allows control plane connection to nodes on port 8443. +Example and solution is borrowed from https://github.com/kubernetes/ingress-nginx/issues/5401 + +``` +# Control pane range (normally 172.16.0.0/28) +# gcloud container clusters describe [CLUSTER_NAME] --region europe-north1 --format json | jq -r '.privateClusterConfig.masterIpv4CidrBlock' +CONTROL_PLANE_RANGE=172.16.0.0/28 + +# Get cluster tag +NETWORK_TAGS=$(gcloud compute instances describe \ + $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') \ + --format="value(tags.items[0])") + +# Print firewall rule command +echo gcloud compute firewall-rules create silta-nginx-lb-ingress \ + --action ALLOW \ + --direction INGRESS \ + --source-ranges ${CONTROL_PLANE_RANGE} \ + --rules tcp:8443 \ + --target-tags ${NETWORK_TAGS} + +``` +Review and execute command printed above. + From c31ff951e4d3e2c9c9f407d55ce61c4fe7800cad Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C4=81nis=20Bebr=C4=ABtis?= Date: Thu, 21 Mar 2024 17:12:24 +0200 Subject: [PATCH 19/20] user directive for php dockerfile when filestore is used --- docs/vendor-gcs.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/vendor-gcs.md b/docs/vendor-gcs.md index 6104a21..ed2a1fb 100644 --- a/docs/vendor-gcs.md +++ b/docs/vendor-gcs.md @@ -82,6 +82,18 @@ mounts: storageClassName: nfs-shared ``` +Add `USER` directive to `silta/php.Dockerfile` right after the COPY line so files are created with correct permissions and can be modified via shell (i.e. `drush cr`). +```yaml +USER www-data +``` + +Dockerfile example of a project +```yaml +FROM wunderio/silta-php-fpm:8.2-fpm-v1 +COPY --chown=www-data:www-data . /app +USER www-data +``` + Full example on using the provisioned storageclass in **new and existing** projects [here](gcp_filestore_migration.md) ### ingress-nginx load balancer on GKE private cluster From f190a9fbe3d6e2b13457dba40fa1de755948566f Mon Sep 17 00:00:00 2001 From: Ragnar Kurm Date: Wed, 29 May 2024 13:32:15 +0300 Subject: [PATCH 20/20] Update vendor-eks.md --- docs/vendor-eks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/vendor-eks.md b/docs/vendor-eks.md index 56fa0fc..6a9bc0a 100644 --- a/docs/vendor-eks.md +++ b/docs/vendor-eks.md @@ -46,7 +46,7 @@ gitAuth: "service.beta.kubernetes.io/aws-load-balancer-type": "nlb" # the length of the list must be equal to the number of subnets "service.beta.kubernetes.io/aws-load-balancer-eip-allocations": "" - "service.beta.kubernetes.io/aws-load-balancer-subnets": ",,..." "service.beta.kubernetes.io/aws-load-balancer-nlb-target-type": "instance" "service.beta.kubernetes.io/aws-load-balancer-ip-address-type": "ipv4" "service.beta.kubernetes.io/aws-load-balancer-target-group-attributes": "stickiness.enabled=true,stickiness.type=source_ip,preserve_client_ip.enabled=true"