diff --git a/docs/database/CLUSTER_DB.MD b/docs/database/CLUSTER_DB.MD index 78c36cab4..bde57717a 100644 --- a/docs/database/CLUSTER_DB.MD +++ b/docs/database/CLUSTER_DB.MD @@ -59,4 +59,62 @@ Once a standby is stood up, it can be promoted to be the primary cluster. **Note Promote the standby cluster by editing the [crunchy_standby.yaml](../../openshift/templates/crunchy_standby.yaml) to set the `standby` field to `false`. -More details here: https://access.crunchydata.com/documentation/postgres-operator/latest/architecture/disaster-recovery#promoting-a-standby-cluster +More details here: + +## Cluster Restore From pg_dump + +In the event that the cluster can't be restored from pgbackrest you can create a new cluster and restore using a pg_dump from S3. + +##### Deploy new cluster + + ``` + oc login --token= --server= + PROJ_TARGET= BUCKET= CPU_REQUEST=75m CPU_LIMIT=2000m MEMORY_REQUEST=2Gi MEMORY_LIMIT=16Gi DATA_SIZE=65Gi WAL_SIZE=45Gi bash ./oc_provision_crunchy.sh apply + ``` + +##### Set superuser permissions in new cluster via OpenShift web GUI + +Login to the OpenShift UI and use `patronictl list` to identify the new cluster's leader pod. The role to update will be something like `wps-crunchydb-16-`. You can confirm the role by exploring the pg_roles table with: +`psql -c "SELECT * FROM pg_roles"`. +Access the terminal of the leader pod and execute: +`psql -c 'ALTER ROLE "" SUPERUSER'` + +##### Setup port forwarding from your local machine to the new cluster + +Use the OpenShift UI to determine the name of the secret the CrunchyDB Operator generated following creation of the empty cluster (eg. wps-crunchydb-16-\-pguser-wps-crunchydb-16-\). Ensure you have stopped your local postgres instance if it is listening on port 5432. + +``` +PG_CLUSTER_PRIMARY_POD= +PGPASSWORD=$(oc get secrets -n "" -o go-template='{{.data.password | base64decode}}') +PGUSER=$(oc get secrets -n "" -o go-template='{{.data.user | base64decode}}') +PGDATABASE=$(oc get secrets -n "" -o go-template='{{.data.dbname | base64decode}}') +oc -n port-forward "${PG_CLUSTER_PRIMARY_POD}" 5432:5432 +``` + +##### Restore sql dump into new cluster in another shell + +Download the latest SQL dump from S3 storage and unzip it. + +``` +PG_CLUSTER_PRIMARY_POD= +PGPASSWORD=$(oc get secrets -n "" -o go-template='{{.data.password | base64decode}}') +PGUSER=$(oc get secrets -n "" -o go-template='{{.data.user | base64decode}}') +PGDATABASE=$(oc get secrets -n "" -o go-template='{{.data.dbname | base64decode}}') +PGPASSWORD=$PGPASSWORD psql -U $PGUSER -d wps -h localhost < wps-crunchydb-sql-dump-name.sql +``` + +##### Remove superuser privileges from pguser + +This step is required as pgbouncer will not connect to the cluster/database with a superuser. +Login to the OpenShift UI and use `patronictl list` to identify the new cluster's leader pod. Access the terminal of the leader pod and execute: +`psql -c 'ALTER ROLE "" NOSUPERUSER'` + +##### Update the prod deployment to use the new crunchydb cluster and pguser secret + +Create a PR with the following changes: + +- Update `CRUNCHYDB_USER` to the new crunchydb pguser secret in all scripts in openshift/scripts. +- Update `PATRONI_CLUSTER_NAME` in `deploy.yaml`. + +Deploy the PR once approved. +See [database upgrade PR](https://github.com/bcgov/wps/pull/4100) for reference.