Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bitnami/postgresql] Enable override the archive_command value in postgresql containers #51872

Closed
wants to merge 3 commits into from
Closed

[bitnami/postgresql] Enable override the archive_command value in postgresql containers #51872

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
30d759a
Add variable for overriding the default archive_command value
Alvaro-Campesino Nov 7, 2023
384a376
Update bitnami/postgresql/README.md
Alvaro-Campesino Nov 15, 2023
f92d568
Update bitnami/postgresql/README.md
Alvaro-Campesino Nov 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions bitnami/postgresql/16/debian-11/rootfs/opt/bitnami/scripts/libpostgresql.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -368,6 +368,19 @@ postgresql_create_replication_user() {
echo "CREATE ROLE \"$POSTGRESQL_REPLICATION_USER\" REPLICATION LOGIN ENCRYPTED PASSWORD '$escaped_password'" | postgresql_execute
}

########################
# Change postgresql.conf by setting archive_command value
# Globals:
# POSTGRESQL_*
# Arguments:
# None
# Returns:
# None
#########################
postgresql_configure_archive_command() {
info "Overriding archive_command default value..."
postgresql_set_property "archive_command" "cp %p ${POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR}/%f"
}
########################
# Change postgresql.conf by setting replication parameters
# Globals:
Expand Down Expand Up @@ -614,15 +627,19 @@ postgresql_initialize() {
ensure_dir_exists "$dir"
am_i_root && chown "$POSTGRESQL_DAEMON_USER:$POSTGRESQL_DAEMON_GROUP" "$dir"
done
is_empty_value "$POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR" || ensure_dir_exists "$POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR" && am_i_root && chown "$POSTGRESQL_DAEMON_USER:$POSTGRESQL_DAEMON_GROUP" "$POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we use a separate volume to store the backups, this piece of code can be removed. Adding a validation in postgresql_validate could be a good idea

Copy link
Author

@Alvaro-Campesino Alvaro-Campesino Nov 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have been thinking about this and even if it is a separate volume , I think it is a good idea to use ensure_dir_exists as for example when using variables you can set the final path in a variable (which is my current setup), in order to have each pods WAL files in a separate folder.

    - name: POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR
      value: /backup/$(MY_POD_NAME)

What do you think?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case everything will work smoothly and I agree that's a good idea, you can have a shared backup volume and each container will write in its own folder.

Now if you have this set up (POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR directly points to the volume):

version: '2'

services:
  postgresql:
    image: bitnami/postgresql:debug
    build: .
    ports:
      - '5432:5432'
    volumes:
      - 'postgresql_data:/bitnami/postgresql'
      - 'postgresql_backup:/backups/postgresql'
    environment:
      - 'ALLOW_EMPTY_PASSWORD=yes'
      - 'POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR=/backups/postgresql'

volumes:
  postgresql_data:
    driver: local
  postgresql_backup:
    driver: local

The container will fail because the code below will try to change volume permissions:

debian-11-postgresql-1  | postgresql 08:17:40.12 INFO  ==> Loading custom pre-init scripts...
debian-11-postgresql-1  | postgresql 08:17:40.13 INFO  ==> Initializing PostgreSQL database...
debian-11-postgresql-1  | chmod: changing permissions of '/backups/postgresql': Operation not permitted
debian-11-postgresql-1 exited with code 1

am_i_root && find "$POSTGRESQL_DATA_DIR" -mindepth 1 -maxdepth 1 -not -name ".snapshot" -not -name "lost+found" -exec chown -R "$POSTGRESQL_DAEMON_USER:$POSTGRESQL_DAEMON_GROUP" {} \;
chmod u+rwx "$POSTGRESQL_DATA_DIR" || warn "Lack of permissions on data directory!"
chmod go-rwx "$POSTGRESQL_DATA_DIR" || warn "Lack of permissions on data directory!"
is_empty_value "$POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR" || chmod u+rw "$POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR"
Alvaro-Campesino marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to the issue mentioned above, something like this could be worth.

Suggested change
is_empty_value "$POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR" || chmod u+rw "$POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR"
is_empty_value "$POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR" || ! is_file_writable "$POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR" || chmod u+rw "$POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR"


is_boolean_yes "$POSTGRESQL_ALLOW_REMOTE_CONNECTIONS" && is_boolean_yes "$create_pghba_file" && postgresql_create_pghba && postgresql_allow_local_connection
# Configure port
postgresql_set_property "port" "$POSTGRESQL_PORT_NUMBER"
is_empty_value "$POSTGRESQL_DEFAULT_TOAST_COMPRESSION" || postgresql_set_property "default_toast_compression" "$POSTGRESQL_DEFAULT_TOAST_COMPRESSION"
is_empty_value "$POSTGRESQL_PASSWORD_ENCRYPTION" || postgresql_set_property "password_encryption" "$POSTGRESQL_PASSWORD_ENCRYPTION"
# Configure WAL backup with archive_command
is_empty_value "$POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR" || postgresql_configure_archive_command
if ! is_dir_empty "$POSTGRESQL_DATA_DIR"; then
info "Deploying PostgreSQL with persisted data..."
export POSTGRESQL_FIRST_BOOT="no"
Expand Down
2 changes: 2 additions & 0 deletions bitnami/postgresql/16/debian-11/rootfs/opt/bitnami/scripts/postgresql-env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ postgresql_env_vars=(
POSTGRESQL_PGHBA_REMOVE_FILTERS
POSTGRESQL_USERNAME_CONNECTION_LIMIT
POSTGRESQL_POSTGRES_CONNECTION_LIMIT
POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR
POSTGRESQL_WAL_LEVEL
POSTGRESQL_DEFAULT_TOAST_COMPRESSION
POSTGRESQL_PASSWORD_ENCRYPTION
Expand Down Expand Up @@ -325,6 +326,7 @@ POSTGRESQL_USERNAME_CONNECTION_LIMIT="${POSTGRESQL_USERNAME_CONNECTION_LIMIT:-"$
export POSTGRESQL_USERNAME_CONNECTION_LIMIT="${POSTGRESQL_USERNAME_CONNECTION_LIMIT:-}"
POSTGRESQL_POSTGRES_CONNECTION_LIMIT="${POSTGRESQL_POSTGRES_CONNECTION_LIMIT:-"${POSTGRES_POSTGRES_CONNECTION_LIMIT:-}"}"
export POSTGRESQL_POSTGRES_CONNECTION_LIMIT="${POSTGRESQL_POSTGRES_CONNECTION_LIMIT:-}"
export POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR="${POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR:-}"
export POSTGRESQL_WAL_LEVEL="${POSTGRESQL_WAL_LEVEL:-replica}"
export POSTGRESQL_DEFAULT_TOAST_COMPRESSION="${POSTGRESQL_DEFAULT_TOAST_COMPRESSION:-}"
export POSTGRESQL_PASSWORD_ENCRYPTION="${POSTGRESQL_PASSWORD_ENCRYPTION:-}"
Expand Down
4 changes: 4 additions & 0 deletions bitnami/postgresql/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,10 @@ A [Streaming replication](http://www.postgresql.org/docs/9.4/static/warm-standby

In a replication cluster you can have one master and zero or more slaves. When replication is enabled the master node is in read-write mode, while the slaves are in read-only mode. For best performance its advisable to limit the reads to the slaves.

### Setting up database incremental backup storage

* `POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR`: The Path were the archive command will store wal increments using `cp %p ${POSTGRESQL_WAL_ARCHIVE_COMMAND_DIR}/%f` by default `archive_command` value is `/bin/true` stored. For the backups to be useful is a good practice to use a dedicated volume for this, in addition when using several replicas, in order to avoid overwriting other replica data it is good to use variables for defining each replica subpath. For example: `/backup/$(MY_POD_NAME)`. No defaults.

Alvaro-Campesino marked this conversation as resolved.
Show resolved Hide resolved
#### Step 1: Create the replication master

The first step is to start the master.
Expand Down
Loading