[#22] Initial pgmoneta_ext_full_backup()

[GuChad369]

Initially, I use the pgmoneta_ext_full_backup() function to test and ensure that both start_backup() and stop_backup() functions work correctly.

Once these two functions are refined, the next step will be to create a new pull request to allow the core to send the manifest to the extension.

[jesperpedersen]

Yes, and like you have written you need to do the full backup, and figure out how to send it back to the client.

If, we have our own socket, then we need to do authentication and so on, so ideally client will connect to :5432, perform authentication, and then the extension takes over since its API is being called

[GuChad369]

I've noticed the following lines in our core project:

pgmoneta_create_start_replication_message(cmd, timeline, config->servers[srv].wal_slot, &start_replication_msg);
ret = pgmoneta_write_message(ssl, socket, start_replication_msg);

Is it possible for me to create a new function, similar to pgmoneta_create_start_replication_message, and use pgmoneta_write_message to initiate a LOGICAL replication to send the data? If I do this, can I rely on the existing functions to handle the authentication process?

Since our project already has many functions related to the protocol, I need to familiarize myself with them and learn more about them. Managing this communication may not be easy, as I'm currently unsure how our project handles the protocol and performs authentication.

[jesperpedersen]

Why would you need logical replication ?

You need to follow BASE_BACKUP from here https://www.postgresql.org/docs/current/protocol-replication.html that uses message types from the core protocol https://www.postgresql.org/docs/devel/protocol-message-formats.html

The core protocol does the authentication - you can sort of follow the authentication since our CLI does that too when connecting remotely. So, security.c with the various security methods should provide hints as well

[GuChad369]

The BASE_BACKUP command is typically used to create a full backup for replication, but I believe it is not designed to send individual or arbitrary files like a manifest file. Although I haven't yet found a proper method to send the file, I noticed that we already have a function, pgmoneta_ext_get_file, in the extension. Could we retrieve the manifest file from the extension side and then compare it with our core project on the core side, using the function we already have for comparison? Is this possible?

[GuChad369]

auth = pgmoneta_server_authenticate(server, database, username, password, replication, ssl, fd);

if (auth == AUTH_SUCCESS)
{
   // Proceed to establish a protocol
   pgmoneta_send_file_slot_message(slot_name, message, version);
   if (pgmoneta_write_message(ssl, socket, slot_request_msg) == MESSAGE_STATUS_OK)
   {
      // send the file
   }
   else
   {
      pgmoneta_log_error();
   }
}
else
{
   pgmoneta_log_error();
}
pgmoneta_close_ssl(ssl);
pgmoneta_disconnect(socket);

The pseudocode above outlines my plan to use our existing authentication mechanism for the authentication part. Once authenticated, we can find a suitable protocol to handle the file transfer.

[jesperpedersen]

The idea is to use the libpq API once we are inside the extension to perform the backup .

You should look how pgbackrest does incremental backup for PostgreSQL < 17 and get ideas from that

[jesperpedersen]

And, full backups as well of course

[GuChad369]

I've tried several methods to send a file from the client to the server, but none have succeeded, even when using PGconn for testing. So many components need to be integrated, and it's becoming difficult for me to solve. I'm feeling stuck at this point. Would you happen to have any advice?

[Jubilee101]

I've tried several methods to send a file from the client to the server, but none have succeeded, even when using PGconn for testing. So many components need to be integrated, and it's becoming difficult for me to solve. I'm feeling stuck at this point. Would you happen to have any advice?

What exactly have you tried and how are they not working out? Maybe we need to consider do comparison on main side. We may then need to get files one by one instead of in one tar. But I wouldn't worry too much about it since we now have start LSN thanks to #23.

There is no rush in this so don't worry too much, we can get support for v17 incr backup out first. If you need new ideas, checkout out pgbackrest about how they do it, especially how they manage to transfer the backups to client side. Postgres is doing the comparison based on WAL summaries, it is a quite interesting new feature, feel free to check it out as well.