Postgresql backup

[sivertism]

Postgresql backup

Hi, thought I'd have a go at enabling backups for postgresql, as it stores a lot of important data, for example for Nextcloud.

I'm not sure what the best way to do it is, so below are some of my thoughts.

One-shot backup vs per-service backup

I suppose there are many ways to set up the backup, but the simplest seems to be pg_dumpall which dumps all tables, schema, users etc. into an SQL file which can be piped back to psql to reproduce the DB. This is nice because it is simple, and because users etc. are preserved. It's also pretty nice for services that don't really have any state but their database -- they get backups for free.

Another way to do it would be to have each service back up its own database, i.e. nextcloud uses pg_dump to back up only the nextcloud database. The main advantage here, I suppose, is that the database dumped for nextcloud is (probably) in sync with the nextcloud backup. The main drawback is that it's a lot more code to set up, as it'd have to be done for all relevant services, and I'm guessing that restoring from backup might be a bit more challenging. At the very least, we'd have to restore from many backed up sql files.

Existing nixpkgs service

There is an existing service for backup from postgresql, but I thought it was better to use our own script. Both because it is very simple, and because it's easier to align the pg_dump with restic backups using the backup contract hooks.

Implementation and testing

I haven't tested out the implementation yet, and I'm not sure whether just putting the backup script in the default for the backup option is the right way to go?

[ibizaman]

I’ll need to look into more details and play with it but the general idea is what I had in mind indeed. Thank you for putting in the work!

[ibizaman]

I continued your work in PR #344

I adapted it to use the new database backup contract. Sorry I couldn't continue on your PR but I can't add commits in your fork.

[sivertism]

That's great, thank you! I'll try it out when I've got time to play with the server again 👍

[ibizaman]

Btw I'm working on better documentation, mostly helping readers in setting up their system, so hopefully that will help you too when you get to it.

[inayet]

Hi @ibizaman and @sivertism thank you to both of you for continuously improving selfhostblocks!

Qq - Will the docs be updated automatically where there are changes noted in CHANGELOG.md https://github.com/ibizaman/selfhostblocks/blob/main/CHANGELOG.md ?

I found this cool tool that is old but still being developed primarily used for Python development called nbdev https://github.com/fastai/nbdev where the docs and code are interlinked the best way I can describe it that documentation changes when the code changes. However I don't know if nbdev can be used / ported over to nix* ecosystem. Or does nix* have their own version of nbdev?

I think such tool would be great to use so that when something is changed the rest of the docs/code is updated so that when new comers starts to play with selfhostblocks they are *not derailed because of changes that happened and were not noted or updated in other areas of the docs/code

[ibizaman]

@inayet actually the documentation on https://shb.skarabox.com is generated from the code! The top-level file is https://github.com/ibizaman/selfhostblocks/blob/main/docs/manual.md and things get imported from there. Then when I merge a PR, this workflow gets started which ultimately deploys the documentation. I agree it's pretty cool to have that!

To develop locally, I run this command to refresh the documentation:

nix shell nixpkgs#entr --command bash -c 'find . | entr nix build .#manualHtml --offline'

Then in my browser I open the corresponding html file.