diff --git a/docs/contracts.md b/docs/contracts.md index 7c7c786..8ddb07a 100644 --- a/docs/contracts.md +++ b/docs/contracts.md @@ -4,12 +4,76 @@ A contract decouples modules that use a functionality from modules that provide intuition for contracts is they are generally related to accessing a shared resource. A few examples of contracts are generating SSL certificates, creating a user or knowing which files -and folders to backup. Indeed, when generating certificates, the service using those do not care how -they were created. They just need to know where the certificate files are located. +and folders to backup. +Indeed, when generating certificates, the service using those do not care how they were created. +They just need to know where the certificate files are located. + +A contract is made between a requester module and a provider module. +For example, a backup contract can be made between the Nextcloud service and the Restic service. +The former is the requester - the one wanted to be backed up - +and the latter is the provider of the contract - the one backing up files. + +## Schema {#contracts-schema} + +In practice, a contract is an attrset of options with a defined behavior. +Currently, the schema for a requester is: + +```nix +let + inherit (lib) mkOption; + inherit (lib.types) submodule; +in +config.${requester}.${contractname} = submodule { + request = mkOption { + type = contracts.${contractname}.request; + default = { + # Values set by the requester + }; + }; + + result = mkOption { + type = contracts.${contractname}.result; + }; +}; +``` + +For a provider, it is: + +```nix +let + inherit (lib) mkOption; + inherit (lib.types) anything submodule; +in +config.${provider}.${contractname} = submodule ({ options, ... }: { + request = mkOption { + type = contracts.${contractname}.request; + }; + + result = mkOption { + type = contracts.${contractname}.result; + default = { + # Values set by the provider + # Can depend on values set by the requester through the `options` variable. + }; + }; + + settings = mkOption { + type = anything; + }; +}); +``` + +## Contract Tests {#contracts-test} + +To make sure all providers module of a contract have the same behavior, +generic NixOS VM tests exist per contract. +They are generic because they work on any module, +as long as the module implements the contract of course. -In practice, a contract is a set of options that any user of a contract expects to exist. Also, the -values of these options dictate the behavior of the implementation. This is enforced with NixOS VM -tests. +For example, the [generic test][generic] for backup contract is instantiated for Restic [here][restic test]. + +[generic]: @REPO@/modules/contracts/backup/test.nix +[restic test]: @REPO@/test/contracts/backup.nix ## Videos {#contracts-videos} @@ -20,6 +84,59 @@ and the second at [NixCon in Berlin in fall of 2024][NixConBerlin2024]. [NixConNA2024]: https://www.youtube.com/watch?v=lw7PgphB9qM [NixConBerlin2024]: https://www.youtube.com/watch?v=CP0hR6w1csc +## Why do we need this new concept? {#contracts-why} + +Currently in nixpkgs, every module needing access to a shared resource must implement the logic +needed to setup that resource themselves. Similarly, if the module is mature enough to let the user +select a particular implementation, the code lives inside that module. + +![](./assets/contracts_before.png "A module composed of a core logic and a lot of peripheral logic.") + +This has a few disadvantages: + +- This leads to a lot of **duplicated code**. If a module wants to support a new implementation of a +contract, the maintainers of that module must write code to make that happen. +- This also leads to **tight coupling**. The code written by the maintainers cannot be reused in + other modules, apart from copy pasting. +- There is also a **lack of separation of concerns**. The maintainers of a service must be experts + in all implementations they let the users choose from. +- Finally, this is **not extensible**. If you, the user of the module, want to use another + implementation that is not supported, you are out of luck. You can always dive into the module's + code and extend it, but that is not an optimal experience. + +We do believe that the decoupling contracts provides helps alleviate all the issues outlined above +which makes it an essential step towards more adoption of Nix, if only in the self hosting scene. + +![](./assets/contracts_after.png "A module containing only logic using peripheral logic through contracts.") + +Indeed, contracts allow: + +- **Reuse of code**. + Since the implementation of a contract lives outside of modules using it, + using that implementation elsewhere is trivial. +- **Loose coupling**. + Modules that use a contract do not care how they are implemented, + as long as the implementation follows the behavior outlined by the contract. +- Full **separation of concerns** (see diagram below). + Now, each party's concern is separated with a clear boundary. + The maintainer of a module using a contract can be different from the maintainers + of the implementation, allowing them to be experts in their own respective fields. + But more importantly, the contracts themselves can be created and maintained by the community. +- Full **extensibility**. + The final user themselves can choose an implementation, + even new custom implementations not available in nixpkgs, without changing existing code. +- **Incremental adoption**. + Contracts can help bridge a NixOS system with any non-NixOS one. + For that, one can hardcode a requester or provider module to match + how the non-NixOS system is configured. + The responsability falls of course on the user to make sure both system agree on the configuration. +- Last but not least, **Testability**. + Thanks to NixOS VM test, we can even go one step further + by ensuring each implementation of a contract, even custom ones, + provides required options and behaves as the contract requires. + +![](./assets/contracts_separationofconcerns.png "Separation of concerns thanks to contracts.") + ## Provided contracts {#contracts-provided} Self Host Blocks is a proving ground of contracts. This repository adds a layer on top of services @@ -53,49 +170,6 @@ modules/contracts/databasebackup/docs/default.md modules/contracts/secret/docs/default.md ``` -## Why do we need this new concept? {#contracts-why} - -Currently in nixpkgs, every module needing access to a shared resource must implement the logic -needed to setup that resource themselves. Similarly, if the module is mature enough to let the user -select a particular implementation, the code lives inside that module. - -![](./assets/contracts_before.png "A module composed of a core logic and a lot of peripheral logic.") - -This has a few disadvantages: - -- This leads to a lot of **duplicated code**. If a module wants to support a new implementation of a -contract, the maintainers of that module must write code to make that happen. -- This also leads to **tight coupling**. The code written by the maintainers cannot be reused in - other modules, apart from copy pasting. -- There is also a **lack of separation of concerns**. The maintainers of a service must be experts - in all implementations they let the users choose from. -- Finally, this is **not extensible**. If you, the user of the module, want to use another - implementation that is not supported, you are out of luck. You can always dive into the module's - code and extend it, but that is not an optimal experience. - -We do believe that the decoupling contracts provides helps alleviate all the issues outlined above -which makes it an essential step towards more adoption of Nix, if only in the self hosting scene. - -![](./assets/contracts_after.png "A module containing only logic using peripheral logic through contracts.") - -Indeed, contracts allow: - -- **Reuse of code**. Since the implementation of a contract lives outside of modules using it, using - that implementation elsewhere is trivial. -- **Loose coupling**. Modules that use a contract do not care how they are implemented, as long as - the implementation follows the behavior outlined by the contract. -- Full **separation of concerns** (see diagram below). Now, each party's concern is separated with a - clear boundary. The maintainer of a module using a contract can be different from the maintainers - of the implementation, allowing them to be experts in their own respective fields. But more - importantly, the contracts themselves can be created and maintained by the community. -- Full **extensibility**. The final user themselves can choose an implementation, even new custom - implementations not available in nixpkgs, without changing existing code. -- Last but not least, **Testability**. Thanks to NixOS VM test, we can even go one step further by - ensuring each implementation of a contract, even custom ones, provides required options and - behaves as the contract requires. - -![](./assets/contracts_separationofconcerns.png "Separation of concerns thanks to contracts.") - ## Are there contracts in nixpkgs already? {#contracts-nixpkgs} Actually not quite, but close. There are some ubiquitous options in nixpkgs. Those I found are: diff --git a/docs/services.md b/docs/services.md index 62f180f..432f5fe 100644 --- a/docs/services.md +++ b/docs/services.md @@ -12,27 +12,28 @@ information is provided in the respective manual sections. | Service | Backup | Reverse Proxy | SSO | LDAP | Monitoring | Profiling | |-----------------------|--------|---------------|-----|-------|------------|-----------| -| [Nextcloud Server][1] | P (1) | Y | Y | Y | Y | P (2) | -| [Vaultwarden][2] | P (1) | Y | Y | Y | N | N | -| [Forgejo][3] | Y | Y | Y | Y | N | N | +| [Nextcloud Server][1] | Y (1) | Y | Y | Y | Y (2) | P (3) | +| [Vaultwarden][2] | Y (1) | Y | Y | Y | Y (2) | N | +| [Forgejo][3] | Y (1) | Y | Y | Y | Y (2) | N | Legend: **N**: no but WIP; **P**: partial; **Y**: yes -1. Does not backup the database yet. -2. Works but the traces are not exported to Grafana yet. +1. Database and data files are backed up separately. +2. Dashboard is common to all services. +3. Works but the traces are not exported to Grafana yet. [1]: services-nextcloud.html [2]: services-vaultwarden.html [3]: services-forgejo.html -```{=include=} chapters html:into-file=//services-vaultwarden.html -modules/services/vaultwarden/docs/default.md -``` - ```{=include=} chapters html:into-file=//services-nextcloud.html modules/services/nextcloud-server/docs/default.md ``` +```{=include=} chapters html:into-file=//services-vaultwarden.html +modules/services/vaultwarden/docs/default.md +``` + ```{=include=} chapters html:into-file=//services-forgejo.html modules/services/forgejo/docs/default.md ``` diff --git a/modules/blocks/postgresql.nix b/modules/blocks/postgresql.nix index cf48ce4..0ade725 100644 --- a/modules/blocks/postgresql.nix +++ b/modules/blocks/postgresql.nix @@ -67,12 +67,12 @@ in ``` ''; - type = contracts.databasebackup.requestType; + type = contracts.databasebackup.request; default = { user = "postgres"; - backupFile = "postgres.sql"; + backupName = "postgres.sql"; backupCmd = '' ${pkgs.postgresql}/bin/pg_dumpall | ${pkgs.gzip}/bin/gzip --rsyncable diff --git a/modules/blocks/restic.nix b/modules/blocks/restic.nix index 2753023..19f713c 100644 --- a/modules/blocks/restic.nix +++ b/modules/blocks/restic.nix @@ -10,7 +10,7 @@ let inherit (lib) generators hasPrefix mkIf nameValuePair optionalAttrs removePrefix; inherit (lib.types) attrsOf enum int ints listOf oneOf nonEmptyListOf nonEmptyStr nullOr path str submodule; - commonOptions = { + commonOptions = { name, options, prefix, ... }: { enable = mkEnableOption '' this backup intance. @@ -18,9 +18,13 @@ let but still provides the helper tool to restore snapshots ''; - passphraseFile = mkOption { - description = "Encryption key for the backups."; - type = path; + passphrase = contracts.secret.mkOption { + description = "Encryption key for the backup repository."; + mode = "0400"; + owner = options.request.value.user; + ownerText = "[shb.restic.${prefix}..request.user](#blocks-restic-options-shb.restic.${prefix}._name_.request.user)"; + restartUnits = [ (fullName name options.settings.value.repository) ]; + restartUnitsText = "[ [shb.restic.${prefix}..settings.repository](#blocks-restic-options-shb.restic.${prefix}._name_.settings.repository) ]"; }; repository = mkOption { @@ -101,7 +105,7 @@ in { options.shb.restic = { instances = mkOption { - description = "Each instance is backing up some directories to one repository."; + description = "Files to backup following the [backup contract](./contracts-backup.html)."; default = {}; type = attrsOf (submodule ({ name, options, ... }: { options = { @@ -121,7 +125,7 @@ in ''; type = submodule { - options = commonOptions; + options = commonOptions { inherit name options; prefix = "instances"; }; }; }; @@ -131,24 +135,27 @@ in Contains the output of the Restic provider. ''; - type = lib.types.anything; # contracts.databasebackup.result; default = { restoreScript = fullName name options.settings.value.repository; backupService = "${fullName name options.settings.value.repository}.service"; }; - defaultText = literalExpression '' - { - restoreScript = "${fullName "" { path = "path/to/repository"; }}"; - backupService = "${fullName "" { path = "path/to/repository"; }}.service"; - } - ''; + defaultText = { + restoreScriptText = "${fullName "" { path = "path/to/repository"; }}"; + backupServiceText = "${fullName "" { path = "path/to/repository"; }}.service"; + }; + type = contracts.backup.result { + restoreScript = fullName name options.settings.value.repository; + backupService = "${fullName name options.settings.value.repository}.service"; + restoreScriptText = "${fullName "" { path = "path/to/repository"; }}"; + backupServiceText = "${fullName "" { path = "path/to/repository"; }}.service"; + }; }; }; })); }; databases = mkOption { - description = "Databases to backup following the database backup contract."; + description = "Databases to backup following the [database backup contract](./contracts-databasebackup.html)."; default = {}; type = attrsOf (submodule ({ name, options, ... }: { options = { @@ -159,7 +166,7 @@ in Accepts values from a requester. ''; - type = contracts.databasebackup.requestType; + type = contracts.databasebackup.request; }; settings = mkOption { @@ -168,7 +175,7 @@ in ''; type = submodule { - options = commonOptions; + options = commonOptions { inherit name options; prefix = "databases"; }; }; }; @@ -178,17 +185,20 @@ in Contains the output of the Restic provider. ''; - type = contracts.databasebackup.resultType; default = { restoreScript = fullName name options.settings.value.repository; backupService = "${fullName name options.settings.value.repository}.service"; }; - defaultText = literalExpression '' - { - restoreScript = "${fullName "" { path = "path/to/repository"; }}"; - backupService = "${fullName "" { path = "path/to/repository"; }}.service"; - } - ''; + defaultText = { + restoreScriptText = "${fullName "" { path = "path/to/repository"; }}"; + backupServiceText = "${fullName "" { path = "path/to/repository"; }}.service"; + }; + type = contracts.databasebackup.result { + restoreScript = fullName name options.settings.value.repository; + backupService = "${fullName name options.settings.value.repository}.service"; + restoreScriptText = "${fullName "" { path = "path/to/repository"; }}"; + backupServiceText = "${fullName "" { path = "path/to/repository"; }}.service"; + }; }; }; })); @@ -249,7 +259,7 @@ in paths = instance.request.sourceDirectories; - passwordFile = toString instance.settings.passphraseFile; + passwordFile = toString instance.settings.passphrase.result.path; initialize = true; @@ -288,7 +298,7 @@ in dynamicFilesFrom = "echo"; - passwordFile = toString instance.settings.passphraseFile; + passwordFile = instance.settings.passphrase.result.path; initialize = true; @@ -310,7 +320,7 @@ in cmd = pkgs.writeShellScriptBin "dump.sh" instance.request.backupCmd; in [ - "--stdin-filename ${instance.request.backupFile} --stdin-from-command -- ${cmd}/bin/dump.sh" + "--stdin-filename ${instance.request.backupName} --stdin-from-command -- ${cmd}/bin/dump.sh" ]); }; }; @@ -371,7 +381,7 @@ in serviceConfig.Type = "oneshot"; script = (shblib.replaceSecrets { userConfig = instance.settings.repository.secrets // { - RESTIC_PASSWORD_FILE = instance.settings.passphraseFile; + RESTIC_PASSWORD_FILE = instance.settings.passphrase.result.path; RESTIC_REPOSITORY = instance.settings.repository.path; }; resultPath = "/run/secrets_restic_env/${fullName name instance.settings.repository}"; @@ -414,7 +424,7 @@ in sudo --preserve-env -u ${instance.request.user} ${pkgs.restic}/bin/restic $@ else shift - sudo --preserve-env -u ${instance.request.user} sh -c "${pkgs.restic}/bin/restic dump $@ ${instance.request.backupFile} | ${instance.request.restoreCmd}" + sudo --preserve-env -u ${instance.request.user} sh -c "${pkgs.restic}/bin/restic dump $@ ${instance.request.backupName} | ${instance.request.restoreCmd}" fi ''; in diff --git a/modules/blocks/restic/docs/default.md b/modules/blocks/restic/docs/default.md index 2ff082c..24fce18 100644 --- a/modules/blocks/restic/docs/default.md +++ b/modules/blocks/restic/docs/default.md @@ -2,7 +2,7 @@ Defined in [`/modules/blocks/restic.nix`](@REPO@/modules/blocks/restic.nix). -This block sets up a backup job using [Restic][restic]. +This block sets up a backup job using [Restic][]. [restic]: https://restic.net/ @@ -12,18 +12,38 @@ Specific integration tests are defined in [`/test/blocks/restic.nix`](@REPO@/tes ## Provider Contracts {#blocks-restic-contract-provider} -This block implements the [backup](contracts-backup.html) and [database backup](contracts-databasebackup.html) contracts. +This block provides: -Contract integration tests are defined in [`/test/contracts/backup.nix`](@REPO@/test/contracts/backup.nix). +- [backup contract](contracts-backup.html) under the [`shb.restic.instances`][instances] option. + It is tested with [contract tests][backup contract tests]. +- [database backup contract](contracts-databasebackup.html) under the [`shb.restic.databases`][databases] option. + It is tested with [contract tests][database backup contract tests]. -### One folder backed up to mounted hard drives {#blocks-restic-contract-provider-one} +[instances]: #blocks-restic-options-shb.restic.instances +[databases]: #blocks-restic-options-shb.restic.databases +[backup contract tests]: @REPO@/test/contracts/backup.nix +[database backup contract tests]: @REPO@/test/contracts/databasebackup.nix + +As requested by those two contracts, when setting up a backup with Restic, +a backup Systemd service and restore script are provided. +The restore script has all the secrets needed to access the repo, +the only requirement to run it is to be able to `sudo` in the expected user. + +## Usage {#blocks-restic-usage} + +The following examples assume usage of SOPS to provide secrets +although any blocks providing the [secrets contract][] works too. +The [secrets setup section](usage.html#usage-secrets) explains +how to setup SOPS. + +### One folder backed up manually {#blocks-restic-usage-provider-manual} The following snippet shows how to configure the backup of 1 folder to 1 repository. -We assume that the folder is used by the `myservice` service and is owned by a user of the same name. +We assume that the folder `/var/lib/myfolder` of the service `myservice` must be backed up. ```nix -shb.restic.instances.myservice = { +shb.restic.instances."myservice" = { request = { user = "myservice"; @@ -56,7 +76,41 @@ shb.restic.instances.myservice = { }; ``` -### One folder backed up to S3 {#blocks-restic-contract-provider-remote} +### One folder backed up with contract {#blocks-restic-usage-provider-contract} + +With the same example as before but assuming the `myservice` service +has a `myservice.backup` option that is a requester for the backup contract, +the snippet above becomes: + +```nix +shb.restic.instances."myservice" = { + request = config.myservice.backup; + + settings = { + enable = true; + + passphraseFile = ""; + + repository = { + path = "/srv/backups/myservice"; + timerConfig = { + OnCalendar = "00:00:00"; + RandomizedDelaySec = "3h"; + }; + }; + + retention = { + keep_within = "1d"; + keep_hourly = 24; + keep_daily = 7; + keep_weekly = 4; + keep_monthly = 6; + }; + }; +}; +``` + +### One folder backed up to S3 {#blocks-restic-usage-provider-remote} Here we will only highlight the differences with the previous configuration. @@ -81,46 +135,6 @@ This assumes you have access to such a remote S3 store, for example by using [Ba } ``` -## Secrets {#blocks-restic-secrets} - -To be secure, the secrets should deployed out of band, otherwise they will be world-readable in the nix store. - -To achieve that, I recommend [sops](usage.html#usage-secrets) although other methods work great too. -The code to backup to Backblaze with secrets stored in Sops would look like so: - -```nix -shb.restic.instances.myfolder.passphraseFile = config.sops.secrets."myservice/backup/passphrase".path; -shb.restic.instances.myfolder.repository = { - path = "s3:s3.us-west-000.backblazeb2.com/"; - secrets = { - AWS_ACCESS_KEY_ID.source = config.sops.secrets."backup/b2/access_key_id".path; - AWS_SECRET_ACCESS_KEY.source = config.sops.secrets."backup/b2/secret_access_key".path; - }; -}; - -sops.secrets."myservice/backup/passphrase" = { - sopsFile = ./secrets.yaml; - mode = "0400"; - owner = "myservice"; - group = "myservice"; -}; -sops.secrets."backup/b2/access_key_id" = { - sopsFile = ./secrets.yaml; - mode = "0400"; - owner = "myservice"; - group = "myservice"; -}; -sops.secrets."backup/b2/secret_access_key" = { - sopsFile = ./secrets.yaml; - mode = "0400"; - owner = "myservice"; - group = "myservice"; -}; -``` - -Pay attention that the owner must be the `myservice` user, the one owning the files to be backed up. -A `secrets` contract is in progress that will allow one to not care about such details. - ## Multiple directories to multiple destinations {#blocks-restic-usage-multiple} The following snippet shows how to configure backup of any number of folders to 3 repositories, diff --git a/modules/contracts/backup.nix b/modules/contracts/backup.nix index 3483d0b..9b0b258 100644 --- a/modules/contracts/backup.nix +++ b/modules/contracts/backup.nix @@ -7,7 +7,11 @@ in request = submodule { options = { user = mkOption { - description = "Unix user doing the backups."; + description = '' + Unix user doing the backups. + + Most of the time, this should be the user owning the files. + ''; type = str; }; @@ -17,7 +21,7 @@ in }; excludePatterns = mkOption { - description = "Patterns to exclude."; + description = "File patterns to exclude."; type = listOf str; default = []; }; @@ -28,13 +32,13 @@ in type = submodule { options = { before_backup = mkOption { - description = "Hooks to run before backup"; + description = "Hooks to run before backup."; type = listOf str; default = []; }; after_backup = mkOption { - description = "Hooks to run after backup"; + description = "Hooks to run after backup."; type = listOf str; default = []; }; @@ -44,16 +48,46 @@ in }; }; - result = submodule { + result = { + restoreScript, + restoreScriptText ? null, + backupService, + backupServiceText ? null, + }: submodule { options = { restoreScript = mkOption { - description = "Name of script that can restore the database."; + description = '' + Name of script that can restore the database. + One can then list snapshots with: + + ```bash + $ ${if restoreScriptText != null then restoreScriptText else restoreScript} snapshots + ``` + + And restore the database with: + + ```bash + $ ${if restoreScriptText != null then restoreScriptText else restoreScript} restore latest + ``` + ''; type = str; + default = restoreScript; + defaultText = restoreScriptText; }; backupService = mkOption { - description = "Name of service backing up the database."; + description = '' + Name of service backing up the database. + + This script can be ran manually to backup the database: + + ```bash + $ systemctl start ${if backupServiceText != null then backupServiceText else backupService} + ``` + ''; type = str; + default = backupService; + defaultText = backupServiceText; }; }; }; diff --git a/modules/contracts/backup/dummyModule.nix b/modules/contracts/backup/dummyModule.nix index 673c3d5..2d77984 100644 --- a/modules/contracts/backup/dummyModule.nix +++ b/modules/contracts/backup/dummyModule.nix @@ -3,10 +3,44 @@ let contracts = pkgs.callPackage ../. {}; inherit (lib) mkOption; + inherit (lib.types) anything submodule; in { options.shb.contracts.backup = mkOption { - description = "Contract for backups."; - type = contracts.backup.request; + description = '' + Contract for backing up files + between a requester module and a provider module. + + The requester communicates to the provider + what files to backup + through the `request` options. + + The provider reads from the `request` options + and backs up the requested files. + It communicates to the requester what script is used + to backup and restore the files + through the `result` options. + ''; + + type = submodule { + options = { + request = mkOption { + description = '' + Options set by a requester module of the backup contract. + ''; + type = contracts.backup.request; + }; + + result = mkOption { + description = '' + Options set by a provider module of the backup contract. + ''; + type = contracts.backup.result { + restoreScript = "my_restore_script"; + backupService = "my_backup_service.service"; + }; + }; + }; + }; }; } diff --git a/modules/contracts/backup/test.nix b/modules/contracts/backup/test.nix index b8b5996..b6b1832 100644 --- a/modules/contracts/backup/test.nix +++ b/modules/contracts/backup/test.nix @@ -15,7 +15,8 @@ in "/opt/files/A" "/opt/files/B" ], - settings, # repository -> attrset + settings, # { repository, config } -> attrset + extraConfig ? null, # { username } -> attrset }: pkgs.testers.runNixOSTest { inherit name; @@ -28,7 +29,10 @@ in inherit sourceDirectories; user = username; }; - settings = settings "/opt/repos/${name}"; + settings = settings { + inherit config; + repository = "/opt/repos/${name}"; + }; }) (mkIf (username != "root") { users.users.${username} = { @@ -37,6 +41,7 @@ in group = "root"; }; }) + (optionalAttrs (extraConfig != null) (extraConfig { inherit username; })) ]; }; @@ -44,10 +49,7 @@ in skipTypeCheck = true; testScript = { nodes, ... }: let - provider = getAttrFromPath providerRoot nodes.machine; - backupService = provider.result.backupService; - restoreScript = provider.result.restoreScript; - onAllSourceDirectories = f: concatMapStringsSep "\n" (path: indent 4 (f path)) sourceDirectories; + provider = (getAttrFromPath providerRoot nodes.machine).result; in '' from dictdiffer import diff @@ -88,8 +90,8 @@ in }) with subtest("First backup in repo"): - print(machine.succeed("systemctl cat ${backupService}")) - machine.succeed("systemctl start ${backupService}") + print(machine.succeed("systemctl cat ${provider.backupService}")) + machine.succeed("systemctl start ${provider.backupService}") with subtest("New content"): for path in sourceDirectories: @@ -110,7 +112,7 @@ in assert_files(path, {}) with subtest("Restore initial content from repo"): - machine.succeed("""${restoreScript} restore latest""") + machine.succeed("""${provider.restoreScript} restore latest""") for path in sourceDirectories: assert_files(path, { diff --git a/modules/contracts/databasebackup.nix b/modules/contracts/databasebackup.nix index 4d1653c..b7d4b6c 100644 --- a/modules/contracts/databasebackup.nix +++ b/modules/contracts/databasebackup.nix @@ -4,16 +4,20 @@ let inherit (lib.types) anything submodule str; in { - requestType = submodule { + request = submodule { options = { user = mkOption { - description = "Unix user doing the backups."; + description = '' + Unix user doing the backups. + + This should be an admin user having access to all databases. + ''; type = str; example = "postgres"; }; - backupFile = mkOption { - description = "Filename of the backup."; + backupName = mkOption { + description = "Name of the backup in the repository."; type = str; default = "dump"; example = "postgresql.sql"; @@ -38,7 +42,12 @@ in }; - resultType = submodule { + result = { + restoreScript, + restoreScriptText ? null, + backupService, + backupServiceText ? null, + }: submodule { options = { restoreScript = mkOption { description = '' @@ -46,17 +55,18 @@ in One can then list snapshots with: ```bash - $ my_restore_script snapshots + $ ${if restoreScriptText != null then restoreScriptText else restoreScript} snapshots ``` And restore the database with: ```bash - $ my_restore_script restore latest + $ ${if restoreScriptText != null then restoreScriptText else restoreScript} restore latest ``` ''; type = str; - example = "my_restore_script"; + default = restoreScript; + defaultText = restoreScriptText; }; backupService = mkOption { @@ -66,11 +76,12 @@ in This script can be ran manually to backup the database: ```bash - $ systemctl start my_backup_service + $ systemctl start ${if backupServiceText != null then backupServiceText else backupService} ``` ''; type = str; - example = "my_backup_service.service"; + default = backupService; + defaultText = backupServiceText; }; }; }; diff --git a/modules/contracts/databasebackup/dummyModule.nix b/modules/contracts/databasebackup/dummyModule.nix index 2671c52..fdf3672 100644 --- a/modules/contracts/databasebackup/dummyModule.nix +++ b/modules/contracts/databasebackup/dummyModule.nix @@ -26,16 +26,19 @@ in options = { request = mkOption { description = '' - Options set by a requester module of the database contract. + Options set by a requester module of the database backup contract. ''; - type = contracts.databasebackup.requestType; + type = contracts.databasebackup.request; }; result = mkOption { description = '' - Options set by a provider module of the database contract. + Options set by a provider module of the database backup contract. ''; - type = contracts.databasebackup.resultType; + type = contracts.databasebackup.result { + restoreScript = "my_restore_script"; + backupService = "my_backup_service.service"; + }; }; }; }; diff --git a/modules/contracts/databasebackup/test.nix b/modules/contracts/databasebackup/test.nix index 34ce383..e9d374a 100644 --- a/modules/contracts/databasebackup/test.nix +++ b/modules/contracts/databasebackup/test.nix @@ -9,7 +9,7 @@ in { name, requesterRoot, providerRoot, - providerExtraConfig ? null, # { username, database } -> attrset + extraConfig ? null, # { username, database } -> attrset modules ? [], username ? "me", database ? "me", @@ -31,7 +31,7 @@ in group = "root"; }; }) - (optionalAttrs (providerExtraConfig != null) (providerExtraConfig { inherit username database; })) + (optionalAttrs (extraConfig != null) (extraConfig { inherit username database; })) ]; }; @@ -72,6 +72,7 @@ in with subtest("drop database"): machine.succeed(peer_cmd("DROP DATABASE me", db="postgres")) + machine.fail(peer_cmd("SELECT * FROM test")) with subtest("restore"): print(machine.succeed("readlink -f $(type ${provider.restoreScript})")) diff --git a/modules/contracts/secret.nix b/modules/contracts/secret.nix index 7ab3ccc..b679bab 100644 --- a/modules/contracts/secret.nix +++ b/modules/contracts/secret.nix @@ -4,8 +4,10 @@ { description, mode ? "0400", owner ? "root", + ownerText ? null, group ? "root", restartUnits ? [], + restartUnitsText ? null, }: lib.mkOption { inherit description; @@ -16,6 +18,15 @@ inherit mode owner group restartUnits; }; + defaultText = lib.optionalString (ownerText != null || restartUnitsText != null) (lib.literalMD '' + { + mode = ${mode}; + owner = ${if ownerText != null then ownerText else owner}; + group = ${group}; + restartUnits = ${if restartUnitsText != null then restartUnitsText else "[ " + lib.concatStringsSep " " restartUnits + " ]"}; + } + ''); + readOnly = true; description = '' @@ -56,6 +67,7 @@ ''; type = lib.types.str; default = owner; + defaultText = if ownerText != null then lib.literalMD ownerText else null; }; group = lib.mkOption { @@ -72,6 +84,7 @@ ''; type = lib.types.listOf lib.types.str; default = restartUnits; + defaultText = if restartUnitsText != null then lib.literalMD restartUnitsText else null; }; }; }; diff --git a/test/common.nix b/test/common.nix index 8e5f2c2..6af88ee 100644 --- a/test/common.nix +++ b/test/common.nix @@ -149,7 +149,7 @@ in shb.hardcodedsecret.ldapUserPassword = config.shb.ldap.ldapUserPassword.request // { content = "ldapUserPassword"; }; - shb.hardcodedsecret.jwtSecret = config.shb.ldap.ldapUserPassword.request // { + shb.hardcodedsecret.jwtSecret = config.shb.ldap.jwtSecret.request // { content = "jwtSecrets"; }; diff --git a/test/contracts/backup.nix b/test/contracts/backup.nix index 3a40186..c9870f5 100644 --- a/test/contracts/backup.nix +++ b/test/contracts/backup.nix @@ -9,10 +9,11 @@ in providerRoot = [ "shb" "restic" "instances" "mytest" ]; modules = [ ../../modules/blocks/restic.nix + ../../modules/blocks/hardcodedsecret.nix ]; - settings = repository: { + settings = { repository, config, ... }: { enable = true; - passphraseFile = toString (pkgs.writeText "passphrase" "PassPhrase"); + passphrase.result.path = config.shb.hardcodedsecret.passphrase.path; repository = { path = repository; timerConfig = { @@ -20,6 +21,12 @@ in }; }; }; + extraConfig = { username, ... }: { + shb.hardcodedsecret.passphrase = { + owner = username; + content = "passphrase"; + }; + }; }; restic_me = contracts.test.backup { @@ -28,10 +35,11 @@ in providerRoot = [ "shb" "restic" "instances" "mytest" ]; modules = [ ../../modules/blocks/restic.nix + ../../modules/blocks/hardcodedsecret.nix ]; - settings = repository: { + settings = { repository, config, ... }: { enable = true; - passphraseFile = toString (pkgs.writeText "passphrase" "PassPhrase"); + passphrase.result.path = config.shb.hardcodedsecret.passphrase.path; repository = { path = repository; timerConfig = { @@ -39,5 +47,11 @@ in }; }; }; + extraConfig = { username, ... }: { + shb.hardcodedsecret.passphrase = { + owner = username; + content = "passphrase"; + }; + }; }; } diff --git a/test/contracts/databasebackup.nix b/test/contracts/databasebackup.nix index b91f4e1..b1c18ea 100644 --- a/test/contracts/databasebackup.nix +++ b/test/contracts/databasebackup.nix @@ -13,7 +13,7 @@ in ]; settings = repository: { enable = true; - passphraseFile = toString (pkgs.writeText "passphrase" "PassPhrase"); + passphrase.result.path = pkgs.writeText "passphrase" "PassPhrase"; repository = { path = repository; timerConfig = { @@ -21,7 +21,7 @@ in }; }; }; - providerExtraConfig = { username, database, ... }: { + extraConfig = { username, database, ... }: { shb.postgresql.ensures = [ { inherit username database;