terraform: add var.special_args, closes #413, supersedes #414.

[KiaraGrouwstra]

This PR adds a Terraform input variable named special_args.

This allows passing in a map from Terraform to expose to NixOS's specialArgs at build-time.
Example usage, in this case presuming deployment to a Hetzner Cloud server (resource.hcloud_server):

let
  servers = ...;
  variable = ...;
  data = ...;
  resource = ...;
in
{
  inherit variable data resource;
  module =
    lib.mapAttrs (server_name: _server_config: let
    in {
      # pin module version by nix flake inputs
      source =
"github.com/numtide/nixos-anywhere?ref=${inputs.nixos-anywhere.sourceInfo.rev}/terraform/all-in-one";
      ...
      special_args = {
        tf = {
          inherit server_name;
          # all variables
          # var = lib.mapAttrs (k: _: lib.tfRef "var.${k}") variable;
          # non-sensitive variables
          var = lib.mapAttrs (k: _: lib.tfRef "var.${k}")
  (lib.filterAttrs (_k: v: !(v ? sensitive && v.sensitive)) variable);
          data = lib.mapAttrs (type: instances: lib.mapAttrs (k: _:
  tfRef "data.${type}.${k}") instances) data;
          resource = lib.mapAttrs (type: instances: lib.mapAttrs (k:
  _: tfRef "resource.${type}.${k}") instances) resource;
          server = lib.tfRef "resource.hcloud_server.${server_name}";
        };
      };
    })
    servers;
}

You can then use these in your nixosConfigurations, in this example thru the tf argument.

As a note on security, information passed this way will hit /nix/store/. As such, the above usage example has defaulted to omitting TF variables marked as sensitive.

This PR incorporates ideas from:

• @aanderse, who implemented a similar feature in teraflops that inspired this PR.
• @Mic92, who suggested (see terraform: add var.extra_build_env_vars, closes #413 #414) to extend the original lib.nixosSystem call to pass in info without --impure or staging to Git.
• @getchoo, who suggested getting the NAR hash by nix flake prefetch over getFlake, fixing an 'unlocked flake reference' error on (non-Lix) Nix.
• @threddast, who suggested to use TF's any type to automate serializing.

An alternative design suggested by @Mic92 involved passing the information not directly, but rather thru a file. The idea would be that this might help reduce the risk of stack overflows, tho I have imagined (perhaps naively) that TF info has tended not to get too big, whereas I also had a bit more trouble getting that approach to work properly so far (involving both NARs that would suddenly mismatch again, while I'd also yet to test if one could put such files in .gitignore).

[Mic92]

Looks much better. Specialargs are also a good idea, because it will allow it uses values also for imports.

[Mic92]

Oh wait, this is computing the nar from the flake that defines the NixOS config and than passes in the special args as a command line argument again. The idea was to do the same nar calculation for the JSON file that is passed in as an input and than just use getFlake on the original configuration and pass in the JSON file path via readFile. This than avoids large string I can follow up later with more specific, currently on my phone. But it's going in the right direction.

[KiaraGrouwstra]

@Mic92 yes, that sounds like this earlier iteration where i tried specialArgs thru a file:

An alternative design suggested by @Mic92 involved passing the information not directly, but rather thru a file. The idea would be that this might help reduce the risk of stack overflows, tho I have imagined (perhaps naively) that TF info has tended not to get too big, whereas I also had a bit more trouble getting that approach to work properly so far (involving both NARs that would suddenly mismatch again, while I'd also yet to test if one could put such files in .gitignore).

[KiaraGrouwstra]

i did consider a hybrid, where an optional filename argument could be set such as to go thru that... but felt like the feature as-is seemed functional enough to submit already

[Mic92]

Okay. I will try out later.

[KiaraGrouwstra]

extra notes:

• this is only supported using flakes currently (edit: pushed a commit to error clarifying this to the user)
• it would also be possible to split this up for toplevel vs diskoScriptNoDeps - so far i had not done that

[Mic92]

I still have it on my radar, but haven't got around to test it.

[threddast]

I'm trying to test this, but I'm always getting this error

nix build --expr "builtins.getFlake ''git+file://${flake_dir}?narHash=sha256-0000000000000000000000000000000000000000000=''"     
error:
       … while calling the 'getFlake' builtin
         at «string»:1:1:
            1| builtins.getFlake ''git+file:///home/threddast/devel/cloud-infra?narHash=sha256-0000000000000000000000000000000000000000000=''
             | ^

       error: cannot call 'getFlake' on unlocked flake reference 'git+file:///home/threddast/devel/cloud-infra?narHash=sha256-0000000000000000000000000000000000000000000=', at «none»:0 (use --impure to override)

Even though I have a flake.lock committed in git.
I get the same also when trying with the flake from https://codeberg.org/kiara/e2ed-hetzner
My Nix version is 2.24.10

Moreover, I'm not sure it works if the flake.nix is in a parent directory relative to where terraform plan is run

[KiaraGrouwstra]

@threddast thank you for your feedback.
i did run into issues like that as well using Nix, tho using Lix seemed to address this.
i guess it might be possible to bisect Nix versions on that to see if/where it gets fixed.
a search in the Nix repo shows similar symptoms had been reported previously, tho it seems the present bug (regression?) seems to pop up even with the NAR hash specified and all.
i imagine nix would fix it eventually, whereas lix seems just more stable already.

[KiaraGrouwstra]

@threddast courtesy of @getchoo's advice on lix matrix, i pushed an update to calculate the NAR hash differently.
it's looking like that should address the issues experienced when running this with Nix.

[threddast]

Thank you, I tested the module and it works really well now.
Just a suggestion, would it make sense to move the jsonencode inside the module, similar to the nix_options variable?

[Mic92]

@KiaraGrouwstra is this example correct? I derived it from your pull request message.

[KiaraGrouwstra]

nit: the user can pass arguments under any name (or multiple even), rather than just terraform - otherwise this sounds right!

[Mic92]

If you could also solve the merge conflict, than we could finally merge this.
Thank you!

[Mic92]

@mergify queue

[mergify]

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 80a2e7d