buildbot-master-pre-start fails ImportError: cannot import name 'exceptions' from 'cryptography.hazmat.bindings._rust'

[malikwirin]

I am currently struggling in setting up the master service.
When I rebuild the system the buildbot-master service fails with the following message:

warning: the following units failed: buildbot-master.service

× buildbot-master.service - Buildbot Continuous Integration Server.
     Loaded: loaded (/etc/systemd/system/buildbot-master.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2024-11-27 16:59:54 CET; 170ms ago
    Process: 286387 ExecStartPre=/nix/store/yp00zv77n4yraaj534lw4nc53w1n7vll-unit-script-buildbot-master-pre-start/bin/buildbot-master-pre-start (code=exited, status=1/FAILURE)
         IP: 0B in, 0B out
        CPU: 3.954s

Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/kc3x7f5ldgc3b856nvnbn8vljixqz7hk-python3-3.11.9-env/lib/python3.11/site-packages/cryptography/hazmat/primitives/kdf/pbkdf2.py", line 10, in <module>
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     from cryptography.exceptions import (
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/kc3x7f5ldgc3b856nvnbn8vljixqz7hk-python3-3.11.9-env/lib/python3.11/site-packages/cryptography/exceptions.py", line 9, in <module>
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]: ImportError: cannot import name 'exceptions' from 'cryptography.hazmat.bindings._rust' (unknown location)
Nov 27 16:59:54 hetzbox systemd[1]: buildbot-master.service: Control process exited, code=exited, status=1/FAILURE
Nov 27 16:59:54 hetzbox systemd[1]: buildbot-master.service: Failed with result 'exit-code'.
Nov 27 16:59:54 hetzbox systemd[1]: Failed to start Buildbot Continuous Integration Server..
Nov 27 16:59:54 hetzbox systemd[1]: buildbot-master.service: Consumed 3.954s CPU time, no IP traffic.
warning: error(s) occurred while switching to the new configuration

When using journalctl I get the following response:

Nov 27 16:59:50 hetzbox systemd[1]: Starting Buildbot Continuous Integration Server....
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]: Traceback (most recent call last):
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/a1hqx8gs6x1qyc9c505agx1rgmh6acgj-buildbot-4.0.3/bin/.buildbot-wrapped", line 9, in <module>
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     sys.exit(run())
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:              ^^^^^
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/a1hqx8gs6x1qyc9c505agx1rgmh6acgj-buildbot-4.0.3/lib/python3.12/site-packages/buildbot/scripts/runner.py", line 873, in run
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     subcommandFunction = reflect.namedObject(subconfig.subcommandFunction)
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/kc3x7f5ldgc3b856nvnbn8vljixqz7hk-python3-3.11.9-env/lib/python3.11/site-packages/twisted/python/reflect.py", line 169, in namedObject
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     module = namedModule(".".join(classSplit[:-1]))
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/kc3x7f5ldgc3b856nvnbn8vljixqz7hk-python3-3.11.9-env/lib/python3.11/site-packages/twisted/python/reflect.py", line 156, in namedModule
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     topLevel = __import__(name)
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:                ^^^^^^^^^^^^^^^^
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/a1hqx8gs6x1qyc9c505agx1rgmh6acgj-buildbot-4.0.3/lib/python3.12/site-packages/buildbot/scripts/create_master.py", line 23, in <module>
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     from buildbot.master import BuildMaster
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/a1hqx8gs6x1qyc9c505agx1rgmh6acgj-buildbot-4.0.3/lib/python3.12/site-packages/buildbot/master.py", line 51, in <module>
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     from buildbot.wamp import connector as wampconnector
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/a1hqx8gs6x1qyc9c505agx1rgmh6acgj-buildbot-4.0.3/lib/python3.12/site-packages/buildbot/wamp/connector.py", line 18, in <module>
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     from autobahn.twisted.wamp import ApplicationSession
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/prph3rfpm4mxlqnip2kb9pmbpm95i2zi-python3.12-autobahn-24.4.2/lib/python3.12/site-packages/autobahn/twisted/__init__.py", line 54, in <module>
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     from autobahn.twisted.wamp import ApplicationSession
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/prph3rfpm4mxlqnip2kb9pmbpm95i2zi-python3.12-autobahn-24.4.2/lib/python3.12/site-packages/autobahn/twisted/wamp.py", line 58, in <module>
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     from autobahn.wamp import protocol, auth
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/prph3rfpm4mxlqnip2kb9pmbpm95i2zi-python3.12-autobahn-24.4.2/lib/python3.12/site-packages/autobahn/wamp/auth.py", line 41, in <module>
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/kc3x7f5ldgc3b856nvnbn8vljixqz7hk-python3-3.11.9-env/lib/python3.11/site-packages/cryptography/hazmat/primitives/kdf/pbkdf2.py", line 10, in <module>
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     from cryptography.exceptions import (
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:   File "/nix/store/kc3x7f5ldgc3b856nvnbn8vljixqz7hk-python3-3.11.9-env/lib/python3.11/site-packages/cryptography/exceptions.py", line 9, in <module>
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]:     from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
Nov 27 16:59:54 hetzbox buildbot-master-pre-start[286396]: ImportError: cannot import name 'exceptions' from 'cryptography.hazmat.bindings._rust' (unknown location)
Nov 27 16:59:54 hetzbox systemd[1]: buildbot-master.service: Control process exited, code=exited, status=1/FAILURE
Nov 27 16:59:54 hetzbox systemd[1]: buildbot-master.service: Failed with result 'exit-code'.
Nov 27 16:59:54 hetzbox systemd[1]: Failed to start Buildbot Continuous Integration Server..
Nov 27 16:59:54 hetzbox systemd[1]: buildbot-master.service: Consumed 3.954s CPU time, no IP traffic.

My master configuration looks like this:

{ config, ... }:

let
  sopsCfg = config.sops;
in {
  services.buildbot-nix.master = {
    enable = true;
    domain = "buildbot.${config.networking.domain}";

    workersFile = sopsCfg.templates."buildbot-workers.json".path;
    admins = [ "malik" ];

    authBackend = "gitea";
    gitea = {
      enable = true;
      tokenFile = sopsCfg.secrets."codeberg-token".path;
      instanceUrl = "https://codeberg.org";
      oauthId = "redacted";
      oauthSecretFile = sopsCfg.secrets."cb-buildbot-secret".path;
      webhookSecretFile = sopsCfg.secrets."buildbot-webhook".path;
      topic = "build-with-buildbot";
    };
  };
}

Could this be caused by the way i configured the workersFile template? Or how I configured the webhook in the codeberg interface?
Thanks for any and every help.

[Mic92]

It's a bit unfortunate but buildbot-nix is broken with the nixos module in nixos 24.05. This is because we haven't backported NixOS/nixpkgs#353161
However I also noted some other weird breakage in 24.05 that I haven't figured out yet (github projects not receiving github events - not sure if this is a general issue).
I would recommend to upgrade to 24.11 already. It will be released very soon anyway. This has the said fix and worked great for me. In future we will hopefully will be a bit more resilient w.r.t. buildbot upgrades.

[malikwirin]

After switching to 24.11 i get a new Systemd error:

× buildbot-master.service - Buildbot Continuous Integration Server.
     Loaded: loaded (/etc/systemd/system/buildbot-master.service; enabled; preset: ignored)
     Active: failed (Result: exit-code) since Fri 2024-11-29 22:39:10 CET; 1s ago
 Invocation: 0e6f7d68cc8347cfb1b93a5a92a4aff7
    Process: 829423 ExecStartPre=/nix/store/m5yzgbhrb1yxi3vk8p3vk7jh1g505xsf-unit-script-buildbot-master-pre-start/bin/buildbot-master-pre-start (code=exited, status=243/CREDENTIALS)
         IP: 0B in, 0B out
         IO: 0B read, 0B written
   Mem peak: 1.5M
        CPU: 13ms

Nov 29 22:39:10 hetzbox systemd[1]: Starting Buildbot Continuous Integration Server....
Nov 29 22:39:10 hetzbox (re-start)[829423]: buildbot-master.service: Failed to set up credentials: Protocol error
Nov 29 22:39:10 hetzbox (re-start)[829423]: buildbot-master.service: Failed at step CREDENTIALS spawning /nix/store/m5yzgbhrb1yxi3vk8p3vk7jh1g505xsf-unit-script-buildbot-master-pre-start/bin/buildbot-master-pre-start: Protocol error
Nov 29 22:39:10 hetzbox systemd[1]: buildbot-master.service: Control process exited, code=exited, status=243/CREDENTIALS
Nov 29 22:39:10 hetzbox systemd[1]: buildbot-master.service: Failed with result 'exit-code'.
Nov 29 22:39:10 hetzbox systemd[1]: Failed to start Buildbot Continuous Integration Server..

[Mic92]

Looks like some credential path is incorrect:

buildbot-nix/nix/master.nix

Line 731 in 2695e33

LoadCredential =

Please make sure all secrets are present.