Please document a way to run k0s when one has to manage systemd units manually

[NobbZ]

Is your feature request related to a problem? Please describe.

My linux distribution (NixOS) manages its systemd services declaratively, and there is no way to write to to /etc/systemd. One has to write appropritate system configuration, in a JSON like language (which also supports functions).

I have no problems manually translating systemd-units into the required syntax.

Describe the solution you'd like

Provide documentation about how a service might look like and what it has to start or how the environment needs to get prepared.

Alternatively provide a way to print the unit file(s) to stdout or a manually specified location.

Describe alternatives you've considered

An alternative would be a clearly documented way to start and stop k0s without having a service installed, this would make k0s compatible with all init systems, not only OpenRC and systemd.

Additional context

• Error when trying to install:

  sudo /tmp/tmp.pWDR3S7Czl/k0s install controller
  INFO[2021-12-08 06:55:26] no config file given, using defaults         
  INFO[2021-12-08 06:55:26] Installing k0s service                       
  Error: failed to install k0s service: failed to install service: open /etc/systemd/system/k0scontroller.service: read-only file system
  2021/12/08 06:55:26 failed to install k0s service: failed to install service: open /etc/systemd/system/k0scontroller.service: read-only file system
  

• NixOS website: https://nixos.org

[github-actions]

The issue is marked as stale since no activity has been recorded in 30 days

[kke]

You can run k0s controllers and workers simply with k0s controller or k0s worker. There aren't many arguments and the defaults are ok.

Adding a "running without installation" or a simple systemd unit example to the docs should be a quick task.

[johbo]

Just got across this issue and realized that it has a counterpart in the Nixpkgs (package collection of NixOs): NixOS/nixpkgs#247158

I've tried to install k0s as described in the documentation via k0sctl, mainly because I was keen to try out the backup and restore options. As mentioned in the issue description, the main problem is the k0s assumes that it can write into /etc/systemd/system/... and in NixOS that's on a read only mounted file system on purpose.

I could think of the following options to make it work:

• Allow to tell k0s which path to write the service files into, I think NixOS has something like /etc/systemd/system-mutable or similar
• Allow to select to dump the suggested service configuration to stdout
• Or simple allow to skip this step with a parameter

I've noticed that it seems to first check if there is an existing configuration already. When I first applied the needed configuration in NixOS, so that the service configuration was present, then I did not have any trouble with the installation procedure triggered from k0sctl anymore. So this can be a workaround until a solution is found.

Note: k0sctl does not recognize nixos as a distribution, I've set it to debian to be able to use it.

[kke]

Maybe k0sctl could be made to support nixos, but looking at their mission objective, I'm not sure if that's the right way.

[twz123]

Hey @johbo! Thanks for linking the NixOS packaging request here, I wasn't aware of it. Being a NixOS user myself, I've thought about how to integrate k0s into NixOS for quite a while already. As you already found out, k0s's build process is based on Docker and not a piece of cake. Trying to repro the build inside Nix without Docker would be a rather thorny and brittle endeavor. (You can run the Go build without Docker by setting the GO and GO_ENV Makefile variables. But this is only for the k0s binary itself, not for all the other upstream binaries.)

Given the declarative way of doing things in NixOS, I'd rather not use k0sctl to distribute k0s to NixOS machines, but rather embed it directly into the Nix configuration that's used to build the whole system and then distribute that config to the machines as you would usually do (e.g. via NixOps or what have you).

I, for myself, came to the conclusion that I would be happy enough to just write my own configuration using the binary release. K0s tries to be as zero dependency as humanly possible, so it's linking all its binaries statically. They all work on NixOS out of the box. So I'd do the following: In the machine's Nix configuration, fetch the k0s binary from GitHub and put it in a simple derivation that just contains the binary as bin/k0s. (Adding shell completions could be done as well if you want them.) Add that derivation to environment.systemPackages (if you want to have k0s in the PATH), and add the systemd service to systemd.services, as you already showcased in NixOS/nixpkgs#247158 (comment). Then rebuild your NixOS config. That should be enough. You can wrap that up into some module, if you want to.

Concerning the content of the unit files, as @kke already mentioned, there's not much in them anyways. Just add the flags that you need to the Exec line. Note that k0s shouldn't require modprobe (given that all the required modules are (auto-)loaded) or mount, so it shouldn't be a hard requirement to add those to k0s's PATH. The docs section about external runtime dependencies might be helpful here.

[TECHNOFAB11]

I was able to build k0s and just give it the other kubernetes tools via PATH on NixOS (see NixOS/nixpkgs#247158 (comment)).
The build process with Docker etc. was pretty complicated to understand and thus I couldn't easily get the external binaries in there.
Open for ideas, I'm currently deciding between k0s and k3s and the latter is already supported in NixOS so I also might just go with that

[twz123]

This works for me™:

{ lib, pkgs, ... }:
let
  pname = "k0s";

  owner = "k0sproject";
  repo = pname;
  description = "k0s - The Zero Friction Kubernetes";

  version = "1.27.4+k0s.0";
  hash = "sha256-JmaCRTMU3qsVu/AzyDHpSwv0j9NPxs11WiRbZYqAPHs=";

  # Build a derivation from binary releases hosted on GitHub
  k0s = pkgs.stdenv.mkDerivation {
    name = "${pname}-${version}";
    src = pkgs.fetchurl {
      url = "https://github.com/${owner}/${repo}/releases/download/v${version}/${repo}-v${version}-amd64";
      inherit hash;
    };
    phases = [ "installPhase" ];
    installPhase = ''
      install -m 555 -D -- "$src" "$out"/bin/'${pname}'
    ''; # Shell completions could be added here.

    # Metadata required for a real package
    # meta = with lib; {
    #   inherit description;
    #   license = licenses.asl20;
    #   homepage = "https://k0sproject.io";
    #   platforms = [ "x86_64-linux" ]; # ARM 32/64 binary releases also available.
    # };
  };

  # Some minimal sample config
  k0sConfig = pkgs.writeText "${pname}.json" (builtins.toJSON {
    apiVersion = "k0s.k0sproject.io/v1beta1";
    kind = "ClusterConfig";
    metadata.name = pname;
    spec.network.provider = "kuberouter";
  });
in
{
  # If k0s should be in the PATH:
  # environment.systemPackages = [ k0s ];

  systemd.services.k0scontroller = {
    inherit description;
    documentation = [ "https://docs.k0sproject.io" ];
    path = with pkgs; [
      util-linux # required by kubelet: https://github.com/k0sproject/k0s/issues/3386
    ];
    after = [ "network-online.target" ];
    wants = [ "network-online.target" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      ExecStart = "${k0s}/bin/k0s controller --config=${k0sConfig} --data-dir=/var/lib/k0s --single=true";
    };
  };
}

[twz123]

Btw, I found out that mount is indeed a hard dependency 😪 (#3386)

[twz123]

As a start, I opened a PR to add the k0s binary releases to nixpkgs: NixOS/nixpkgs#258846. That could maybe be the base for a proper k0s NixOS module?

[twz123]

Closing this, as there's no secret sauce to writing a systemd unit for k0s. Just create a simple systemd unit and add the CLI flags that you need for your desired setup to the Exec directive. The documentation on manually running k0s should give some hints: https://docs.k0sproject.io/v1.28.4+k0s.0/k0s-multi-node/

[johbo]

@twz123 just a hint that there might be a misunderstanding: The issue which the discussing people have is not that they don't know how to manage the systemd unit, that's straight forward.

The real issue is that on NixOS you simply cannot write into /etc/systemd/system, this leads to the problem that the calls to k0s install fail which renders also the tooling like k0sctl to fail:

/usr/local/bin/k0s install controller -c /etc/k0s/k0s.yaml 
Error: failed to install k0s service: failed to install service: open /etc/systemd/system/k0scontroller.service: read-only file system

If there would be something like a flag to disable the service part of the installation or to specify the output directory, then it would be straight forward to handle the remaining parts of setting things up.

[twz123]

@twz123 just a hint that there might be a misunderstanding: The issue which the discussing people have is not that they don't know how to manage the systemd unit, that's straight forward.

I agree that the issue digressed a bit. As for the request of the original poster, it was about exactly that: how to manage systemd units manually, especially in the context of NixOS.

The real issue is that on NixOS you simply cannot write into /etc/systemd/system, this leads to the problem that the calls to k0s install fail which renders also the tooling like k0sctl to fail:

/usr/local/bin/k0s install controller -c /etc/k0s/k0s.yaml 
Error: failed to install k0s service: failed to install service: open /etc/systemd/system/k0scontroller.service: read-only file system

If there would be something like a flag to disable the service part of the installation or to specify the output directory, then it would be straight forward to handle the remaining parts of setting things up.

This sounds like a different issue that could be raised against k0sctl and discussed there. If I understand you correctly, you'd like to be able to bypass the installation part of k0sctl and just do the cluster boostrapping? If you're really interested, I'd ask you to describe your use case and how k0sctl could be improved to make it work in a separate issue in the k0sctl repository. We can discuss it further there.

But as mentioned in previous comments, k0sctl may not be the best way to manage k0s on NixOS. It might make more sense to have a NixOS module take care of this.

[johbo]

Good approach! 👍 I'll tweak things on my end a little bit further until I have it working in a PoC fashion and then try to distill out of this a proposal. It's gonna take on my end some time until I get back to this in a fresh issue on k0sctl.

[johbo]

Adding a reference for others who should arrive here.

My first investigations have shown that making things work the k0sctl way is a bit of a rabbit hole. I have put put up a small repository with a Nix flake to get things started in the NixOS way to continue and document my journey. I will get back with a proposal / pull request if this stabilizes and matures enough.

Pointer: https://github.com/johbo/k0s-nix