singularity build --oci fails when $HOME/.singularity/singularity-buildkitd is on NFS

[nathanweeks]

Before you report an issue...

• Have you searched for a duplicate report?
• Have you replicated the problem on the latest release?
• Have you checked the latest documentation?
• Are you using an up-to-date system?

Version of Singularity

What version of Singularity are you using? Run:

$ singularity --version

singularity-ce version 4.2.1-1

Describe the bug

singularity build --oci fails with a failed to Lchown... operation not permitted error when .singularity/singularity-buildkitd (or more specifically, ~/.singularity/singularity-buildkitd/runc-overlayfs) is on an NFS file system (a common scenario for user home directories on an HPC cluster).

To Reproduce

$ singularity.sh build --oci test.oci-sif Dockerfile
INFO:    Did not find usable system buildkitd daemon. Starting built-in singularity-buildkitd.
INFO:    singularity-buildkitd: running server on /tmp/run-user-1000/singularity-buildkitd/singularity-buildkitd-2176343.sock
[+] Building 2.2s (5/5)
[+] Building 2.3s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                    0.1s
 => => transferring dockerfile: 145B                                                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/ubuntu:24.04                                                                                                         0.3s
 => [internal] load .dockerignore                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                         0.0s
 => [1/2] FROM docker.io/library/ubuntu:24.04@sha256:99c35190e22d294cdace2783ac55effc69d32896daaa265f0bbedbcde4fbe3e5                                                   1.4s
 => => resolve docker.io/library/ubuntu:24.04@sha256:99c35190e22d294cdace2783ac55effc69d32896daaa265f0bbedbcde4fbe3e5                                                   0.1s
 => => extracting sha256:ff65ddf9395be21bfe1f320b7705e539ee44c1053034f801b1a3cbbf2d0f4056                                                                               1.4s
 => ERROR [2/2] RUN apt update                                                                                                                                          0.0s
------
 > [2/2] RUN apt update:
------
INFO:    Terminating singularity-buildkitd (PID 2176372)
FATAL:   while building from dockerfile: failed to solve: failed to compute cache key: mount callback failed on /tmp/run-user-1000/containerd-mount901823101: failed to Lchown "/tmp/run-user-1000/containerd-mount901823101/etc/gshadow" for UID 0, GID 42: lchown /tmp/run-user-1000/containerd-mount901823101/etc/gshadow: operation not permitted

Expected behavior

I expect the build to succeed

OS / Linux Distribution

Which Linux distribution are you using? Is it up-to-date?

Rocky Linux 8.9

Installation Method

Singularity 4.2.1 was installed via RPM from https://github.com/sylabs/singularity/releases/tag/v4.2.1

Additional context

Symlinking ~/.singularity/singularity-buildkit to a node-local storage path (or more specifically, ~/.singularity/singularity-buildkit/runc-overlayfs) to a node-local storage path works around the error, and allows singularity build --oci to succeed.

Requiring the users to manually create a symlink to node-local storage isn't very user-friendly (especially if the target node-local directory path needs to be created manually as well). Is there a way to configure just the ~/.singularity/singularity-buildkit path (e.g., environment variable or config file)?

I'm aware of the SINGULARITY_CONFIGDIR environment variable (which isn't documented in the SingularityCE 4.2 User Guide, though it should be?), but that sets the path of the .singularity directory, which is not the intention in this case (e.g., it contains other state the user probably doesn't want on ephemeral/node-local storage).

Alternatively, if the contents of ~/.singularity/singularity-buildkitd instead went to the same directory as the singularity-buildkitd-<pid>.sock file (or perhaps just the runc-overlayfs subdirectory, if that wouldn't break anything?), that would seem to resolve the issue as well.

[dtrudg]

Hmmm - yes, its unfortunate this happens due to lack of subuid support on the NFS filesystem.

There isn't a way to configure the path for singularity-buildkit data separately, yet. Likely we do want to add a flag / configuration option.

We could perhaps look at how podman has an option (to address the same underlying issue), that allows a path incorporating $HOME/$UID/$USER

https://www.redhat.com/en/blog/nfs-rootless-podman

[nathanweeks]

Hmmm - yes, its unfortunate this happens due to lack of subuid support on the NFS filesystem.

Thanks for the explanation. I would surmise that even if NFS (or other network filesystem) could support subuids, it might be unreliable to run concurrent instances of singularity-buildkitd on different nodes?

We could perhaps look at how podman has an option (to address the same underlying issue), that allows a path incorporating $HOME/$UID/$USER

https://www.redhat.com/en/blog/nfs-rootless-podman

Perhaps podman's support for using $XDG_DATA_HOME/containers/storage if $XDG_DATA_HOME is set could be mimicked by SingularityCE (e.g., $XDG_DATA_HOME/singularity/singularity-buildkitd, where XDG_DATA_HOME could be set to a path to node-local storage by a user or sysadmin)? FWIW, NERSC's podman-hpc wrapper defaults to /tmp/{uid}_hpc on compute nodes.

[dtrudg]

Hmmm - yes, its unfortunate this happens due to lack of subuid support on the NFS filesystem.

Thanks for the explanation. I would surmise that even if NFS (or other network filesystem) could support subuids, it might be unreliable to run concurrent instances of singularity-buildkitd on different nodes?

We could perhaps look at how podman has an option (to address the same underlying issue), that allows a path incorporating $HOME/$UID/$USER
https://www.redhat.com/en/blog/nfs-rootless-podman

Perhaps podman's support for using $XDG_DATA_HOME/containers/storage if $XDG_DATA_HOME is set could be mimicked by SingularityCE (e.g., $XDG_DATA_HOME/singularity/singularity-buildkitd, where XDG_DATA_HOME could be set to a path to node-local storage by a user or sysadmin)? FWIW, NERSC's podman-hpc wrapper defaults to /tmp/{uid}_hpc on compute nodes.

I think using $XDG_DATA_HOME should probably only be implemented as part of a proper rewrite of all ~/.singularity usage into correct XDG locations. It would be strange to honor XDG_DATA_HOME for one thing, but still not honor the XDG defaults and env vars for config, non-buildkitd cache etc.

We'll try and address the general buildkitd issue via a configuration option / env-var / flag when time allows. Proper XDG directory support may be something for the next major version.

[dtrudg]

@nathanweeks - we have some code elsewhere to detect filesystems that aren't suitable for some purposes and direct things to $TMPDIR. I could re-use this approach here, avoiding any configuration or provisioning of directories... but using TMPDIR would mean no persistence of the buildkit cache between builds, and assumes $TMPDIR is local.

Any thoughts?

[nathanweeks]

Uncached builds are better than no builds, and it's usually safe to assume $TMPDIR is local.

A few minor questions:

1. Could a warning that the build is uncached be emitted?
2. If $TMPDIR isn't local, would a helpful error message be displayed?
3. If $TMPDIR is unset, is there a fallback to /tmp (by default)?

[nathanweeks]

3. If $TMPDIR is unset, is there a fallback to /tmp (by default)?

I suppose $TMPDIR refers to SINGULARITY_TMPDIR > TMPDIR > /tmp for temporary folders?
https://docs.sylabs.io/guides/4.2/user-guide/build_env.html#temporary-folders

[dtrudg]

Uncached builds are better than no builds, and it's usually safe to assume $TMPDIR is local.

A few minor questions:

1. Could a warning that the build is uncached be emitted?

Definitely - I would add a warning.

2. If $TMPDIR isn't local, would a helpful error message be displayed?

Yes, we can do the same check on $TMPDIR as for the default buildkit directory location.

3. If $TMPDIR is unset, is there a fallback to /tmp (by default)?

As you note elsewhere, $SINGULARITY_TMPDIR > $TMPDIR > /tmp

[dtrudg]

@nathanweeks - I've had a go at this today, and the fallback to a tempDir works. The challenge is getting cleanup to fire reliably and correctly in all circumstances given how things are split across a buildkit daemon and the singularity binary, and partly due to how the buildkit daemon starts up / exits.

Will get there... but this is probably going to be something I have to come back to in the near future due to some other work.

[dtrudg]

The PR at #3407 is the 'proof-of-concept' for a fallback... without the work done on rewriting the startup / exit handling for singularity-buildkitd yet.

Might yet be the case that the TMPDIR handling moves into the client if that's easier, but I'd like to keep it in the daemon if possible. Will probably get a bit more time on this toward the end of next week.