`podman run ... nixos/nix nix-build ...` fails with "error: renaming..." on kernel overlayfs #13507

voidus · 2022-03-13T23:54:25Z

voidus
Mar 13, 2022

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running a specific container command gives an error if using overlayfs. I've tried to find someone with knowledge about the application in question at https://discourse.nixos.org/t/weird-error-when-running-nixos-nix-under-podman/18056 before reporting this, but nobody stepped up.

Steps to reproduce the issue:

As a user with kernel overlayfs (which afaict is the default if no storage.conf is present)

> podman run --rm -ti nixos/nix nix-build -E \
    "builtins.fetchTarball { name = \"nixpkgs-src\"; url = \"https://github.com/NixOS/nixpkgs/archive/59ee4cee790d077072dc1392e94fa6ac0099760d.tar.gz\"; sha256 = \"1k7qkpr7ay4qvkzmljl1kalpvj28ni36pmyy15nyfylrb18q3d9b\"; }"

Describe the results you received:

error: renaming '/nix/store/add-1-2/x' to '/nix/store/9liwm7rnx4357qzgz3bk8v37nkbsj07z-nixpkgs-src'

Describe the results you expected:

error: expression does not evaluate to a derivation (or a set or list of those)

Additional information you deem important (e.g. issue happens only occasionally):
Doesn't happen as root or with fuse-overlayfs.
I configured fuse-overlayfs by adding the following ~/.config/containers/storage.conf and podman system reset:

[storage]
driver = "overlay"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

I feel like straceing could be helpful, but when I tried I wasn't able to spot anything.

Output of podman version:

Client:       Podman Engine
Version:      4.0.1
API Version:  4.0.1
Go Version:   go1.17.8
Git Commit:   c8b9a2e3ec3630e9172499e15205c11b823c8107
Built:        Sun Mar 13 12:44:23 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.0-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c'
  cpus: 6
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: myHostName
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.16.14-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 10899664896
  memTotal: 16622039040
  networkBackend: cni
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.4.3-1
    path: /usr/bin/crun
    version: |-
      crun version 1.4.3
      commit: 61c9600d1335127eba65632731e2d72bc3f0b9e8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 19327348736
  swapTotal: 19327348736
  uptime: 9m 23.46s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/myusername/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.8.2-1
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.8.2
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
  graphRoot: /home/myusername/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: f2fs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/myusername/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.1
  Built: 1647171863
  BuiltTime: Sun Mar 13 12:44:23 2022
  GitCommit: c8b9a2e3ec3630e9172499e15205c11b823c8107
  GoVersion: go1.17.8
  OsArch: linux/amd64
  Version: 4.0.1

This is with fuse, here's the diff after removing storage.conf and running podman system reset:

--- with_fuse	2022-03-14 00:36:14.128811110 +0100
+++ without_fuse	2022-03-14 00:36:28.738978395 +0100
@@ -34,7 +34,7 @@
   kernel: 5.16.14-arch1-1
   linkmode: dynamic
   logDriver: journald
-  memFree: 10694381568
+  memFree: 11605184512
   memTotal: 16622039040
   networkBackend: cni
   ociRuntime:
@@ -68,7 +68,7 @@
       libseccomp: 2.5.3
   swapFree: 19327348736
   swapTotal: 19327348736
-  uptime: 10m 28.02s
+  uptime: 10m 42.64s
 plugins:
   log:
   - k8s-file
@@ -90,24 +90,16 @@
     running: 0
     stopped: 0
   graphDriverName: overlay
-  graphOptions:
-    overlay.mount_program:
-      Executable: /usr/bin/fuse-overlayfs
-      Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.8.2-1
-      Version: |-
-        fusermount3 version: 3.10.5
-        fuse-overlayfs: version 1.8.2
-        FUSE library version 3.10.5
-        using FUSE kernel interface version 7.31
+  graphOptions: {}
   graphRoot: /home/myusername/.local/share/containers/storage
   graphStatus:
     Backing Filesystem: f2fs
-    Native Overlay Diff: "false"
+    Native Overlay Diff: "true"
     Supports d_type: "true"
     Using metacopy: "false"
   imageCopyTmpDir: /var/tmp
   imageStore:
-    number: 1
+    number: 0
   runRoot: /run/user/1000/containers
   volumePath: /home/myusername/.local/share/containers/storage/volumes
 version:

Package info (e.g. output of rpm -q podman or apt list podman):

Current arch as of 2022-03-13

> pacman -Qi podman
Name            : podman
Version         : 4.0.1-2
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/podman
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : conmon  containers-common  crun  iptables  libdevmapper.so=1.02-64  libgpgme.so=11-64  libseccomp.so=2-64  slirp4netns
Optional Deps   : apparmor: for AppArmor support
                  btrfs-progs: support btrfs backend devices [installed]
                  catatonit: --init flag support
                  netavark: for a new container-network-stack implementation
                  podman-compose: for docker-compose compatibility [installed]
                  podman-docker: for Docker-compatible CLI
Required By     : podman-compose
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 79,67 MiB
Packager        : David Runge <[email protected]>
Build Date      : So 13 Mär 2022 12:44:23 CET
Install Date    : Mo 14 Mär 2022 00:17:00 CET
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes, 4.0.1 is even more current than what the readme says D:

There's an entry about overlayfs but the fact that strace doesn't report the mknod system call makes me think it's not that:

> podman run --rm -ti --cap-add=SYS_PTRACE nixos/nix
bash-4.4# strace -e mknod nix-build -E "builtins.fetchTarball { name = \"nixpkgs-src\"; url = \"https://github.com/NixOS/nixpkgs/archive/59ee4cee790d077072dc1392e94fa6ac0099760d.tar.gz\"; sha256 = \"1k7qkpr7ay4qvkzmljl1kalpvj28ni36pmyy15nyfylrb18q3d9b\"; }"
error: renaming '/nix/store/add-23-2/x' to '/nix/store/9liwm7rnx4357qzgz3bk8v37nkbsj07z-nixpkgs-src'
+++ exited with 1 +++
bash-4.4# strace -e mknod mknod bar c 1 3
mknod("bar", S_IFCHR|0666, makedev(0x1, 0x3)) = -1 EPERM (Operation not permitted)
mknod: bar: Operation not permitted
+++ exited with 1 +++

Additional environment details (AWS, VirtualBox, physical, etc.):
Desktop PC on arch linux with nix installed, but I'm pretty confident my system nix doesn't interact with this in any way.

Answered by giuseppe

Mar 15, 2022

I've converted the issue into a discussion, since there is nothing we can do from Podman.

It is a known and documented behavior in overlayfs: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html?highlight=overlayfs#renaming-directories

The best would be for nixos to handle the EXDEV as mv does and fallback to manually copy the directory.

View full answer

giuseppe · 2022-03-14T12:02:49Z

giuseppe
Mar 14, 2022
Maintainer

That is expected. mknod is a privileged operation and cannot be used by unprivileged users, even in a user namespace.

mknod can be used only when running as root on the host.

0 replies

voidus · 2022-03-14T13:06:19Z

voidus
Mar 14, 2022
Author

I specifically addressed this

 strace **doesn't** report the mknod system call

@giuseppe please reopen this issue

Also, it works as a user when using fuse-overlayfs, so I'm pretty sure that's not it

0 replies

giuseppe · 2022-03-14T13:57:11Z

giuseppe
Mar 14, 2022
Maintainer

sorry. Do you see any EXDEV error coming from rename?

0 replies

voidus · 2022-03-14T14:29:13Z

voidus
Mar 14, 2022
Author

No worries, I know it's sometimes hard to triage issues.

Sadly, I'm not sure what nix does internally. I've created the thread on the nix discourse to debug this, but nobody could offer any insight. I'm tempted to create a matching issue in the nix github repo and cross-link this one.

0 replies

voidus · 2022-03-15T02:49:53Z

voidus
Mar 15, 2022
Author

EXDEV was a great hint. I looked through the 180MB strace log and found this:

rename("/nix/store/add-36-2/x", "/nix/store/9liwm7rnx4357qzgz3bk8v37nkbsj07z-nixpkgs-src") = -1 EXDEV (Invalid cross-device link)

I found this patch referenced by this arch forum thread, but a) I'm on kernel 5.16.14-arch1-1 and this patch is ancient and b) it still happens after echo N > /sys/module/overlay/parameters/metacopy.

0 replies

giuseppe · 2022-03-15T08:05:54Z

giuseppe
Mar 15, 2022
Maintainer

I've converted the issue into a discussion, since there is nothing we can do from Podman.

It is a known and documented behavior in overlayfs: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html?highlight=overlayfs#renaming-directories

The best would be for nixos to handle the EXDEV as mv does and fallback to manually copy the directory.

1 reply

voidus Mar 15, 2022
Author

Thanks for the guidance, I'm sure you've been getting variations of this regularly 😶

voidus · 2022-03-15T13:19:38Z

voidus
Mar 15, 2022
Author

I created NixOS/nix#6262 to follow up

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`podman run ... nixos/nix nix-build ...` fails with "error: renaming..." on kernel overlayfs #13507

{{title}}

Replies: 7 comments 1 reply

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

podman run ... nixos/nix nix-build ... fails with "error: renaming..." on kernel overlayfs #13507

voidus Mar 13, 2022

Replies: 7 comments · 1 reply

giuseppe Mar 14, 2022 Maintainer

voidus Mar 14, 2022 Author

giuseppe Mar 14, 2022 Maintainer

voidus Mar 14, 2022 Author

voidus Mar 15, 2022 Author

giuseppe Mar 15, 2022 Maintainer

voidus Mar 15, 2022 Author

voidus Mar 15, 2022 Author

`podman run ... nixos/nix nix-build ...` fails with "error: renaming..." on kernel overlayfs #13507

voidus
Mar 13, 2022

Replies: 7 comments 1 reply

giuseppe
Mar 14, 2022
Maintainer

voidus
Mar 14, 2022
Author

giuseppe
Mar 14, 2022
Maintainer

voidus
Mar 14, 2022
Author

voidus
Mar 15, 2022
Author

giuseppe
Mar 15, 2022
Maintainer

voidus Mar 15, 2022
Author

voidus
Mar 15, 2022
Author