Podman with `fuse-overlayfs` and `force_mask = "0700"` mangles ownership and permissions inside running containers #22810

jiridanek · 2024-05-26T08:37:08Z

Issue Description

$ cat ~/.config/containers/containers.conf 
[containers]

[engine]

[machine]

[network]
default_rootless_network_cmd="slirp4netns"

[secrets]

[configmaps]

$ cat ~/.config/containers/storage.conf 
[storage]
driver="overlay"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"
force_mask = "0700"

Steps to reproduce the issue

apply the configs from above
run the command from below

Describe the results you received

$ podman system reset
WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all networks
        - all build cache
        - all machines
        - all volumes
        - the graphRoot directory: "/home/jdanek/.local/share/containers/storage"
        - the runRoot directory: "/run/user/1000/containers"
Are you sure you want to continue? [y/N] y
 A "/home/jdanek/.config/containers/storage.conf" config file exists.
Remove this file if you did not modify the configuration.

$ podman run --rm -it registry.fedoraproject.org/fedora bash -c 'useradd pepa; ls -AlFd /home/pepa; chown pepa /home/pepa; ls -AlFd /home/pepa; sudo -u pepa ls /home'
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob 9caf8ba21fce done   | 
Copying config 5858ca0e9f done   | 
Writing manifest to image destination
drwx------. 2 root root 104 May 26 08:29 /home/pepa/
drwx------. 2 pepa nobody 104 May 26 08:29 /home/pepa/
sudo: unable to execute /usr/bin/ls: Permission denied

Describe the results you expected

After I comment out the force_mask = "0700", or delete storage.conf alltogether, I get

$ podman system reset
WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all networks
        - all build cache
        - all machines
        - all volumes
        - the graphRoot directory: "/home/jdanek/.local/share/containers/storage"
        - the runRoot directory: "/run/user/1000/containers"
Are you sure you want to continue? [y/N] y
 A "/home/jdanek/.config/containers/storage.conf" config file exists.
Remove this file if you did not modify the configuration.

$ podman run --rm -it registry.fedoraproject.org/fedora bash -c 'useradd pepa; ls -AlFd /home/pepa; chown pepa /home/pepa; ls -AlFd /home/pepa; sudo -u pepa ls /home'
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob 9caf8ba21fce done   | 
Copying config 5858ca0e9f done   | 
Writing manifest to image destination
drwx------. 2 pepa pepa 104 May 26 08:32 /home/pepa/
drwx------. 2 pepa pepa 104 May 26 08:32 /home/pepa/
pepa

podman info output

$ podman info
host:
  arch: amd64
  buildahVersion: 1.35.4
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 85.42
    systemPercent: 1.81
    userPercent: 12.76
  cpus: 12
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "40"
  eventLogger: journald
  freeLocks: 2048
  hostname: fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    - container_id: 65537
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
  kernel: 6.8.9-300.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 5639622656
  memTotal: 33392324608
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240510.g7288448-1.fc40.x86_64
    version: |
      pasta 0^20240510.g7288448-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 27000832
  swapTotal: 8589930496
  uptime: 145h 13m 23.00s (Approximately 6.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  image-registry.openshift-image-registry.svc:
    Blocked: false
    Insecure: false
    Location: default-route-openshift-image-registry.apps-crc.testing
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: image-registry.openshift-image-registry.svc
    PullFromMirror: ""
store:
  configFile: /home/jdanek/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.force_mask: "0700"
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.13-1.fc40.x86_64
      Version: |-
        fusermount3 version: 3.16.2
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
  graphRoot: /home/jdanek/.local/share/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 400291479552
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/jdanek/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.3
  Built: 1715299200
  BuiltTime: Fri May 10 02:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

I use btrfs on Fedora 40.

I discovered this when exploring around

Add an option to disable enforcing of force_mask='700' for network file systems storage#1839

In the issue I am reporting now, I deliberately did not use any networked filesystem but still got weird permissions/ownership errors.

Additional information

Article https://www.redhat.com/sysadmin/podman-rootless-overlay suggests that fuse-overlayfs is supposed to preserve permissions and ownership in file xattrs.

This issue seems to be similar to a several previous issues

And it might be related to

The text was updated successfully, but these errors were encountered:

jiridanek · 2024-06-12T08:10:24Z

It's quite likely the same issue. I decided to report problems about a single configuration to make the bug report simpler, but I'm convinced that any force_mask that is less restrictive than 0777 will cause problems.

giuseppe · 2024-06-17T12:06:17Z

might be addressed by containers/fuse-overlayfs#422

jiridanek · 2024-06-17T12:13:56Z

The tests in the PR do look promising!

legobeat · 2024-06-19T22:45:49Z

My issue is resolved after a system package upgrade (so might be different, given containers/fuse-overlayfs#422 is not yet released).

$ sudo dnf list --installed | grep -E 'podman|container|overlay'
container-selinux.noarch                    2:2.232.1-1.fc40                  @updates
containerd.io.x86_64                        1.6.33-3.1.fc40                   @docker-ce-stable
containers-common.noarch                    5:0.59.1-2.fc40                   @updates
containers-common-extra.noarch              5:0.59.1-2.fc40                   @updates
fuse-overlayfs.x86_64                       1.13-1.fc40                       @fedora
overlayfs-tools.x86_64                      0-0.9.20200817git77bf3f7.fc40     @fedora
podman.x86_64                               5:5.1.1-1.fc40                    @updates

jiridanek · 2024-06-20T13:32:17Z

My issue is resolved after a system package upgrade (so might be different, given containers/fuse-overlayfs#422 is not yet released).

@legobeat Did you do podman system reset after changing force_mask?

Initially, I just edited config to put force_mask back in, tried the reproducer command, and was about to celebrate, when I realized that it does not come into effect unless container storage is reset! So I did that, and bug started manifesting again.

jdanek@fedora:~$ vim ~/.config/containers/storage.conf 
jdanek@fedora:~$ podman run --rm -it registry.fedoraproject.org/fedora bash -c 'useradd pepa; ls -AlFd /home/pepa; chown pepa /home/pepa; ls -AlFd /home/pepa; sudo -u pepa ls /home'
drwx------. 2 root root 104 Jun 20 13:28 /home/pepa/
drwx------. 2 pepa nobody 104 Jun 20 13:28 /home/pepa/
pepa
jdanek@fedora:~$ podman system reset
WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all networks
        - all build cache
        - all machines
        - all volumes
        - the graphRoot directory: "/home/jdanek/.local/share/containers/storage"
        - the runRoot directory: "/run/user/1000/containers"
Are you sure you want to continue? [y/N] y
 A "/home/jdanek/.config/containers/storage.conf" config file exists.
Remove this file if you did not modify the configuration.
jdanek@fedora:~$ podman run --rm -it registry.fedoraproject.org/fedora bash -c 'useradd pepa; ls -AlFd /home/pepa; chown pepa /home/pepa; ls -AlFd /home/pepa; sudo -u pepa ls /home'
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob 0eb6f43df594 done   | 
Copying config 5c521474f1 done   | 
Writing manifest to image destination
drwx------. 2 root root 104 Jun 20 13:30 /home/pepa/
drwx------. 2 pepa nobody 104 Jun 20 13:30 /home/pepa/
sudo: unable to execute /usr/bin/ls: Permission denied
jdanek@fedora:~$

jiridanek · 2024-11-23T08:58:49Z

My fancy images have fancy file types in them. When I tried this time, I got hit by

Setting shared xattr fails on images with named pipes storage#2174

jiridanek added the kind/bug Categorizes issue or PR as related to a bug. label May 26, 2024

jiridanek changed the title ~~Podman with fuse-overlayfs and force_mask = "0700" set mangles ownership and permissions inside running containers~~ Podman with fuse-overlayfs and force_mask = "0700" mangles ownership and permissions inside running containers May 26, 2024

This comment was marked as resolved.

Sign in to view

legobeat mentioned this issue Jun 17, 2024

Allow setting mountopt for vfs driver containers/storage#1972

Open

jiridanek mentioned this issue Jun 16, 2024

Add an option to disable enforcing of force_mask='700' for network file systems containers/storage#1839

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Podman with `fuse-overlayfs` and `force_mask = "0700"` mangles ownership and permissions inside running containers #22810

Podman with `fuse-overlayfs` and `force_mask = "0700"` mangles ownership and permissions inside running containers #22810

jiridanek commented May 26, 2024 •

edited

Loading

This comment was marked as resolved.

jiridanek commented Jun 12, 2024

giuseppe commented Jun 17, 2024

jiridanek commented Jun 17, 2024

legobeat commented Jun 19, 2024 •

edited

Loading

jiridanek commented Jun 20, 2024

jiridanek commented Nov 23, 2024

Podman with fuse-overlayfs and force_mask = "0700" mangles ownership and permissions inside running containers #22810

Podman with fuse-overlayfs and force_mask = "0700" mangles ownership and permissions inside running containers #22810

Comments

jiridanek commented May 26, 2024 • edited Loading

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

This comment was marked as resolved.

jiridanek commented Jun 12, 2024

giuseppe commented Jun 17, 2024

jiridanek commented Jun 17, 2024

legobeat commented Jun 19, 2024 • edited Loading

jiridanek commented Jun 20, 2024

jiridanek commented Nov 23, 2024

Podman with `fuse-overlayfs` and `force_mask = "0700"` mangles ownership and permissions inside running containers #22810

Podman with `fuse-overlayfs` and `force_mask = "0700"` mangles ownership and permissions inside running containers #22810

jiridanek commented May 26, 2024 •

edited

Loading

legobeat commented Jun 19, 2024 •

edited

Loading