Weird error message after reboot: Error: current system boot ID differs from cached boot ID

[groovyman]

Dear friends,

i am running a Fedora F40 Server in order to run my apps, where some of them use podman to run rootless container.
In order to archive this, i created some technical users, that acts like domains, that run in turn some container apps.

One of these technical users shows a strange behaviour after some updates. Whenever i reboot my sever and i login
on my technical user i get the following message, whenever i call podman:

podman ps
Error: current system boot ID differs from cached boot ID; an unhandled reboot has occurred. Please delete directories "/run/user/1002/containers" and "/Home/techlxoffice/rteDataSpace/podmanTMP" and re-run Podman

Then i go into the podmanTMP directory and i see a couple of files:

$ ls -l /Home/techlxoffice/rteDataSpace/podmanTMP
insgesamt 4
-rw-r--r--. 1 techlxoffice techlxoffice  37  3. Jul 21:55 alive
-rw-r--r--. 1 techlxoffice techlxoffice   0  3. Jul 21:55 alive.lck
drwxr-x---. 1 techlxoffice techlxoffice   0  4. Jul 01:26 exits
drwxr-x---. 1 techlxoffice techlxoffice 128  3. Jul 21:55 persist
drwx------. 1 techlxoffice techlxoffice   0  3. Jul 21:55 rp

So when i remove all these files, podman is running, and the systemd --user is also able to run the application, that normally lauch up, when the host is starting.

Do you have an idea, what went wrong. This configuration suddently fails, maybe due to a update, that took place a week ago.

[Luap99]

https://github.com/containers/podman/releases/tag/v5.1.0

Podman now detects unhandled system reboots and advises the user on proper mitigations.

In general the podman tmpdir and rundir must be on a tmpfs or deleted on boot in order to allow for podman to function properly

[hakong]

Sounds contradictory to what /usr/lib/tmpfiles.d/podman.conf contains.

user@workstation:/usr/lib/tmpfiles.d$ rpm -qf podman.conf
podman-5.2.2-9.el9_5.x86_64

user@workstation:/usr/lib/tmpfiles.d$ cat podman.conf
# /tmp/podman-run-* directory can contain content for Podman containers that have run
# for many days. This following line prevents systemd from removing this content.
x /tmp/podman-run-*
x /tmp/storage-run-*
x /tmp/containers-user-*
x /tmp/run-*/libpod
D! /var/lib/containers/storage/tmp 0700 root root
D! /run/podman 0700 root root
D! /var/lib/cni/networks
# Remove /var/tmp/container_images* podman temporary directories on each
# boot which are created when pulling or saving images.
R! /var/tmp/container_images*

We've just had a lot of our services go down due to this issue. We had to put a workaround in place to remove these files on boot.

I think podman should be able to function properly if these temp files are still in place.

[groovyman]

Salut @Luap99

so the best strategy is to ignore the --tmpdir argument, when building the image and let podman decide to choose the correct default tmp-directory for this step.

Would that be the right conclusion from your answer?

Thank you for your help!

[Luap99]

you can use --tmpdir but you should make sure that this directory is on a tmpfs or deleted on boot before you launch podman.

[maweil]

Thank you for your help, this hint helped me a lot.

In case others have the same issue even without using the --tmpdir argument:
I also ran into this problem on my Almalinux 9 system after it upgraded to 9.5 (which includes Podman 5.2.2). At least on my machine the issue was that /tmp was not mounted on a tmpfs. I had to enable tmp.mount first.

[beeritis]

Have the same issue today as podman was upgraded to 5.2.2.

This broke some of our containers that run as a systemd user service.
We initially downgraded back to podman 4.0.2 (previously installed version before servers got patched) as that doesn't have this problem.

The current workaround is to remove these files before the service starts .
For example : ExecStartPre=/usr/bin/rm -rf /tmp/containers-user-<id>/containers /tmp/podman-run-<id>/libpod/tmp

Part of our problem is that /tmp exists as an xfs file system (don't ask) so the files persist upon reboot.
Having said that , I don't know what is classed as a "unhandled system reboot", given that reboots are usually a result of patching which I wouldn't class as "unhandled". A flag to disable this feature would be useful or if there is a cleaner way to start containers /pods on reboot that would be better.

[beeritis]

This works perfectly ! Thank you :) .

Means I can remove my temporary workaround of using ExecStartPre= in systemd units :) .

[shay1197]

I had also considered doing this but any configuration changes seem to get overwritten when podman next gets upgraded. (i.e podman.conf gets replaced). If podman respects it, a configuration file in ~/.config/user-tmpfiles.d/*.conf might take precedence over the global podman.conf so the configuration persists but that would need testing.

@beeritis and anyone else really According to the tmpfiles.d manual page

Files in /etc/tmpfiles.d are reserved for
the local administrator, who may use this logic to override the
configuration files installed by vendor packages

To resolve this issue we copied the settings from our /usr/lib/tmpfiles.d/podman.conf to /etc/tmpfiles.d/podman.conf and appended at the bottom

R! /tmp/storage-run-*/containers/
R! /tmp/storage-run-*/libpod/tmp/

This should persist if podman is updated and can happily report our service now withstands a reboot.

Hi,
I tried to configure this in the image I am building, but the path /etc/tmpfiles.d does not exist.
How can I handle this in the image build? :\

[beeritis]

@shay1197 - This isn't within an image, this is on the host itself (i.e. where podman is installed).

[shay1197]

Yes, but I am running on Kubernetes, so I create the path and file during the image build process.
when the pod is up I can see the folder with this file:

x /tmp/podman-run-*
R! /tmp/storage-run-*
x /tmp/containers-user-*
x /tmp/run-/libpod
D! /var/lib/containers/storage/tmp 0700 root root
D! /run/podman 0700 root root
D! /var/lib/cni/networks
R! /var/tmp/container_images

[beeritis]

Oh , interesting use case!

Can't say I've run podman within Kubernetes before :). I would assume you still get a unique boot id when the pod is run?

Sounds a bit tricky as tmpfiles.d is related to the systemd-tmpfiles/ systemd-tmpfiles-clean.service which clears out the files and I don't recall an instance where systemd is being utilised within a container.

If a boot ID doesn't change then it might not be an issue but otherwise someone else can hopefully answer that question

[mhrivnak]

Anyone have a current reproducer for this? I have an older centos stream 9 system that I installed last summer; it exhibits this behavior for one user I created back then, but I'm not able to reproduce it with a different user on that system now nor on a freshly-installed CS9 system now. I used the same ansible role to create a system user and install a rootless containerized service in all cases.

I noticed that when running podman system info with the affected user, I see:

runRoot: /tmp/containers-user-989/containers

Whereas if I try to reproduce with a different user on the same system, or a fresh system, I see:

runRoot: /run/user/982/containers

In no case is there a ~/.config/containers/storage.conf file. I'm not able to figure out why the runroot is different for that one old user, nor where the setting is stored.

@mheon does any of this ring a bell? I see that you added #22278 some time ago...

[mheon]

This is indeed #22278 working as intended. Podman detected that your runRoot is not being cleared after a reboot, indicating it's not on a tmpfs. You may have started the user with a /tmp/ runRoot from a session that systemd didn't recognize, hence /run/user/$UID not being mounted. The correct fix is to either set up that runRoot to be automatically removed on reboot (I believe systemd-tmpfiles can do this) or, if you don't mind losing the containers/images/etc of the user, do a podman system reset and then run a container afterwards while making sure that /run/user/$UID exists and is accessible for the user in question (which may require a loginctl enable-linger for the user).

[mhrivnak]

run a container afterwards while making sure that /run/user/$UID exists and is accessible for the user in question (which may require a loginctl enable-linger for the user).

Ah, so if I ran a container with that user prior to enabling lingering, that might result in /tmp being used as the runRoot?

[mheon]

Yes, we cache the paths in use in the database to make sure they don’t change as that can break things.

…

On Fri, Jan 3, 2025 at 10:51 Michael Hrivnak ***@***.***> wrote: run a container afterwards while making sure that /run/user/$UID exists and is accessible for the user in question (which may require a loginctl enable-linger for the user). Ah, so if I ran a container with that user prior to enabling lingering, that might result in /tmp being used as the runRoot? — Reply to this email directly, view it on GitHub <#23193 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCFJO4MJLEQBODMCRQ32I2WWXAVCNFSM6AAAAABKK4NEH6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNZSGY4TMNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>