apply: Don't use staged deployments when /boot is automounted #301
Conversation
|
I haven't tested this yet but wanted to get it out there. I have another version that instead checks if |
dbnicholson
force-pushed
the
T33136-no-staged-deploy-autofs
branch
from
b6fc706 to
14c6350
Compare
On some hosts, systemd sets up `/boot` as an autofs automount. Unfortunately, this configuration has some bugs that make usage of `/boot` less than robust. Add a method to see if `/boot` is an automount. This uses libmount to parse `/proc/self/mountinfo` and check if the first mountpoint for the sysroot's boot directory has the `autofs` filesystem type. libmount version 2.24 was chosen as a minimum because that's when `libmnt_table` and `libmnt_cache` reference counting was added. https://phabricator.endlessm.com/T33136
The ostree staged deployment process works by waiting until shutdown to swap the `/boot` symlinks to make the new deployment the default. However, when `/boot` is the EFI System Partition and there's no `fstab` entry, `systemd-gpt-auto-generator` sets up an automount so that the VFAT filesystem is only exposed when needed. Unfortunately, there are 2 bugs that make this process very fragile: * Once a systemd automount unit is scheduled to be stopped, it ignores notifications from autofs that the target filesystem should be mounted. Therefore, if `/boot` isn't mounted when shutdown begins, `ostree admin finalize-staged` will fail. See systemd/systemd#22528. * autofs is not mount namespace aware, so it will begin the expiration timer for a mount unit unless a process in the root namespace is keeping it active. Since `ostree admin finalize-staged` is run from a mount namespace (either via systemd or its own to ensure `/sysroot` and `/boot` are mounted read-write), the automount daemon (systemd) will try to unmount the filesystem if it expires during this process. See https://bugzilla.redhat.com/show_bug.cgi?id=2056090. Therefore, if `/boot` is an autofs filesystem, use a full deployment instead of a staged deployment. Since systems with an automounted `/boot` are not common, we want to retain the benefit of staged deployments for more normal systems. See ostreedev/ostree#2543 for potential future fixes in ostree. https://phabricator.endlessm.com/T33136
dbnicholson
force-pushed
the
T33136-no-staged-deploy-autofs
branch
from
14c6350 to
a19821a
Compare
|
I tested this on a PAYG system and could see that it did a finalized deployment instead of a staged deployment. I ended up splitting out the method to the util library so I could unit test it. |
The new changes look good to me -- not sure if you want an extra pair of eyes on this, since this is critical code. In any case, master PAYG systems are currently failing to boot, so we won't be able to test it on ostrees until that is fixed. |
|
I think it's good, but if @pwithnall has a moment to review the util side and tests that would be nice. I'll wait a day and merge if he doesn't. It's still useful master to check that normal Endless systems still get staged deployments. |
|
I think @pwithnall is too deep in gnome-software land to take a look, so I'm going to go ahead and merge. |
The ostree staged deployment process works by waiting until shutdown to
swap the
/bootsymlinks to make the new deployment the default.However, when
/bootis the EFI System Partition and there's nofstabentry,
systemd-gpt-auto-generatorsets up an automount so that theVFAT filesystem is only exposed when needed.
Unfortunately, there are 2 bugs that make this process very fragile:
Once a systemd automount unit is scheduled to be stopped, it ignores
notifications from autofs that the target filesystem should be
mounted. Therefore, if
/bootisn't mounted when shutdown begins,ostree admin finalize-stagedwill fail. SeeExecStop failures with Automount units systemd/systemd#22528.
autofs is not mount namespace aware, so it will begin the expiration
timer for a mount unit unless a process in the root namespace is
keeping it active. Since
ostree admin finalize-stagedis run from amount namespace (either via systemd or its own to ensure
/sysrootand
/bootare mounted read-write), the automount daemon (systemd)will try to unmount the filesystem if it expires during this process.
See https://bugzilla.redhat.com/show_bug.cgi?id=2056090.
Therefore, if
/bootis an autofs filesystem, use a full deploymentinstead of a staged deployment. Since systems with an automounted
/bootare not common, we want to retain the benefit of stageddeployments for more normal systems. See
ostreedev/ostree#2543 for potential future
fixes in ostree.
https://phabricator.endlessm.com/T33136