otcore: add debug logging capability for ostree-prepare-root

[ericcurtin]

This commit introduces a new function, proc_cmdline_has_key, to check for specific keys in the kernel command line. Additionally, it adds a debug logging mechanism in ostree-prepare-root.c, which is controlled by the presence of the "ostree-prepare-root.debug" key in the kernel command line.

These changes provide enhanced visibility into the mounting process, aiding in troubleshooting.

[cgwalters]

which is controlled by the presence of the "ostree-prepare-root.debug" key in the kernel command line.

Hmm, this feels pretty ad-hoc. I think it'd be nicer to enable this via environment variables...if systemd had something like systemd.unit.setenv= on the kernel cmdline then it'd be really convenient to use to set debug logging for specific units via the cmdline.

[cgwalters]

These changes provide enhanced visibility into the mounting process, aiding in troubleshooting.

Can you speak a little bit more to what you were troubleshooting?

It may be for example that we could just log at least a few of these key mount invocations by default to stdout (which goes to the journal); then there'd be no need for an opt-in debug invocation.

[ericcurtin]

We could do both maybe? I'll describe in more details tomorrow, I need to head offline it's late. I was assigned to a bug that wasn't a bug today, but I would love to tell bug reporters when they see some file mounting errors in the kernel logs, just to re-run again with ostree-prepare-root.debug karg, just to remind myself what the flow is and to see the userspace logging interleaved with the kernel logging...

[ericcurtin]

Our build can't find composefs for some reason for cs9:

 > [build 2/5] RUN dnf -y install dnf-utils zstd && dnf config-manager --enable crb && dnf builddep -y ostree:
11.85 CentOS Stream 9 - BaseOS                        7.5 kB/s | 1.4 kB     00:00    
12.29 CentOS Stream 9 - BaseOS - Source               1.2 MB/s | 488 kB     00:00    
12.54 CentOS Stream 9 - AppStream                      12 kB/s | 1.5 kB     00:00    
13.21 CentOS Stream 9 - AppStream - Source            2.1 MB/s | 1.2 MB     00:00    
13.97 CentOS Stream 9 - CRB                            14 MB/s | 6.2 MB     00:00    
15.86 CentOS Stream 9 - CRB - Source                  365 kB/s | 169 kB     00:00    
16.53 CentOS Stream 9 - Extras packages - Source       23 kB/s |  13 kB     00:00    
17.63 No matching package to install: 'pkgconfig(composefs)'
17.63 Not all dependencies satisfied
17.64 Error: Some packages could not be found.
------
Containerfile.c9s:2
--------------------
   1 |     FROM quay.io/centos/centos:stream9 as build
   2 | >>> RUN dnf -y install dnf-utils zstd && dnf config-manager --enable crb && dnf builddep -y ostree
   3 |     COPY . /build
   4 |     WORKDIR /build
--------------------
ERROR: failed to solve: process "/bin/sh -c dnf -y install dnf-utils zstd && dnf config-manager --enable crb && dnf builddep -y ostree" did not complete successfully: exit code: 1
Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c dnf -y install dnf-utils zstd && dnf config-manager --enable crb && dnf builddep -y ostree" did not complete successfully: exit code: 1

[ericcurtin]

The bug logged was around kernel error messages being logged with overlayfs/composefs (it doesn't exist in the latest upstream of composefs and the kernel, so it makes sense to ignore the error logs until the changes trickle down)... But... The mount call was returning success, the error logs were coming from the kernel logs... But then the question is which overlay /etc or /usr and what were the options passed? Sure one can read the code, but it's also nice to tell the person re-run with "ostree-prepare-root.debug".

ostree-prepare-root is a frequent component we see bug reports on, in some cases people can readily reproduce, would be nice to get the complete sequence of mounts interleaved with kernel logs.

Normally we can do this with strace etc. but when it's an initramfs component it's hard to use tracing tools.

[cgwalters]

Our build can't find composefs for some reason for cs9:

Yeah, it's not part of appstream or crb so...yes, it's pretty maddening, but it is intentional at the current time. In c9s one can pull from the public-but-not-mirrored buildroot repos, or rebuild things from source.

[cgwalters]

The bug logged was around kernel error messages being logged with overlayfs/composefs

Yes, there are a few copies of bugs related to this (e.g. https://gitlab.com/fedora/bootc/base-images/-/issues/6 also) - I think containers/composefs@f884f57 may fix this.

[cgwalters]

But then the question is which overlay /etc or /usr and what were the options passed?

Right, I get the use case. But sorry, I think this patch won't help, because the relevant mount here is happening in libcomposefs as part of lcfs_mount_image().

So my inclination here is to close this.