bcachefs support #84

alexminder · 2024-10-15T07:10:20Z

Add bcachefs rootfs support please.

bcachefs have features wich can be used in initrd:

multiple devices, mount looks like /dev/sdc:/dev/sdb:/dev/sda on MOUNTPOINT type bcachefs;
encryption as LUKS alternative;
subvolumes, snapshots (boot from another root snapshot or subvolume)

The text was updated successfully, but these errors were encountered:

alexminder · 2024-10-15T07:19:02Z

There is interesting hook for dracut to checking, if bcachefs device complete:

https://github.com/kode54/dracut-hook-bcachefs/tree/main

desultory · 2024-10-15T15:16:34Z

I briefly looked into this, I think the main issue I found was the source device notation bcachefs uses. Do you know if that's used by any other system? I'm wondering if I should make it more generic or make a separate parsing method for bcachefs. I've found most common filesystems use slaves in the /sys info for the device.

https://github.com/desultory/ugrd/blob/main/src/ugrd/fs/mounts.py#L215

It does simple parsing by reading /proc/mounts and splitting it, it will see /dev/sdc:/dev/sdb:/dev/sda as the "device" and im not sure if that will potentially break anything. If bcachefs has userspace stuff which just handles this, it's not a big deal, but currently ugrd tries to resolve UUID information (by default) to be later used to mount the correct device. At the least this would have to be disabled for bcachefs, especially with multidevice.

With btrfs multidevice, you can just btrfs device scan then mount one of them by uuid, and it mounts the whole set. I don't really see how that module handles encryption or multidevice stuff, unless the bcachefs tools handle that transparently. https://github.com/kode54/dracut-hook-bcachefs/blob/main/bcachefs_finished.sh#L13

alexminder · 2024-10-16T15:35:44Z

Do you know if that's used by any other system?

As I know, no other.

With btrfs multidevice, you can just btrfs device scan then mount one of them by uuid, and it mounts the whole set.

From version 2.39 of libblkid it is possible to use UUID="", but with both of source notation systemd (mount generator) have problems.

I've found most common filesystems use slaves in the /sys info for the device.

Yes, /sys/fs/bcachefs/UUID/dev-N are presented as slaves and inside this dirs links exists to real block devices.

The command bcachefs show-super <DEVICE> on any bcachefs device shows all the information about FS and all the slaves (how many slaves, external, internal UUIDs).

If bcachefs has userspace stuff which just handles this,

bcachefs userspace utility from bcachefs-tools can do this.

I don't really see how that module handles encryption or multidevice stuff, unless the bcachefs tools handle that transparently.

This module don't check encryption. It is just try to mount it readonly. If one of slave not available mount will fail and process returns to initqueue for waiting. When udev enumerate all bcachefs type devices and bcachefs_finished.sh mounts success, then unmount and boot continues where itis possibe dracut to mount with UUID or /dev/sdc:/dev/sdb:/dev/sda as source.
For encrypted FS need bcachefs unlock before mounting.

desultory · 2024-10-16T15:41:15Z

Thanks for that info, I will try to test this soon.

One thing to consider is that ugrd does not use udev, so it would have to do something to manually bring devices online at boot time. It pulls blkid (currently 2.39.4 on gentoo), so using UUID notation for the filesystem may just be the most straightforward method.

If you want to test, you can add bcachefs to binaries in the config and set the "recovery" kernel cmdline arg to get a shell when it fails to boot.

If bcachefs says it's encrypted somewhere in the fs info under sys, it should be easy to make a simple hook that runs bcachefs unlock to handle decryption, and the typical uuid/partuuiid/label based mounting should be able to handle it from there.

alexminder · 2024-10-18T14:48:18Z

currently I can not build initramfs with ugrd:

ERROR    | Failed to run command: blkid /dev/sdc:/dev/sda
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/ugrd/main.py", line 158, in main
    generator.build()
  File "/usr/lib/python3.12/site-packages/ugrd/initramfs_generator.py", line 87, in build
    self.run_build()
  File "/usr/lib/python3.12/site-packages/ugrd/initramfs_generator.py", line 225, in run_build
    self.run_hook(task, force_exclude=True)
  File "/usr/lib/python3.12/site-packages/ugrd/initramfs_generator.py", line 141, in run_hook
    if function_output := self.run_func(function, *args, **kwargs):
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ugrd/initramfs_generator.py", line 103, in run_func
    if function_output := function(self):
                          ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/zenlib/util/dict_check.py", line 36, in _contains
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/zenlib/util/dict_check.py", line 36, in _contains
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ugrd/fs/mounts.py", line 491, in autodetect_root
    get_blkid_info(self, root_dev)
  File "/usr/lib/python3.12/site-packages/zenlib/util/dict_check.py", line 36, in _contains
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ugrd/fs/mounts.py", line 212, in get_blkid_info
    blkid_output = self._run(['blkid', device]).stdout.decode().strip()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ugrd/generator_helpers.py", line 157, in _run
    raise RuntimeError("Failed to run command: %s" % " ".join(cmd.args))
RuntimeError: Failed to run command: blkid /dev/sdc:/dev/sda

desultory · 2024-10-18T15:38:29Z

Ah, I think you may have to disable hostonly (--no-hostonly or hostonly = false) for this to work without patches. That was the main issue, I try to use blkid to get more device info for detection/validation, but blkid doesn't like that compound device notation. I could possibly just make it split off the first device and use that, but I'm not sure if there is some value in checking all devices.

desultory · 2024-10-21T16:14:34Z

Do you know if bcachefs supports mounting multi-device mounts with uuid? I'm only able to get it to work with paths.

I've added some stuff to the dev branch which should help with getting started testing this.

alexminder · 2024-10-22T07:24:04Z

Do you know if bcachefs supports mounting multi-device mounts with uuid? I'm only able to get it to work with paths.

Yes.
what versions of tools do you use?

root ~ # mount -t bcachefs
root ~ #
root ~ # blkid |grep ba4e85e0-9511-487b-b8af-f89725e8dcc8
/dev/loop1: UUID="ba4e85e0-9511-487b-b8af-f89725e8dcc8" BLOCK_SIZE="512" UUID_SUB="5fa315fa-d56a-46a6-9313-0d6eb734257b" TYPE="bcachefs"
/dev/loop2: UUID="ba4e85e0-9511-487b-b8af-f89725e8dcc8" BLOCK_SIZE="512" UUID_SUB="a41d9d75-0eba-46b7-b9e9-483291e9239b" TYPE="bcachefs"

root ~ # mount -t bcachefs UUID=ba4e85e0-9511-487b-b8af-f89725e8dcc8 /mnt/bcachefs/
root ~ # mount -t bcachefs
/dev/loop1:/dev/loop2 on /mnt/bcachefs type bcachefs (rw,relatime,metadata_replicas=2,data_replicas=2)

in my case:

root ~ # blkid -v
blkid from util-linux 2.39.4  (libblkid 2.39.4, 04-Apr-2024)
root ~ # mount -V
mount from util-linux 2.39.4 (libmount 2.39.4: selinux, btrfs, verity, namespaces, idmapping, statx, assert, debug)
root ~ # bcachefs version
1.12.0
root ~ # qlist -Iv bcachefs-tools
sys-fs/bcachefs-tools-1.13.0

as failback you can try to mount direct with bcachefs userspace utility:
bcachefs mount <Device, or UUID=uuid> [mountpoint]

If you have old userspace utility you can try to use OLD_BLKID_UUID=

alexminder · 2024-10-22T07:58:39Z

The issue systemd/systemd#8234 not directly related to this, but can get some more information.

My question: does ugrd wait to all the devices comes up? With dracut we need hooks which used by initqueue script to report that all the devices are ready to assemble filesystem.
In my expirience, I have bcachefs rootfs on 3 different devices: 1 direct attached which use ahci kernel module and 2 usb attached which use uas kernel module. USB devices much longer initialized, but dracut can see UUID from 1st device tries to mount, get mount error and fail to boot without retry, because it use /dev/disk/by-uuid/* path, which does not work with bcachefs.
Besides dracut hook https://github.com/kode54/dracut-hook-bcachefs there is hook https://github.com/breavyn/dracut-bcachefs which changes device source.

alexminder · 2024-10-22T08:38:40Z

I've added some stuff to the dev branch which should help with getting started testing this.

no_kmod: False
cpio_rotate: True
mknod_cpio: True
cpio_compression: xz
_archive_out_path: .
_cpio_archive:
check_cpio: True
check_in_cpio:
check_included_funcs: True
check_in_file:

ERROR    | No source type found in mount: {'options': {'ro'}, 'destination': PosixPath('/target_rootfs'), 'base_mount': False}
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/ugrd/main.py", line 158, in main
    generator.build()
  File "/usr/lib/python3.12/site-packages/ugrd/initramfs_generator.py", line 87, in build
    self.run_build()
  File "/usr/lib/python3.12/site-packages/ugrd/initramfs_generator.py", line 225, in run_build
    self.run_hook(task, force_exclude=True)
  File "/usr/lib/python3.12/site-packages/ugrd/initramfs_generator.py", line 141, in run_hook
    if function_output := self.run_func(function, *args, **kwargs):
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ugrd/initramfs_generator.py", line 103, in run_func
    if function_output := function(self):
                          ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ugrd/fs/mounts.py", line 717, in export_mount_info
    self["exports"]["MOUNTS_ROOT_SOURCE"] = _get_mount_str(self, self["mounts"]["root"])
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ugrd/fs/mounts.py", line 151, in _get_mount_str
    mount_type, mount_name = _get_mount_source_type(self, mount, with_val=True)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ugrd/fs/mounts.py", line 144, in _get_mount_source_type
    raise ValueError("No source type found in mount: %s" % mount)
ValueError: No source type found in mount: {'options': {'ro'}, 'destination': PosixPath('/target_rootfs'), 'base_mount': False}

this is with host_only = false

alexminder · 2024-10-22T08:46:21Z

there is some more error related with musl libc:

ERROR    | Failed to run command: ldconfig -p
ERROR    | Command output:

ERROR    | Command error:
Unimplemented option: -p

CRITICAL | Unable to run ldconfig -p, if GCC is being used, this is fatal

musl provides ldconfig as script which does not have -p option:

/usr/sbin/ldconfig:

...
get_options() {
        LDSO_CONF=""
        while getopts "vnNXf:C:r:p" opt "$@"; do
                case $opt in
                v)
                        echo "ldconfig for musl in Gentoo"
                        VERBOSE=1
                        ;;
                r)
                        ROOT=${OPTARG}
                        ;;
                f)
                        LDSO_CONF=${OPTARG}
                        ;;
                X)
                        UPDATE_LINKS=0
                        ;;
                \?)
                        echo "Invalid option: -${opt}" >&2
                        exit 1
                        ;;
                n|N|C|p)
                        echo "Unimplemented option: -${opt}" >&2
                        exit 1
                        ;;
                esac
        done
        if [[ -z ${LDSO_CONF} ]]; then
                LDSO_CONF=${ROOT}${EPREFIX}${LDSO_CONF_FILE}
        fi

        if [[ ${UPDATE_LINKS} == 1 ]]; then
                echo "Updating links is not implemented."
        fi
}
...

alexminder · 2024-10-22T11:23:04Z

I have tried to boot from dev branch and got same result.

Note, it tries to mount different devices.

desultory · 2024-10-22T13:53:51Z

there is some more error related with musl libc:

ERROR    | Failed to run command: ldconfig -p
ERROR    | Command output:

ERROR    | Command error:
Unimplemented option: -p

CRITICAL | Unable to run ldconfig -p, if GCC is being used, this is fatal

musl provides ldconfig as script which does not have -p option:

/usr/sbin/ldconfig:

...
get_options() {
        LDSO_CONF=""
        while getopts "vnNXf:C:r:p" opt "$@"; do
                case $opt in
                v)
                        echo "ldconfig for musl in Gentoo"
                        VERBOSE=1
                        ;;
                r)
                        ROOT=${OPTARG}
                        ;;
                f)
                        LDSO_CONF=${OPTARG}
                        ;;
                X)
                        UPDATE_LINKS=0
                        ;;
                \?)
                        echo "Invalid option: -${opt}" >&2
                        exit 1
                        ;;
                n|N|C|p)
                        echo "Unimplemented option: -${opt}" >&2
                        exit 1
                        ;;
                esac
        done
        if [[ -z ${LDSO_CONF} ]]; then
                LDSO_CONF=${ROOT}${EPREFIX}${LDSO_CONF_FILE}
        fi

        if [[ ${UPDATE_LINKS} == 1 ]]; then
                echo "Updating links is not implemented."
        fi
}
...

Yes, this "critical" is mostly visual, there is a toggle to not look for glibc, but it warns otherwise.

Maybe that error should say glibc instead of GCC.

I have tried to boot from dev branch and got same result.

Note, it tries to mount different devices.

Yes, this was the issue I'm facing. For multi-device bcachefs filesystems, you seemingly MUST mount by /dev/dev1:/dev/dev2 or similar, I don't know how you can mount by UUID if that is possible.

From that branch, you should be able to use the recovery console to manually mount it, you can do like setvar MOUNTS_ROOT_SOURCE "/dev/sda1:/dev/sdb1" or whatever works for your system. If I make this work using paths, I will need to make a bash function which resolves those from the UUID or something. I just really do not like this method.

alexminder · 2024-10-22T14:53:43Z

I have tried to boot from dev branch and got same result.

It seems there is no mount helper in initramfs exists: mount.bcachefs. It should be link to bcachefs binary. Without helper it use one device and kernel syscall.

alexminder · 2024-10-22T15:10:03Z

I'm trying to add mount helper and with binaries section. At host system it links to /usr/sbin, but you put bcachefs to /usr/bin.
binaries = ['/usr/sbin/mount.bcachefs','/usr/sbin/bcachefs'] puts in both places and I got warning:

WARNING  | [usr/bin/bcachefs] Hash matches existing entry: usr/sbin/bcachefs
WARNING  | [usr/bin/bcachefs] Setting filesize to 0

Can you add correct symlink with appropriate module?

desultory · 2024-10-22T15:29:36Z

I have tried to boot from dev branch and got same result.

It seems there is no mount helper in initramfs exists: mount.bcachefs. It should be link to bcachefs binary. Without helper it use one device and kernel syscall.

You should be able to do mount -t bcachefs or possibly use the bcachefs binary directly.

I'm trying to add mount helper and with binaries section. At host system it links to /usr/sbin, but you put bcachefs to /usr/bin. binaries = ['/usr/sbin/mount.bcachefs','/usr/sbin/bcachefs'] puts in both places and I got warning:
WARNING  | [usr/bin/bcachefs] Hash matches existing entry: usr/sbin/bcachefs
WARNING  | [usr/bin/bcachefs] Setting filesize to 0
Can you add correct symlink with appropriate module?

for binaries you don't use the absolute path, it resolves it from the PATHS defined in the environment where it was run.

pycpio sets the file size to 0 and makes a hardlink if there is a hash match, so that is fine. You can double check by opening the CPIO archive or using the recovery shell.

This is in the dev branch and should be automatically activated if a bcachefs mount is configured: https://github.com/desultory/ugrd/blob/4e2cae6897f6dd1e0cd2aa6b4818fe5cdb0cb6fb/src/ugrd/fs/bcachefs.toml

alexminder · 2024-10-22T15:35:58Z

You should be able to do mount -t bcachefs or possibly use the bcachefs binary directly

Without mount.bcachefs it does not works. You can notice it in my screenshot.

This is in the dev branch and should be automatically activated if a bcachefs mount is configured:

bcachefs binary not enough. mount.bcachefs symlink should be added to initramfs.

desultory · 2024-10-22T15:39:20Z

You should be able to do mount -t bcachefs or possibly use the bcachefs binary directly

Without mount.bcachefs it does not works. You can notice it in my screenshot.

I was able to do something like setvar MOUNTS_ROOT_SOURCE "/dev/vdb1:/dev/vdc1" and then exit the shell and it worked. That makes it eventually run mount -t bcachefs /dev/vdb1:/dev/vbc1 /target_rootfs which works.

This is in the dev branch and should be automatically activated if a bcachefs mount is configured:

bcachefs binary not enough. mount.bcachefs symlink should be added to initramfs.

You're able to manually define a symlink, but it won't end up running mount.bcachefs as it is. ugrd passes -t to mounts to be more generic.
https://github.com/desultory/ugrd/blob/main/docs/configuration.md#symlink-creation

2024-10-22.10-40-14.mp4

alexminder · 2024-10-22T16:15:39Z

# strace mount -vvv -t bcachefs UUID=ba4e85e0-9511-487b-b8af-f89725e8dcc8 /mnt/bcachefs/ 2>&1|grep mount.bcachef
statx(AT_FDCWD, "/sbin/mount.bcachefs", AT_STATX_DONT_SYNC|AT_NO_AUTOMOUNT, STATX_TYPE|STATX_MODE|STATX_INO, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=3194568, ...}) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xe9421fd0b90) = 1403669
wait4(1403669, [INFO  src/commands/mount.rs:303] mounting with params: device: /dev/loop1:/dev/loop2, target: /mnt/bcachefs, options: rw
[INFO  src/commands/mount.rs:43] mounting filesystem
[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 1403669
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1403669, si_uid=0, si_status=0, si_utime=0, si_stime=2 /* 0.02 s */} ---
# ls -l /sbin/mount.bcachefs
lrwxrwxrwx. 1 root root 8 Oct 18 20:56 /sbin/mount.bcachefs -> bcachefs

As you can see generic mount calls mount helper mount.bcachefs which is bcachefs binary usespace utility. If you add symlynk in to initramfs boot with UUID should works.

desultory · 2024-10-22T16:24:36Z

Adding a symlink didn't seem to help. You can do that by simply adding mount.bcachefs to binaries

Running it manually and using the UUID doesn't seem to work either:

desultory · 2024-10-26T19:18:00Z

I think this is related to udev not being used here, bcachefs seems to require that for multi-device mounts. I'm hoping bcachefs can support this without udev, otherwise I will have to make something that gets device paths from blkid output

alexminder · 2024-11-07T16:34:25Z

mount helper bcachefs binary honor BCACHEFS_BLOCK_SCAN variable to scan for devices.

More info here: koverstreet/bcachefs-tools#308 (comment)

alexminder · 2024-11-28T11:02:01Z

I think this is related to udev not being used here, bcachefs seems to require that for multi-device mounts. I'm hoping bcachefs can support this without udev, otherwise I will have to make something that gets device paths from blkid output

@desultory, Can BCACHEFS_LOG=trace help to debug?

desultory · 2024-11-28T20:12:59Z

I think this is related to udev not being used here, bcachefs seems to require that for multi-device mounts. I'm hoping bcachefs can support this without udev, otherwise I will have to make something that gets device paths from blkid output

@desultory, Can BCACHEFS_LOG=trace help to debug?

Maybe, have you tried testing this? It seems like bcachefs tools want udev, and it will potentially be unsafe to try to use them without udev.

desultory mentioned this issue Oct 24, 2024

Multi-device filesystems will not mount by UUID without udev koverstreet/bcachefs-tools#344

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bcachefs support #84

bcachefs support #84

alexminder commented Oct 15, 2024 •

edited

Loading

alexminder commented Oct 15, 2024

desultory commented Oct 15, 2024

alexminder commented Oct 16, 2024

desultory commented Oct 16, 2024

alexminder commented Oct 18, 2024

desultory commented Oct 18, 2024

desultory commented Oct 21, 2024

alexminder commented Oct 22, 2024 •

edited

Loading

alexminder commented Oct 22, 2024

alexminder commented Oct 22, 2024 •

edited

Loading

alexminder commented Oct 22, 2024

alexminder commented Oct 22, 2024

desultory commented Oct 22, 2024 •

edited

Loading

alexminder commented Oct 22, 2024 •

edited by desultory

Loading

alexminder commented Oct 22, 2024 •

edited

Loading

desultory commented Oct 22, 2024 •

edited

Loading

alexminder commented Oct 22, 2024

desultory commented Oct 22, 2024 •

edited

Loading

alexminder commented Oct 22, 2024

desultory commented Oct 22, 2024 •

edited

Loading

desultory commented Oct 26, 2024

alexminder commented Nov 7, 2024

alexminder commented Nov 28, 2024

desultory commented Nov 28, 2024

bcachefs support #84

bcachefs support #84

Comments

alexminder commented Oct 15, 2024 • edited Loading

alexminder commented Oct 15, 2024

desultory commented Oct 15, 2024

alexminder commented Oct 16, 2024

desultory commented Oct 16, 2024

alexminder commented Oct 18, 2024

desultory commented Oct 18, 2024

desultory commented Oct 21, 2024

alexminder commented Oct 22, 2024 • edited Loading

alexminder commented Oct 22, 2024

alexminder commented Oct 22, 2024 • edited Loading

alexminder commented Oct 22, 2024

alexminder commented Oct 22, 2024

desultory commented Oct 22, 2024 • edited Loading

alexminder commented Oct 22, 2024 • edited by desultory Loading

alexminder commented Oct 22, 2024 • edited Loading

desultory commented Oct 22, 2024 • edited Loading

alexminder commented Oct 22, 2024

desultory commented Oct 22, 2024 • edited Loading

alexminder commented Oct 22, 2024

desultory commented Oct 22, 2024 • edited Loading

desultory commented Oct 26, 2024

alexminder commented Nov 7, 2024

alexminder commented Nov 28, 2024

desultory commented Nov 28, 2024

alexminder commented Oct 15, 2024 •

edited

Loading

alexminder commented Oct 22, 2024 •

edited

Loading

alexminder commented Oct 22, 2024 •

edited

Loading

desultory commented Oct 22, 2024 •

edited

Loading

alexminder commented Oct 22, 2024 •

edited by desultory

Loading

alexminder commented Oct 22, 2024 •

edited

Loading

desultory commented Oct 22, 2024 •

edited

Loading

desultory commented Oct 22, 2024 •

edited

Loading

desultory commented Oct 22, 2024 •

edited

Loading