GRUB creates wrong root device

[Ryushin]

Running Debian Jessie. GRUB 2.02-beta2.9-ZOL11-7 is installed.

GRUB is creating the root device incorrectly. Running update grub produces the root device as "root=ZFS=/ROOT" instead of the proper root device of "root=ZFS=rpool/ROOT"

[Ryushin]

Richard Yao mentioned that it had to do with It has to do with grub2-probe failing silently and that the problem file was /etc/grub.d/10_linux.

[ryao]

@Ryushin and I did some debugging on his system. /usr/sbin/grub-probe --device /dev/sda1 --target=fs is not detecting ZFS. His grub2-probe is linked to zfs, but it isn't detecting it:

    linux-vdso.so.1 (0x00007ffd2edb3000)
    libzfs.so.2 => /lib64/libzfs.so.2 (0x00007f388b117000)
    libnvpair.so.1 => /lib64/libnvpair.so.1 (0x00007f388af01000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f388ab66000)
    libzfs_core.so.1 => /lib64/libzfs_core.so.1 (0x00007f388a961000)
    libzpool.so.2 => /lib64/libzpool.so.2 (0x00007f388a462000)
    libuutil.so.1 => /lib64/libuutil.so.1 (0x00007f388a24d000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f3889f49000)
    libblkid.so.1 => /lib64/libblkid.so.1 (0x00007f3889d06000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3889aea000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f388b35b000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f38898d4000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f38896cc000)
    libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f38894c7000)

My guess is that he has the latest feature flags on his pool and grub2-probe only reports zfs when it fully understands the disk format, despite it not needing to know anything about the disk format in this situation.

[Ryushin]

Updated information on my systems. This is occurring on three systems. All three systems are configured similarly.

System has GRUB installed to a mdraid1 device made up of two 16GB USB sticks. GRUB is install in the MBR of both USB sticks. A 1GB partition created on each USB stick that is mirrored using mdraid1. /boot is installed on this partition.

System has two pools. rpool and netshares:

zfs list:
NAME USED AVAIL REFER MOUNTPOINT
netshares 7.44T 46.1T 219K /netshares
netshares/movies 7.44T 46.1T 7.44T /netshares/movies
netshares/temp 219K 46.1T 219K /netshares/temp
netshares/timemachine 219K 46.1T 219K /netshares/timemachine
rpool 74.7G 4.12T 192K none
rpool/ROOT 9.54G 4.12T 7.85G /
rpool/speed_test 50.9G 4.12T 192K /speed_test
rpool/virtual_machines 14.2G 4.12T 14.2G /var/lib/libvirt

zpool status -v:
pool: netshares
state: ONLINE
scan: none requested
config:

NAME                        STATE     READ WRITE CKSUM
netshares                   ONLINE       0     0     0
  raidz2-0                  ONLINE       0     0     0
    wwn-0x5000cca249d1979e  ONLINE       0     0     0
    wwn-0x5000cca24cea12a6  ONLINE       0     0     0
    wwn-0x5000cca24cea415b  ONLINE       0     0     0
    wwn-0x5000cca249d084aa  ONLINE       0     0     0
    wwn-0x5000cca24ce90c6a  ONLINE       0     0     0
    wwn-0x5000cca24ce83a07  ONLINE       0     0     0
    wwn-0x5000cca24cea0c42  ONLINE       0     0     0
    wwn-0x5000cca24cea1261  ONLINE       0     0     0
    wwn-0x5000cca249d168bc  ONLINE       0     0     0
    wwn-0x5000cca24cebeb5b  ONLINE       0     0     0
  raidz2-1                  ONLINE       0     0     0
    wwn-0x5000cca24ce7585e  ONLINE       0     0     0
    wwn-0x5000cca249d13c38  ONLINE       0     0     0
    wwn-0x5000cca24cec2348  ONLINE       0     0     0
    wwn-0x5000cca24ceb91ba  ONLINE       0     0     0
    wwn-0x5000cca24cec23f1  ONLINE       0     0     0
    wwn-0x5000cca24cec23e9  ONLINE       0     0     0
    wwn-0x5000cca24ce92b76  ONLINE       0     0     0
    wwn-0x5000cca24ce9864a  ONLINE       0     0     0
    wwn-0x5000cca249d1478f  ONLINE       0     0     0
    wwn-0x5000cca24ce85bac  ONLINE       0     0     0

errors: No known data errors

pool: rpool
state: ONLINE
scan: none requested
config:

NAME                        STATE     READ WRITE CKSUM
rpool                       ONLINE       0     0     0
  raidz2-0                  ONLINE       0     0     0
    wwn-0x5000cca0320d7c60  ONLINE       0     0     0
    wwn-0x5000cca0320d7ab8  ONLINE       0     0     0
    wwn-0x5000cca0320d84d0  ONLINE       0     0     0
    wwn-0x5000cca0320d87c8  ONLINE       0     0     0
    wwn-0x5000cca0320d870c  ONLINE       0     0     0
    wwn-0x5000cca0320d7310  ONLINE       0     0     0

[ryao]

Something like this could work here:

diff --git a/util/grub.d/10_linux.in b/util/grub.d/10_linux.in
index 00d1931..b9a39a6 100644
--- a/util/grub.d/10_linux.in
+++ b/util/grub.d/10_linux.in
@@ -60,5 +60,5 @@ case x"$GRUB_FS" in
        fi;;
     xzfs)
-       rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
+       rpool=`blkid -p -o value -s LABEL /dev/sda1 /dev/sdb1 | uniq 2>/dev/null || true`
        bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
        LINUX_ROOT_DEVICE="ZFS=${rpool}${bootfs}"

It can be applied to /etc/grub.d/10_linux on a live system.

That said, this is not quite right because there is likely a chance that ${GRUB_DEVICE} would have multiple devices from different pools that I have not ruled out, but this likely only makes sense in the context of the Ubuntu PPA's GRUB2 patchset that uses blkid to avoid the necessity of linking to libzfs. I am unfamiliar with that patch set, so I did not make an attempt to handle that edge case.

[Ryushin]

I'm running Debian (Devuan) actually and the patch works great for me. Thanks much.

[Ryushin]

The patch really only worked for certain situations. I think a requirement would be to have the zpool bootfs parameter set. I wrote a new patch that works against the current version (2.02~beta2-22+deb8) of grub in Debian Jessie.

diff -c /tmp/10_linux.orig /etc/grub.d/10_linux
*** /tmp/10_linux.orig 2017-02-02 08:02:36.926542777 -0700
--- /etc/grub.d/10_linux 2017-02-02 15:26:02.138285042 -0700

---

*** 77,85 ****
GRUB_CMDLINE_LINUX="rootflags=subvol=${rootsubvol} ${GRUB_CMDLINE_LINUX}"
fi;;
xzfs)
! rpool=${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true
! bootfs="make_system_path_relative_to_its_root / | sed -e "s,@$,,""
! LINUX_ROOT_DEVICE="ZFS=${rpool}${bootfs}"
;;
esac

--- 77,96 ----
GRUB_CMDLINE_LINUX="rootflags=subvol=${rootsubvol} ${GRUB_CMDLINE_LINUX}"
fi;;
xzfs)
! zfsbootfs=zpool get bootfs | sed '2,100!d' | grep -v "-" | awk {'print $3'} 2>/dev/null || true
! if [ "${zfsbootfs}" = "" ]
! then
! echo "zpool bootfs parameter not set. System will fail to boot!" 1>&2
! elif [ "$(echo ${zfsbootfs} | wc -w)" -gt "1" ]
! then
! echo "Multiple zpool bootfs parameters detected:" 1>&2
! echo ${zfsbootfs} 1>&2
! echo 1>&2
! zfsbootfs=echo ${zfsbootfs} | awk {'print $1'}
! echo "Using the first bootfs listed: ${zfsbootfs}" 1>&2
! echo "Final LINUX_ROOT_DEVICE=ZFS=${zfsbootfs}" 1>&2
! fi
! LINUX_ROOT_DEVICE="ZFS=${zfsbootfs}"
;;
esac

[fulminemizzega]

Why is the default behaviour to ignore the bootfs setting? I was experimenting with snapshots and clones (debian zfs initramfs support is nice!), then found that changing bootfs did nothing and that grub scripts always generate the same bootfs= string.
Debian 9.4 (root on zfs) with zfs-dkms and zfs-initramfs from backports (version 0.7.6-1~bpo9+1)

[0x3333]

Same issue on Debian Bullseye. In my case, I overrode root=ZFS in /etc/default/grub, but it would be great if it could respect bootfs property.