9

I have installed Ubuntu 18.04 on a Dell PowerEdge R720xd using ZFS. There are two 1TB boot drives in a ZFS mirror configuration. I followed the instructions in the ZFS on Linux Wiki.

(Note: My system is using an LSI LSI00244 (9201-16i) Host-bus Adapter (HBA) instead of the onboard RAID card, since ZFS and this RAID card don't get along.)

When booting Ubuntu, it takes about ten seconds for the drives to be enumerated by the system (there are 14 drives - two for the OS and 12 for data storage that will be set up in other zpools later). However, the boot process tries to import the boot pool before the drives have been enumerated.

A BusyBox error message flashes past on the screen, and it basically says:

The pool failed to import.

Manually import the pool in this BusyBox shell and then type exit to continue the boot process.

If I wait a few seconds after that message, I see that the 14 drives get listed.

I type zpool import rpool at the BusyBox prompt, which works (confirmed with zpool list) and then exit to continue the boot process. (This then leads to my next problem, a Kernel crash, but that's a separate question.)

I tried adding rootdelay=15 to the boot options, but that doesn't seem to work since it seems to want to run that delay after the ZFS pool import.

How can I get the boot process to wait for the devices to show up before it tries to import the pool?

Moshe Katz
  • 3,261

2 Answers2

13

I finally found this in /etc/default/zfs:

# Wait for this many seconds in the initrd mountroot?
# This delays startup and should be '0' on most systems. This might help on
# systems which have their ZFS root on a USB disk that takes just a little
# longer to be available
# Only applicable for Debian GNU/Linux {dkms,initramfs}.
ZFS_INITRD_POST_MODPROBE_SLEEP='0'

Here's how to set it.

  1. Boot from a Ubuntu Live CD (or any other recovery media)
  2. Import the pool to /mnt using zpool import rpool -R /mnt
  3. Bind the necessary filesyststems mount --rbind /dev /mnt/dev; mount --rbind /proc /mnt/proc; mount --rbind /sys /mnt/sys
  4. Chroot into /mnt: chroot /mnt /bin/bash --login
  5. Edit /etc/default/zfs to change the value above from 0 to 15
  6. Run update-initramfs and update-grub
  7. Exit the Chroot environment and reboot
Moshe Katz
  • 3,261
1

Update May 27, 2022:

I installed and ran Ubuntu 22.04 with ZFS-on-root (and an ext4 /boot partition).

However, based on problems I encountered, I recommend against using ZFS-on-root with Linux.

Specifically, it seems that: Linux + ZFS-on-root + snapshots (possibly also in combination with bind mounts and/or containers) can result in problems. More specifically: problems with snapshots.

It is possibly that these problems would be mitigated or avoided by creating one ZFS pool for the root filesystem, and a separate ZFS pool for everything else. But I have not tried this.

If you do want to try to run ZFS-on-root on Linux, I recommend first searching for both open and closed(!) issues at GitHub to familiarize yourself with the problems you may encounter.

The problem(s) I encountered seem to be mentioned in the following GitHub issues reports: 816, 4514, 9461, 9479, 9958, 10348, and possibly more.

Due to license conflicts, OpenZFS will (most probably) never be merged with the Linux kernel. Consequently, I suspect OpenZFS will never receive as much testing as a root-filesystem on Linux as will the GPL-licensed filesystems that are built directly into the Linux kernel (e.g. ext4, Btrfs, XFS).

Original answer from May 20, 2022:

As of Ubuntu 22.04, adding ZFS_INITRD_POST_MODPROBE_SLEEP='15' to /etc/default/zfs still solves the problem. (Aside: in my case, I only needed 1 second of delay.)

However, I believe it is worth pointing out that both ...

  • ZFS_INITRD_POST_MODPROBE_SLEEP
  • and ZFS_INITRD_PRE_MOUNTROOT_SLEEP

... are now deprecated and have been removed from /etc/default/zfs (note1, note2, pull request).

The rationale for the deprecation is described here.

In a nutshell, it seems that there is now a kernel command-line option of rootdelay=n that should be used instead.

If you want to see the actual OpenZFS initramfs script that uses these variables, it is here:
https://github.com/openzfs/zfs/blob/master/contrib/initramfs/scripts/zfs

mpb
  • 283