ZFS boot problems following upgrade to 24.04

Support and Help
,
1

I have a system which was installed with Ubuntu 22.04 in March 2024 using ZFS with ZFS native encryption (creating bpool and rpool). Following the upgrade to 24.04, I have 3 kernel versions, starting with the same parameters, which have 1 working and 2 failing outcomes with different errors related to ZFS. I cannot understand why the kernel being loaded should matter and how to address the issue.

  • linux-image-6.8.0-47-generic - works
  • linux-image-6.8.0-48-generic - fails, “cannot import rpool: no such pool or dataset”, zpool status shows “no pools available”
  • linux-image-6.8.0-49-generic - fails, “mounting rpool/ROOT/ubuntu_[snip] on /root// failed: Permission denied”

Where are these different behaviors coming from?

1 Like
2

Without having any system hardware specs, is nVidia involved?

I had to stop using encryption with anything newer than 22.04
my layout currently:

df -T
Filesystem                                       Type      Size  Used Avail Use% Mounted on
tmpfs                                            tmpfs     1.4G  2.3M  1.4G   1% /run
rpool/ROOT/ubuntu_a1d4up                         zfs       426G  4.8G  421G   2% /
rpool/ROOT/ubuntu_a1d4up/srv                     zfs       421G  128K  421G   1% /srv
rpool/ROOT/ubuntu_a1d4up/usr/local               zfs       421G  256K  421G   1% /usr/local
rpool/ROOT/ubuntu_a1d4up/var/games               zfs       421G  128K  421G   1% /var/games
rpool/ROOT/ubuntu_a1d4up/var/lib                 zfs       425G  3.5G  421G   1% /var/lib
rpool/ROOT/ubuntu_a1d4up/var/lib/AccountsService zfs       421G  128K  421G   1% /var/lib/AccountsService
rpool/ROOT/ubuntu_a1d4up/var/lib/NetworkManager  zfs       421G  256K  421G   1% /var/lib/NetworkManager
rpool/ROOT/ubuntu_a1d4up/var/lib/apt             zfs       422G   53M  421G   1% /var/lib/apt
rpool/ROOT/ubuntu_a1d4up/var/lib/dpkg            zfs       421G   41M  421G   1% /var/lib/dpkg
rpool/ROOT/ubuntu_a1d4up/var/log                 zfs       421G   23M  421G   1% /var/log
rpool/ROOT/ubuntu_a1d4up/var/mail                zfs       421G  128K  421G   1% /var/mail
rpool/ROOT/ubuntu_a1d4up/var/snap                zfs       421G  128K  421G   1% /var/snap
rpool/ROOT/ubuntu_a1d4up/var/spool               zfs       421G  128K  421G   1% /var/spool
rpool/ROOT/ubuntu_a1d4up/var/www                 zfs       421G  128K  421G   1% /var/www
tmpfs                                            tmpfs     6.8G  516K  6.8G   1% /dev/shm
efivarfs                                         efivarfs  148K  123K   21K  86% /sys/firmware/efi/efivars
tmpfs                                            tmpfs     5.0M   16K  5.0M   1% /run/lock
tmpfs                                            tmpfs     1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
tmpfs                                            tmpfs     1.0M     0  1.0M   0% /run/credentials/systemd-udev-load-credentials.service
tmpfs                                            tmpfs     1.0M     0  1.0M   0% /run/credentials/systemd-tmpfiles-setup-dev-early.service
tmpfs                                            tmpfs     1.0M     0  1.0M   0% /run/credentials/systemd-sysctl.service
tmpfs                                            tmpfs     1.0M     0  1.0M   0% /run/credentials/systemd-tmpfiles-setup-dev.service
rpool/USERDATA/root_al5glg                       zfs       421G  512K  421G   1% /root
rpool/USERDATA/home_al5glg                       zfs       444G   23G  421G   6% /home
tmpfs                                            tmpfs     6.8G   84K  6.8G   1% /tmp
bpool/BOOT/ubuntu_a1d4up                         zfs       1.7G   99M  1.6G   6% /boot
/dev/sdc1                                        vfat      1.1G  6.2M  1.1G   1% /boot/efi
tank                                             zfs       896G  287G  610G  32% /tank
tmpfs                                            tmpfs     1.0M     0  1.0M   0% /run/credentials/systemd-tmpfiles-setup.service
tmpfs                                            tmpfs     1.0M     0  1.0M   0% /run/credentials/systemd-resolved.service
tmpfs                                            tmpfs     1.0M     0  1.0M   0% /run/credentials/getty@tty1.service
tmpfs                                            tmpfs     1.4G  240K  1.4G   1% /run/user/1000
/dev/sdd1                                        ext4      916G  467G  404G  54% /media/me/WD-Blue-1TB

Apparmor might be at play here as well, can you try to disable it for a short time to check?

Also would be helpful to others if we knew what DE your are using.

BTW Your not alone: https://ubuntu-mate.community/t/23-10-to-24-04-upgrade-wont-boot-encrypted-zfs/27431

And one more Note ZFS and ZFS Encryption is stilled Labeled Experimental

1 Like
3

I’ve solved the issue.

  • no nvidia
  • apparmor is not relevant to the issue
  • MATE desktop environment - not relevant to the issue
  • Yes I found several reports of this issue, none with any explanation on why/how this can happen. Some speculation that reimporting the pool might fix the issue, though given my solution I’m not sure how. I didn’t think it was useful to list all of the links to people failing to get support on this issue. Perhaps the relevant launchpad bugs would have been appropriate though.

Because this was a bootup issue and a difference between kernels, the obvious culprit would seem to be initramfs. I was not able to eliminate this as an option because I didn’t want to risk an unbootable system (though I did have a complete backup, that was a big time commitment to restore). So I didn’t ever upgrade the initramfs of the image that was working.

The solution was:

sudo apt install cryptsetup cryptsetup-initramfs
sudo update-initramfs -c -k 6.8.0-48-generic
sudo update-initramfs -c -k 6.8.0-49-generic

Presumably the necessary scripts for zfs encryption are in cryptsetup-initramfs and none of these depend on that package:

  • ubuntu-desktop
  • ubuntu-desktop-minimal
  • ubuntu-mate-desktop
  • ubuntu-standard
  • ubuntu-minimal

My assumption is that some package depended on it in 22.04 that was changed in 24.04 and cryptsetup and cryptsetup-initramfs were removed as part of the cleanup of unnecessary packages following the 24.04 upgrade.

1 Like
4

Confirmation on the purge during the upgrade:

zgrep Purge /var/log/apt/history.log.1.gz | grep cryptsetup-initramfs

cryptsetup-initramfs:amd64 (2:2.7.0-1ubuntu4.1),
5

It did mine. :slightly_smiling_face:

But nice share on your solution, one more note to add to my collection for ZFS.

3 Likes
6

Nice. glad you was able to work past the issues