LXD 6.7 init fails due to ZFS version mismatch

[petross]

Hi everyone,

I reported an issue with a ZFS mismatch on Ubuntu 22.04 when upgrading the LXD snap package to 6.7-1f11451 (refresh on latest/stable)

lxd init fails due to ZFS version mismatch

Thanks to tomponline(Tom Parrott)

However, I am still stuck.

First, I didn’t know that the “snap set lxd zfs.external=true” needs a systemctl reload snap.lxd.daemon. During last attempts I made full reboot and checked the variable using snap get lxd zfs.external - Just to be on the safe side that the setting is applied.

Still, lxd init failed with

Error: Failed to join cluster: Failed initializing member: Failed fetching information about cluster storage pools: Required tool “zpool” is missing. The snap does not contain ZFS tools matching the module version (“2.1.5-1ubuntu6~22.04.5”). Consider installing ZFS tools in the host and use ‘snap set lxd zfs.external=true’

I followed the suggestion to upgrade to a HWE kernel to have ZFS 2.2. That is true,

modinfo zfs

filename: /lib/modules/6.8.0-101-generic/kernel/zfs/zfs.ko
version: 2.2.2-0ubuntu9.4

snap set lxd zfs.external=true or false - I always find the same error.

What bugs me is that it always talks about ZFS tools matching the module version (“2.1.5-1ubuntu6~22.04.5”)

- before HWE kernel it was 2.1.5-1ubuntu6~22.04.6

*now it is “*2.2.2-0ubuntu9.4”.

Bth don’t match the version the lxd init is complaining about.. Where does it get the ZFS module information from, and why is it wrong?

Any help appreciated.

Regards

Peter

[tomp]

You may be missing the zfsutils-linux package on your host if you have snap set lxd zfs.external=true enabled (even with the HWE kernel).

I tried this out on a LXD 22.04 VM this morning:

With zfs.external=true and zfsutils-linux without the HWE kernel:

lxc launch ubuntu-daily:22.04 v1 --vm
lxc exec v1 -- snap refresh lxd --channel=6/stable --cohort=+
lxc exec v1 -- lxd init --auto
lxc exec v1 -- lxc storage create zfs zfs
Error: Required tool "zpool" is missing. The snap does not contain ZFS tools matching the module version ("2.1.5-1ubuntu6~22.04.6"). Consider installing ZFS tools in the host and use 'snap set lxd zfs.external=true'

snap set lxd zfs.external=true
systemctl reload snap.lxd.daemon
apt install zfsutils-linux
modinfo zfs | grep version
version:        2.1.5-1ubuntu6~22.04.6
srcversion:     241340E3A8EC91622D66F7B
vermagic:       5.15.0-1093-kvm SMP mod_unload modversions 

lxc exec v1 -- lxc storage create zfs zfs
# Completed ok

Without zfs.external=true and zfsutils-linux, but with HWE kernel:

lxc launch ubuntu-daily:22.04 v1 --vm
lxc exec v1 -- snap refresh lxd --channel=6/stable --cohort=+
lxc exec v1 -- lxd init --auto
lxc exec v1 -- lxc storage create zfs zfs
Error: Required tool "zpool" is missing. The snap does not contain ZFS tools matching the module version ("2.1.5-1ubuntu6~22.04.6"). Consider installing ZFS tools in the host and use 'snap set lxd zfs.external=true'

lxc exec v1 -- apt update
lxc exec v1 -- apt install --install-recommends linux-generic-hwe-22.04
# Installs the Noble GA 6.8 kernel
lxc exec v1 -- reboot
lxc exec v1 -- uname -a
Linux v1 6.8.0-101-generic #101~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 11 13:19:54 UTC  x86_64 x86_64 x86_64 GNU/Linux
lxc exec v1 --  cat /sys/module/zfs/version
2.2.2-0ubuntu9.4

lxc exec v1 -- lxc storage create zfs zfs
# Completed ok