LXD 5.21.1 LTS is now available in the new 5.21 LTS track in the snap channel 5.21/candidate (as well as latest/candidate) and will be rolled out to 5.21/stable and latest/stable channels soon.
Please refresh to the 5.21/* channels if you want switch to the new LTS series, as the latest/* channels will continue on to future feature releases (6.x) in the coming weeks.
This is progressively rolling out to 5.21/stable channel now.
Hey! We have put some of our hosts to track 5.21/stable because it’s going to be the new LTS! woohoo.
This morning we noticed that we couldn’t use one of our lxd cloud due to LXD unix socket not accessible.
When troubleshooting I noticed that the host has updated it self to 5.21.1-feaabe1 from 5.20-f3dd836, which is totally fine.
I can’t just start the services again without getting this error:
WARNING[2024-04-09T11:26:47Z] - Couldn't find the CGroup blkio.weight, disk priority will be ignored
WARNING[2024-04-09T11:26:47Z] - Couldn't find the CGroup memory swap accounting, swap limits will be ignored
ERROR [2024-04-09T11:26:47Z] Unable to run feature checks during QEMU initialization: Unable to locate the file for firmware "OVMF_CODE.4MB.fd"
WARNING[2024-04-09T11:26:47Z] Instance type not operational driver=qemu err="QEMU failed to run feature checks" type=virtual-machine
WARNING[2024-04-09T11:26:49Z] Failed to initialize fanotify, falling back on inotify err="Failed to initialize fanotify: invalid argument"
Journal log when lxd stopped working:
2024-04-09T06:42:50Z systemd[1]: Stopping Service for snap application lxd.daemon...
2024-04-09T06:42:50Z lxd.daemon[1839705]: => Stop reason is: snap refresh
2024-04-09T06:42:50Z lxd.daemon[1839705]: => Stopping LXD
2024-04-09T06:42:50Z lxd.daemon[531348]: => LXD exited cleanly
2024-04-09T06:42:51Z lxd.daemon[1839705]: ==> Stopped LXD
2024-04-09T06:42:51Z systemd[1]: snap.lxd.daemon.service: Succeeded.
2024-04-09T06:42:51Z systemd[1]: Stopped Service for snap application lxd.daemon.
2024-04-09T06:43:27Z systemd[1]: Starting Service for snap application lxd.activate...
2024-04-09T06:43:27Z lxd.activate[1856399]: => Starting LXD activation
2024-04-09T06:43:27Z lxd.activate[1856399]: ==> Loading snap configuration
2024-04-09T06:43:27Z lxd.activate[1856399]: ==> Checking for socket activation support
2024-04-09T06:43:28Z lxd.activate[1856399]: ==> Setting LXD socket ownership
2024-04-09T06:43:28Z lxd.activate[1856399]: ==> Setting LXD user socket ownership
2024-04-09T06:43:28Z lxd.activate[1856399]: ==> Checking if LXD needs to be activated
2024-04-09T06:43:28Z systemd[1]: Started Service for snap application lxd.daemon.
2024-04-09T06:43:29Z lxd.daemon[1856662]: => Preparing the system (28057)
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Loading snap configuration
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Setting up mntns symlink (mnt:[4026532865])
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Setting up kmod wrapper
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Preparing /boot
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Preparing a clean copy of /run
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Preparing /run/bin
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Preparing a clean copy of /etc
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Preparing a clean copy of /usr/share/misc
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Setting up ceph configuration
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Setting up LVM configuration
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Setting up OVN configuration
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Rotating logs
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Unsupported ZFS version (0.8)
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Escaping the systemd cgroups
2024-04-09T06:43:29Z lxd.daemon[1856662]: ====> Detected cgroup V1
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Escaping the systemd process resource limits
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Exposing LXD documentation
2024-04-09T06:43:29Z lxd.daemon[1856662]: => Re-using existing LXCFS
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> snap base has changed, restart system to upgrade LXCFS
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Cleaning up existing LXCFS namespace
2024-04-09T06:43:29Z lxd.daemon[1856662]: => Starting LXD
2024-04-09T06:43:29Z lxd.daemon[1857232]: time="2024-04-09T06:43:29Z" level=warning msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
2024-04-09T06:43:29Z lxd.daemon[1857232]: time="2024-04-09T06:43:29Z" level=warning msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
2024-04-09T06:43:31Z lxd.daemon[1857232]: time="2024-04-09T06:43:31Z" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
2024-04-09T06:43:31Z lxd.daemon[1857232]: time="2024-04-09T06:43:31Z" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=juju-zfs
2024-04-09T06:43:31Z lxd.daemon[1857232]: time="2024-04-09T06:43:31Z" level=error msg="Failed to start the daemon" err="Failed applying patch \"storage_move_custom_iso_block_volumes_v2\": Failed loading pool \"juju-zfs\": Required tool 'zpool' is missing"
2024-04-09T06:43:31Z lxd.daemon[1857232]: Error: Failed applying patch "storage_move_custom_iso_block_volumes_v2": Failed loading pool "juju-zfs": Required tool 'zpool' is missing
2024-04-09T06:43:31Z lxd.activate[1856484]: Error: Get "http://unix.socket/1.0": EOF
2024-04-09T06:43:31Z lxd.activate[1856399]: ====> Activation check failed, forcing activation
2024-04-09T06:43:31Z systemd[1]: snap.lxd.activate.service: Succeeded.
2024-04-09T06:43:31Z systemd[1]: Finished Service for snap application lxd.activate.
2024-04-09T06:43:32Z lxd.daemon[1856662]: Killed
2024-04-09T06:43:32Z lxd.daemon[1856662]: => LXD failed to start
2024-04-09T06:43:32Z systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
2024-04-09T06:43:32Z systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
2024-04-09T06:43:32Z systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 1.
2024-04-09T06:43:32Z systemd[1]: Stopped Service for snap application lxd.daemon.
ZFS is installed and working.
One strange thing is that the latest release for 5.21/stable is 5.20-f3dd836.
Current installed version (snap list)
Any idea what happen?
This due to snap’s progressive rollout. We originally put LXD 5.20 into the 5.21/stable channel before it was released so that Ubuntu could use that channel for its pre-release Noble testing.
Then when LXD 5.21.0 was released it was placed into the 5.21/candidate channel, but due to the metrics certificates issue it never made it to 5.21/stable, and now LXD 5.21.1 is released it has been promoted to 5.21/stable but because of snap’s progressive rollouts not all clients will see it yet.
If you use the following snap refresh invocation you should get it:
sudo snap refresh lxd --cohort="+" --channel=5.21/stable
See https://documentation.ubuntu.com/lxd/en/latest/howto/cluster_manage/#upgrade-cluster-members
With regard to this message, LXD is trying to apply a patch to one of your storage pools, but it is marked offline due to missing ZFS tooling:
2024-04-09T06:43:31Z lxd.daemon[1857232]: time="2024-04-09T06:43:31Z" level=error msg="Failed to start the daemon" err="Failed applying patch \"storage_move_custom_iso_block_volumes_v2\": Failed loading pool \"juju-zfs\": Required tool 'zpool' is missing"
This is because the LXD snap package has not detected a compatible ZFS kernel module in your system and so isn’t providing the ZFS tooling to LXD.
ZFS 2.1 or higher is required, see https://documentation.ubuntu.com/lxd/en/latest/requirements/#zfs
Thanks @tomp for the fast answer!
Just wanted to let you know that are aware about tracking 5.21/stable was still using 5.20.
But we weren’t prepared for the zfs requirments. We’re still using Focal on some host’s and this was one of them. I have now put every focal host on hold until we have upgraded them to jammy.
Thanks again!
It may be of interest that because LXD snap provides its own ZFS tooling we only need the kernel ZFS version to support ZFS 2.1 or higher. Not the OS userspace.
So it maybe that the Focal HWE kernel may support ZFS 2.1 as that should be similar to Jammy’s kernel.
This is progressively rolling out to latest/stable channel now.
Dear,
Something goes wrong this morming (2024-04-10 11h00):
Error: Required tool 'zpool' is missing
uname -a:
Linux ta3352215 5.4.0-176-generic #196-Ubuntu SMP Fri Mar 22 16:46:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux`
snap list:
Name Version Rev Tracking Publisher Notes
core22 20240111 1122 latest/stable canonical# base
lxd 5.21.1-43998c6 28155 latest/candidate canonical# -
snapd 2.61.2 21184 latest/stable canonical# snapd
modinfo zfs:
version: 0.8.3-1ubuntu12.17
srcversion: 3CA966C3C34DC2BFBA99C85
vermagic: 5.4.0-176-generic SMP mod_unload modversions
So how to resolve this problem ?
Regards,
Error: Required tool ‘zpool’ is missing
Please see LXD 5.21.1 LTS has been released - #6 by tomp and LXD 5.21.1 LTS has been released - #8 by tomp
Hello,
I am posting in this thread as after the upgrade to 5.21.1 I am experiencing a weird bug.
I have an unprivileged container which can’t start if 2 specific devices are added in the config when the container is stopped, but if I add those devices while the container is running they work as expected.
The 2 devices are 2 folders under /mnt on the lxd host, which in turn are mount points for 2 CIFS folders.
This config is working since about 2019 and I’m tracking lxd latest/stable in snap with this container in there since the beginning, first on ubuntu 18.04 LTS, then 20.04 LTS and now on 22.04 LTS.
lxd-host $ ls -l /mnt/
total 0
drwxrwxr-x 2 1065534 1065534 0 Feb 3 00:47 dati_for_internal
drwxrwxr-x 2 1065534 1065534 0 Feb 12 2023 monitoring_for_internal
lxd-host $ mount | grep internal
systemd-1 on /mnt/dati_for_internal type autofs (rw,relatime,fd=54,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=581)
systemd-1 on /mnt/monitoring_for_internal type autofs (rw,relatime,fd=55,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=584)
//10.10.70.20/Dati on /mnt/dati_for_internal type cifs (rw,nosuid,nodev,noexec,relatime,vers=3.0,cache=strict,username=nucleo,uid=1065534,noforceuid,gid=1065534,noforcegid,addr=10.10.70.20,file_mode=0775,dir_mode=0775,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1,x-systemd.automount)
//10.10.70.20/Monitoring on /mnt/monitoring_for_internal type cifs (rw,nosuid,nodev,noexec,relatime,vers=3.0,cache=strict,username=monitoring,uid=1065534,noforceuid,gid=1065534,noforcegid,addr=10.10.70.20,file_mode=0775,dir_mode=0775,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1,x-systemd.automount)
This is the container config snip
lxd-host $ lxc config edit internal-lxd
<...>
devices:
monitoring:
path: /mnt/monitoring
source: /mnt/monitoring_for_internal
type: disk
dati:
path: /mnt/dati
source: /mnt/dati_for_internal
type: disk
<...>
When the config is present before starting the container I get
lxd-host $ sudo lxc info --show-log internal-lxd
Name: internal-lxd
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2019/05/11 15:59 CEST
Last Used: 2024/04/10 19:22 CEST
Log:
lxc internal-lxd 20240410172253.865 WARN idmap_utils - ../src/src/lxc/idmap_utils.c:lxc_map_ids:165 - newuidmap binary is missing
lxc internal-lxd 20240410172253.865 WARN idmap_utils - ../src/src/lxc/idmap_utils.c:lxc_map_ids:171 - newgidmap binary is missing
lxc internal-lxd 20240410172253.866 WARN idmap_utils - ../src/src/lxc/idmap_utils.c:lxc_map_ids:165 - newuidmap binary is missing
lxc internal-lxd 20240410172253.866 WARN idmap_utils - ../src/src/lxc/idmap_utils.c:lxc_map_ids:171 - newgidmap binary is missing
lxc internal-lxd 20240410172253.930 ERROR conf - ../src/src/lxc/conf.c:mount_entry:2262 - Operation not permitted - Failed to mount "/var/snap/lxd/common/lxd/devices/internal-lxd/disk.dati.mnt-dati" on "/var/snap/lxd/common/lxc//mnt/dati"
lxc internal-lxd 20240410172253.930 ERROR conf - ../src/src/lxc/conf.c:lxc_setup:3915 - Failed to setup mount entries
lxc internal-lxd 20240410172253.930 ERROR start - ../src/src/lxc/start.c:do_start:1273 - Failed to setup container "internal-lxd"
lxc internal-lxd 20240410172253.932 ERROR sync - ../src/src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
lxc internal-lxd 20240410172253.946 WARN network - ../src/src/lxc/network.c:lxc_delete_network_priv:3671 - Failed to rename interface with index 0 from "eth0" to its initial name "veth5244401e"
lxc internal-lxd 20240410172253.946 ERROR lxccontainer - ../src/src/lxc/lxccontainer.c:wait_on_daemonized_start:837 - Received container state "ABORTING" instead of "RUNNING"
lxc internal-lxd 20240410172253.946 ERROR start - ../src/src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "internal-lxd"
lxc internal-lxd 20240410172253.946 WARN start - ../src/src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 17 for process 9795
lxc 20240410172254.415 ERROR af_unix - ../src/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20240410172254.416 ERROR commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"
But if I remove the config, start the container, wait for it to boot, put the config back the 2 folders are correctly mounted in the container and all works as expected.
Can someone shed some light? At the next system update/reboot this container is going to fail and I’ll be without DNS once again 
Thanks in advance
Let me know how I can facilitate troubleshooting and fixing and I’ll follow up
the snap upgrade with the ZFS dependency has hit me. I’m on 20.04 LTS , and I can’t see a way to get to the required ZFS version. possibly something I’m missing? a roll back of LXD has then produced a DB version error.
Good call! I just installed the HWE kernel in focal, rebooted, and everything came back up normal.
Thanks!
Hey there! There’s no need to rollback. All I had to do was install the HWE kernel per @tomp’s suggestion and I was back up and running. Give it a try.
Good luck!
any good RTFM resources you could point me to ? Ubuntu kernel lifecycle and enablement stack | Ubuntu ?
You already got it but here’s the specific page for Focal:
https://ubuntu.com/kernel/lifecycle#installation-20-04
I did exactly as the guide says except for without the “–install-recommends”. Rebooted and voila!
I’m back up and running. thankyou all. my fault for not paying enough attention to my snap config for prod and pre-prod/test.
Glad those with ZFS 0.8 are back up and running with the Focal HWE kernel.
Those tracking the latest/stable channel will always get the latest feature release of LXD, and over time (but especially at the boundary of a new LTS release) the minimum requirements for LXD will change.
With the LXD 5.21.0 LTS release as planned, we have now changed the default snap track from latest to 5.21. This is to ensure that those installing LXD by way of just doing snap install lxd wouldn’t end up on the latest/stable channel but would instead get the latest LTS channel (now set to 5.21/stable).
We won’t remove support for specific ZFS versions during the lifecycle of an LTS release, so those following a particular LTS track will need to consciously refresh to a different track (i.e latest/stable or a future LTS track) to get new features.
thankyou this will be a big help