Hey! We have put some of our hosts to track 5.21/stable because it’s going to be the new LTS! woohoo.
This morning we noticed that we couldn’t use one of our lxd cloud due to LXD unix socket not accessible.
When troubleshooting I noticed that the host has updated it self to 5.21.1-feaabe1 from 5.20-f3dd836, which is totally fine.
I can’t just start the services again without getting this error:
WARNING[2024-04-09T11:26:47Z] - Couldn't find the CGroup blkio.weight, disk priority will be ignored
WARNING[2024-04-09T11:26:47Z] - Couldn't find the CGroup memory swap accounting, swap limits will be ignored
ERROR [2024-04-09T11:26:47Z] Unable to run feature checks during QEMU initialization: Unable to locate the file for firmware "OVMF_CODE.4MB.fd"
WARNING[2024-04-09T11:26:47Z] Instance type not operational driver=qemu err="QEMU failed to run feature checks" type=virtual-machine
WARNING[2024-04-09T11:26:49Z] Failed to initialize fanotify, falling back on inotify err="Failed to initialize fanotify: invalid argument"
Journal log when lxd stopped working:
2024-04-09T06:42:50Z systemd[1]: Stopping Service for snap application lxd.daemon...
2024-04-09T06:42:50Z lxd.daemon[1839705]: => Stop reason is: snap refresh
2024-04-09T06:42:50Z lxd.daemon[1839705]: => Stopping LXD
2024-04-09T06:42:50Z lxd.daemon[531348]: => LXD exited cleanly
2024-04-09T06:42:51Z lxd.daemon[1839705]: ==> Stopped LXD
2024-04-09T06:42:51Z systemd[1]: snap.lxd.daemon.service: Succeeded.
2024-04-09T06:42:51Z systemd[1]: Stopped Service for snap application lxd.daemon.
2024-04-09T06:43:27Z systemd[1]: Starting Service for snap application lxd.activate...
2024-04-09T06:43:27Z lxd.activate[1856399]: => Starting LXD activation
2024-04-09T06:43:27Z lxd.activate[1856399]: ==> Loading snap configuration
2024-04-09T06:43:27Z lxd.activate[1856399]: ==> Checking for socket activation support
2024-04-09T06:43:28Z lxd.activate[1856399]: ==> Setting LXD socket ownership
2024-04-09T06:43:28Z lxd.activate[1856399]: ==> Setting LXD user socket ownership
2024-04-09T06:43:28Z lxd.activate[1856399]: ==> Checking if LXD needs to be activated
2024-04-09T06:43:28Z systemd[1]: Started Service for snap application lxd.daemon.
2024-04-09T06:43:29Z lxd.daemon[1856662]: => Preparing the system (28057)
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Loading snap configuration
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Setting up mntns symlink (mnt:[4026532865])
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Setting up kmod wrapper
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Preparing /boot
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Preparing a clean copy of /run
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Preparing /run/bin
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Preparing a clean copy of /etc
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Preparing a clean copy of /usr/share/misc
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Setting up ceph configuration
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Setting up LVM configuration
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Setting up OVN configuration
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Rotating logs
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Unsupported ZFS version (0.8)
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Escaping the systemd cgroups
2024-04-09T06:43:29Z lxd.daemon[1856662]: ====> Detected cgroup V1
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Escaping the systemd process resource limits
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Exposing LXD documentation
2024-04-09T06:43:29Z lxd.daemon[1856662]: => Re-using existing LXCFS
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> snap base has changed, restart system to upgrade LXCFS
2024-04-09T06:43:29Z lxd.daemon[1856662]: ==> Cleaning up existing LXCFS namespace
2024-04-09T06:43:29Z lxd.daemon[1856662]: => Starting LXD
2024-04-09T06:43:29Z lxd.daemon[1857232]: time="2024-04-09T06:43:29Z" level=warning msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
2024-04-09T06:43:29Z lxd.daemon[1857232]: time="2024-04-09T06:43:29Z" level=warning msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
2024-04-09T06:43:31Z lxd.daemon[1857232]: time="2024-04-09T06:43:31Z" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
2024-04-09T06:43:31Z lxd.daemon[1857232]: time="2024-04-09T06:43:31Z" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=juju-zfs
2024-04-09T06:43:31Z lxd.daemon[1857232]: time="2024-04-09T06:43:31Z" level=error msg="Failed to start the daemon" err="Failed applying patch \"storage_move_custom_iso_block_volumes_v2\": Failed loading pool \"juju-zfs\": Required tool 'zpool' is missing"
2024-04-09T06:43:31Z lxd.daemon[1857232]: Error: Failed applying patch "storage_move_custom_iso_block_volumes_v2": Failed loading pool "juju-zfs": Required tool 'zpool' is missing
2024-04-09T06:43:31Z lxd.activate[1856484]: Error: Get "http://unix.socket/1.0": EOF
2024-04-09T06:43:31Z lxd.activate[1856399]: ====> Activation check failed, forcing activation
2024-04-09T06:43:31Z systemd[1]: snap.lxd.activate.service: Succeeded.
2024-04-09T06:43:31Z systemd[1]: Finished Service for snap application lxd.activate.
2024-04-09T06:43:32Z lxd.daemon[1856662]: Killed
2024-04-09T06:43:32Z lxd.daemon[1856662]: => LXD failed to start
2024-04-09T06:43:32Z systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
2024-04-09T06:43:32Z systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
2024-04-09T06:43:32Z systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 1.
2024-04-09T06:43:32Z systemd[1]: Stopped Service for snap application lxd.daemon.
ZFS is installed and working.
One strange thing is that the latest release for 5.21/stable is 5.20-f3dd836.
Current installed version (snap list)
Any idea what happen?