"QEMU failed to run feature checks" -- how to find out what checks?

rptb1-ubuntu · September 21, 2023, 4:46pm

Installed Alpine Linux 3.18. Installed QEMU/KVM, libvirt-client, etc. I can run QEMU/KVM VMs using virt-install, virsh, etc. without difficulty. However, if I ask LXD to start a VM, it says:

/sudo:root@bluejay:/root/ #$ lxc launch images:alpine/3.18 --vm
Creating the instance
Error: Failed instance creation: Failed creating instance record: Instance type "virtual-machine" is not supported on this server: QEMU failed to run feature checks

Since QEMU/KVM clearly does work on the host, I suspect this is a configuration problem. My question is, how do I find out what checks these are?

Also, is it true that QEMU is “failing to run the checks” or is it that QEMU is running checks and saying no to features? What does this message actually mean?

Running with --verbose or --debug or running lxc monitor or examining log files does not reveal any detail. How do I debug?

Thanks.

rptb1-ubuntu · September 21, 2023, 5:12pm

Just to demonstrate that this is not a problem with the actual machine capabilities or the Alpine image, I can bring up a copy of LXD’s image in Libvirt without problems:

qemu-img create -f qcow2 -b /var/lib/lxd/images/0ad7d0446cc35426724e7986b54dee40d0037b391ef943d85430345f9416bef4.rootfs -F qcow2 /var/lib/libvirt/images/alpine-3.18.qcow2
virsh pool-refresh default 
virt-install --name "Alpine" --memory 1024 --osinfo alpinelinux3.17 --boot uefi --import --disk vol=default/alpine-3.18.qcow2 --graphics vnc,listen=0.0.0.0,port=5900 --autoconsole none

I can then connect via VNC to this machine and all appears to be working fine.

markylaing · September 22, 2023, 7:15am

Can you please inspect your LXD logs again? There should be an error log starting with Unable to run feature checks during QEMU initialization. The relevant code is here https://github.com/canonical/lxd/blob/2d8481712055d0cbb3bd952e25a1df469939990d/lxd/instance/drivers/driver_qemu.go#L7876-L8110

tomp · September 22, 2023, 7:29am

Yes, often its missing kernel modules when Alpine users try this. Its come up in the past a few times.

rptb1-ubuntu · September 22, 2023, 8:30am

Many thanks for the source code link. It has been helpful.

It seems that LXD logging is not enabled by default in Alpine’s package. I edited /etc/conf.d/lxd to enable -v --logfile /var/log/lxd/lxd.log and tried again. That revealed this message:

time=“2023-09-22T09:04:05+01:00” level=error msg=“Unable to run feature checks during QEMU initialization: QEMU premature exit: exit status 1 (qemu: could not load PC BIOS ‘/usr/share/OVMF/OVMF_CODE.fd’)”

which was fixed by apk add ovmf. But I still got

Error: Failed instance creation: Failed creating instance record: Instance type “virtual-machine” is not supported on this server: QEMU failed to run feature checks

and no log output for this command. None at all. So I enabled --debug on the lxd daemon as well. I get this in the log:

time=“2023-09-22T09:03:25+01:00” level=info msg=“LXD is starting” mode=normal path=/var/lib/lxd version=5.0.2

…

time=“2023-09-22T09:08:18+01:00” level=debug msg=“Failure for operation” class=task description=“Creating instance” err=“Failed creating instance record: Instance type "virtual-machine" is not supported on this server: QEMU failed to run feature checks” operation=0800c00f-eae5-4cdc-9be8-2af291657f67 project=default

but nothing relevant before it. No message from https://github.com/canonical/lxd/blob/2d8481712055d0cbb3bd952e25a1df469939990d/lxd/instance/drivers/driver_qemu.go#L7930 (which is strange) and none of the messages from https://github.com/canonical/lxd/blob/2d8481712055d0cbb3bd952e25a1df469939990d/lxd/instance/drivers/driver_qemu.go#L7876-L8110

There is no instance log because no instance is created.

Any suggestions on how to debug further?

rptb1-ubuntu · September 22, 2023, 4:43pm

Here is a complete reproduction using a VM (VMception!) in my case on Ubuntu 22. You can repeat this in about 10 mins.

On the host:

curl -O https://dl-cdn.alpinelinux.org/alpine/v3.18/releases/x86_64/alpine-standard-3.18.3-x86_64.iso
qemu-img create -f qcow2 repro-38824.qcow2 4G
virt-install --name "repro-38824" --memory 2048 --osinfo alpinelinux3.18 --cdrom alpine-standard-3.18.3-x86_64.iso --disk repro-38824.qcow2

On the guest, log in as root, run setup-alpine, details don’t matter, but install to /dev/vda using the “sys” configuration when asked. Then halt, and on the host again:

virsh destroy repro-38824
virsh start repro-38824
virt-viewer repro-38824 &

On the guest, edit /etc/apk/repositories to enable community
repository, then follow Alpine KVM instructions
https://wiki.alpinelinux.org/wiki/KVM :

apk add libvirt-daemon qemu-img qemu-system-x86_64 qemu-modules openrc
rc-update add libvirtd
modprobe tun
echo "tun" >> /etc/modules-load.d/tun.conf
cat /etc/modules | grep tun || echo tun >> /etc/modules
apk add libvirt-client virt-install
reboot

Then follow Alpine LXD instructions https://wiki.alpinelinux.org/wiki/LXD :

apk add lxd lxd-client ovmf
rc-update add lxc
rc-update add lxd
reboot
lxd init # answer defaults for everything
lxc launch images:alpine/3.18 --vm

And you will see the error. To prove that QEMU/KVM works, you can try:

virsh pool-define-as default dir - - - - "/var/lib/libvirt/images"
qemu-img create -f qcow2 -b /var/lib/lxd/images/0ad7d0446cc35426724e7986b54dee40d0037b391ef943d85430345f9416bef4.rootfs -F qcow2 /var/lib/libvirt/images/alpine-3.18.qcow2
pool-start default
virt-install --name "Alpine" --memory 1024 --osinfo alpinelinux3.17 --boot uefi --import --disk vol=default/alpine-3.18.qcow2 --graphics vnc,listen=0.0.0.0,port=5900 --autoconsole none

and connect with a VNC client from your host. This is an Alpine guest in an Alpine guest. Other guests of guests are available, e.g. my personal favourite:

virt-install --name "Doom" --memory 1024 --osinfo msdos6.22 --import --disk vol=default/dos622flat.qcow2 --graphics vnc,listen=0.0.0.0,port=5901 --autoconsole none --network none

tomp · October 3, 2023, 3:54pm

You have to restart lxd time to get it to re-run the suitability checks.

I am still seeing:

level=error msg="Unable to run feature checks during QEMU initialization: QEMU premature exit: exit status 1 (qemu: could not load PC BIOS '/usr/share/OVMF/OVMF_CODE.fd')"

After installing ovmf package.

Anyway I got it working after finding this post:

https://gitlab.alpinelinux.org/alpine/aports/-/issues/15175#note_328888

/usr/share/OVMF# mv OVMF_CODE.fd OVMF_CODE.fd.original
/usr/share/OVMF# ln -s OVMF.fd OVMF_CODE.fd

Also please note that on LXD 5.0.2 you also need to do this workaround (which won’t be needed with LXD 5.0.3 when it is released):

modprobe vhost-net
chmod 777  /dev/vhost-net

rptb1-ubuntu · October 4, 2023, 11:09am

Thanks. That aports link and these steps are just what I needed.

I will spend some time trying to figure out why LXD’s messaging in this case is so poor and see if I can submit an improvement suggestion.

tomp · October 4, 2023, 11:22am

Yeah I’m not sure what the issue is with the ovmf package but the error itself comes from qemu.

ocanera · January 1, 2024, 9:08am

Hi LXD Alpine package maintainer here. I’m investigating the OVMF load issue. I’ve sum up my findings here: https://gitlab.alpinelinux.org/alpine/aports/-/issues/15629

According to various documentation it appears that using “-bios” option it’s a legacy thing, for loading OVMF, pflash parameter is recommended and it works just fine. If “-bios” is really what you want, “OVMF.fd” should be loaded instead.

Can you please enlighten me why LXD is using “-bios” option?

tomp · January 2, 2024, 9:47am

LXD stopped using the -bios flag in LXD 5.20.

https://github.com/canonical/lxd/pull/12249

ocanera · January 2, 2024, 10:27am

Thanks! I ported the commit to our package and it fixes the startup QEMU feature check.