VM not booting

LXD Support
1

I am having problems with a VM.
It’s a publicly available Firewall distro that’s downloadable from SourceForge.

If launched directly from qemu:

qemu-img create efw.img 8G
qemu-system-x86_64 -boot d -cdrom efw.iso -m 512 -hda efw.img

all is well and I get to the setup screens, while going through LXD, e.g.:

lxc init efw --empty --vm
lxc config device add efw bootdisk disk source=$(pwd)/efw.iso boot.priority=10
lxc config set efw security.secureboot=false
lxc start efw --console=vga

boot fails with a cryptic “Failed to mount recovery” (most likely emitted by .iso initrd as boot is well in progress).

Only (maybe relevant) trace I see in logs is a single line:

time="2023-07-10T23:24:12+02:00" level=warning msg="Using writeback cache I/O" devPath=/var/lib/lxd/storage-pools/default/virtual-machines/efw/root.img device=root fsType=btrfs instance=efw instanceType=virtual-machine project=default

but I have no clue on real meaning and implications, I only know it doesn’t appear on other (similar) VMs.

Any hint about how to overcome problem would be very much appreciated.

2

LXD uses the virtio storage devices so it sounds like that firewall distro doesn’t support them.

3

There might be something of use here:

https://discuss.linuxcontainers.org/t/how-to-override-vm-uefi-boot-in-lxd-5-3-and-use-seabios/14479

4

Thanks @tomp.
I think you are right, but I seem unable to modify default root device; is it at all possible?

My current VM is:

mcon@lxd:~$ lxc profile show firewall 
config: {}
description: Profile to be used for Firewall VMs
devices:
  eth0:
    nictype: macvlan
    parent: enp1s0
    type: nic
  eth1:
    nictype: macvlan
    parent: enxa0cec887415e
    type: nic
  eth2:
    network: ORANGE
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: firewall
used_by:
- /1.0/instances/opnsense
- /1.0/instances/ipfire
- /1.0/instances/efw
mcon@lxd:~$ lxc config show efw --expanded
architecture: x86_64
config:
  security.secureboot: "false"
  volatile.apply_template: create
  volatile.cloud-init.instance-id: c71bf58e-bf00-42b3-8b50-aac53e79d92e
  volatile.eth0.hwaddr: 00:16:3e:ba:6a:1e
  volatile.eth1.hwaddr: 00:16:3e:9e:c9:50
  volatile.eth2.hwaddr: 00:16:3e:71:c6:c7
  volatile.uuid: ae77e787-1554-46e2-a373-c949073e9e81
devices:
  bootdisk:
    boot.priority: "10"
    source: /home/mcon/efw.iso
    type: disk
  eth0:
    nictype: macvlan
    parent: enp1s0
    type: nic
  eth1:
    nictype: macvlan
    parent: enxa0cec887415e
    type: nic
  eth2:
    network: ORANGE
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- firewall
stateful: false
description: ""
mcon@lxd:~$

I seem to understand problem is rootdevice which is handled through virtio, so I tried something along the lines:

mcon@lxd:~$ lxc config device override efw root pool="" source=/var/lib/lxd/storage-pools/default/virtual-machines/efw/root.img
Error: Invalid devices: Device validation failed for "root": Root disk entry may not have a "source" property set
mcon@lxd:~$

I am guessing I should try to override directly qemu parameters, but I was unable to find what to overridden in my case.
I tried something like:

mcon@lxd:~$ lxc config set efw raw.qemu=" -hda /var/lib/lxd/storage-pools/default/virtual-machines/efw/root.img"

but it didn’t work (no error, but same behavior as before).

Update:
Trying to use a different (autogenerated) image using -hda /home/mcon/efw.img bombs with a different message:

qemu-system-x86_64: -drive file=/home/mcon/efw.img,format=raw,index=0,media=disk: Failed to lock byte 100
5

what about adding it as we do with ISOs, e.g. the winiso device below:

devices:
  root:
    path: /
    pool: sp00
    size: 60GiB
    type: disk
  winiso:
    boot.priority: "10"
    source: /sharedpath/iso/ws2022-std-core-2108.14_lxd.iso
    type: disk

The path also has to be accessible and full, not relative.

6

I don’t need a “second disk” mounted somewhere.
I need to mount the image as root disk (i.e.: on /).
I either lost something in your suggestion or that’s a different issue.

I also tried following this thread

I am specifically trying to use recent FreeBSD “VM image” FreeBSD-13.2-RELEASE-amd64.qcow2.xz (after unzipping, of course).

My attempts are partially successful, meaning boot starts, apparently without error, but it stops after a while (before reaching command prompt).
Full log of my current attempt follows:

mcon@cinderella:~/projects/LXD$ lxc list 
+--------+---------+------+------+-----------+-----------+
|  NAME  |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+--------+---------+------+------+-----------+-----------+
| spksrc | STOPPED |      |      | CONTAINER | 0         |
+--------+---------+------+------+-----------+-----------+
| test1  | STOPPED |      |      | CONTAINER | 0         |
+--------+---------+------+------+-----------+-----------+
mcon@cinderella:~/projects/LXD$ lxc image list 
+-------+-------------+--------+-------------+--------------+------+------+-------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCHITECTURE | TYPE | SIZE | UPLOAD DATE |
+-------+-------------+--------+-------------+--------------+------+------+-------------+
mcon@cinderella:~/projects/LXD$ cat make_vm.sh 
#!/bin/bash
set -x
set -e

vm_name=${1:-freebsd}
vm_root=${2:-~/VirtualBox VMs/FreeBSD-13.2-RELEASE-amd64.qcow2}
vm_stub=$(basename "$vm_root" .qcow2)
echo \"$vm_name\" \"$vm_root\"
pushd /tmp
cat >metadata.yaml <<EOF
architecture: x86_64
creation_date: $(date +%s)
properties:
  description: $vm_stub
  os: FreeBSD
  release: 13.2
EOF
tar cf $vm_stub.tar metadata.yaml
lxc image import "$vm_stub.tar" "$vm_root" --alias $vm_name
rm metadata.yaml $vm_stub.tar
popd
out_dev=$(ip route get "$(getent ahosts "${1:-google.com}" | awk '{print $1; exit}')" | grep -Po '(?<=dev ).+?(?= (src|proto))')
lxc init $vm_name $vm_name --vm -c security.secureboot=false
lxc config device override $vm_name eth0 nictype=macvlan parent=$out_dev
lxc start $vm_name --console=vga
mcon@cinderella:~/projects/LXD$ ./make_vm.sh 
+ set -e
+ vm_name=freebsd
+ vm_root='/home/mcon/VirtualBox VMs/FreeBSD-13.2-RELEASE-amd64.qcow2'
++ basename '/home/mcon/VirtualBox VMs/FreeBSD-13.2-RELEASE-amd64.qcow2' .qcow2
+ vm_stub=FreeBSD-13.2-RELEASE-amd64
+ echo '"freebsd"' '"/home/mcon/VirtualBox' 'VMs/FreeBSD-13.2-RELEASE-amd64.qcow2"'
"freebsd" "/home/mcon/VirtualBox VMs/FreeBSD-13.2-RELEASE-amd64.qcow2"
+ pushd /tmp
/tmp ~/projects/LXD
+ cat
++ date +%s
+ tar cf FreeBSD-13.2-RELEASE-amd64.tar metadata.yaml
+ lxc image import FreeBSD-13.2-RELEASE-amd64.tar '/home/mcon/VirtualBox VMs/FreeBSD-13.2-RELEASE-amd64.qcow2' --alias freebsd
Image imported with fingerprint: 8c00511503dd92bb804d14b1d3cf2bba657343983a96ea6a58057542bd685da6
+ rm metadata.yaml FreeBSD-13.2-RELEASE-amd64.tar
+ popd
~/projects/LXD
++ grep -Po '(?<=dev ).+?(?= (src|proto))'
+++ getent ahosts google.com
+++ awk '{print $1; exit}'
++ ip route get 142.251.209.46
+ out_dev=enp7s0
+ lxc init freebsd freebsd --vm -c security.secureboot=false
Creating freebsd
+ lxc config device override freebsd eth0 nictype=macvlan parent=enp7s0
Device eth0 overridden for freebsd
+ lxc start freebsd --console=vga

Boot process always stops at the same point:

image
image1280×847 31.1 KB

At this point I have:

mcon@cinderella:~/projects/LXD$ lxc image list 
+---------+--------------+--------+----------------------------+--------------+-----------------+-----------+------------------------------+
|  ALIAS  | FINGERPRINT  | PUBLIC |        DESCRIPTION         | ARCHITECTURE |      TYPE       |   SIZE    |         UPLOAD DATE          |
+---------+--------------+--------+----------------------------+--------------+-----------------+-----------+------------------------------+
| freebsd | 8c00511503dd | no     | FreeBSD-13.2-RELEASE-amd64 | x86_64       | VIRTUAL-MACHINE | 3605.95MB | Jul 15, 2023 at 4:17pm (UTC) |
+---------+--------------+--------+----------------------------+--------------+-----------------+-----------+------------------------------+
mcon@cinderella:~/projects/LXD$ lxc list 
+---------+---------+------+------+-----------------+-----------+
|  NAME   |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS |
+---------+---------+------+------+-----------------+-----------+
| freebsd | RUNNING |      |      | VIRTUAL-MACHINE | 0         |
+---------+---------+------+------+-----------------+-----------+
| spksrc  | STOPPED |      |      | CONTAINER       | 0         |
+---------+---------+------+------+-----------------+-----------+
| test1   | STOPPED |      |      | CONTAINER       | 0         |
+---------+---------+------+------+-----------------+-----------+
mcon@cinderella:~/projects/LXD$ lxc config show freebsd --expanded
architecture: x86_64
config:
  image.description: FreeBSD-13.2-RELEASE-amd64
  image.os: FreeBSD
  image.release: "13.2"
  security.secureboot: "false"
  volatile.base_image: 8c00511503dd92bb804d14b1d3cf2bba657343983a96ea6a58057542bd685da6
  volatile.cloud-init.instance-id: 1c57065f-70cf-432c-8173-9584bca0a342
  volatile.eth0.host_name: macee3d68dd
  volatile.eth0.hwaddr: 00:16:3e:b3:85:e4
  volatile.eth0.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.uuid: 81e29d6e-62f4-479c-ba73-54c0db2a10cc
  volatile.vsock_id: "16"
devices:
  eth0:
    name: eth0
    nictype: macvlan
    parent: enp7s0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

Any hint about what could be wrong would be very welcome.

7

BSD needs a few extras hurdles…
https://www.youtube.com/watch?v=OeU2SUKV5sQ
So according to the former project lead, migration.stateful disables some flags which interfere with BSD, but a comment that raw.qemu -cpu host might be more flexible.

I’ve got a Citrix BSD based appliance and had to load an old BIOS (pc-q35-2.6) to get it to work and has been stable for quite a while, as per @tomp’s link. That’s more for something that’s a customised appliance without UEFI support and the recent security.csm config doesn’t work this appliance. You’d probably get going with the one of the two in the first paragraph.