Problem:
On an LXD cluster, I’m getting permission denied when launching VMs on a dir storage pool hosted on NFS share.
LXD Versions: 5.0/stable, 5.21/stable, Ubuntu 22.04
Repro steps:
- Mount an NFS share on /lxd on each of the 4 nodes
sudo mount -t nfs -o rw,nosuid,noacl,nodev,tcp,intr,hard,rsize=1048576,wsize=1048576 <host>:<share> /lxd
- Create a separate directory in the NFS share for each node
mkdir -p /lxd/240
mkdir -p /lxd/241
mkdir -p /lxd/242
mkdir -p /lxd/243
- Create LXD dir pool against /lxd
lxc storage create nfspool dir --target <node1>
lxc storage create nfspool dir --target <node2>
lxc storage create nfspool dir --target <node3>
lxc storage set nfspool source=/lxd/240 --target <node1>
lxc storage set nfspool source=/lxd/241 --target <node2>
lxc storage set nfspool source=/lxd/242 --target <node3>
- Push the pool from PENDING to CREATED state
lxc storage create nfspool dir
- We now see LXD has created the pool directory structure
sudo find /lxd/
/lxd/
/lxd/240
/lxd/240/buckets
/lxd/240/custom
/lxd/240/custom-snapshots
/lxd/240/images
/lxd/240/containers
[continued]
- Verify pool state
lxc storage ls
+---------+--------+-------------+---------+---------+
| NAME | DRIVER | DESCRIPTION | USED BY | STATE |
+---------+--------+-------------+---------+---------+
| local | lvm | | 22 | CREATED |
+---------+--------+-------------+---------+---------+
| nfspool | dir | | 0 | CREATED |
+---------+--------+-------------+---------+---------+
- Try to launch an instance using this dir/nfs pool – fails
$ lxc launch --vm testvm -s nfspool
Creating testvm
Error: Failed instance creation: Failed creating instance from image: Failed converting image to raw at "/var/snap/lxd/common/lxd/storage-pools/nfspool/virtual-machines/testvm/root.img": Failed to run: nice -n19 qemu-img convert -f qcow2 -O raw -T none /var/snap/lxd/common/lxd/images/ce0fb8befe9bd3c653925163e9a5971db96c3e892502fe4e98cbb963b33310c2.rootfs /var/snap/lxd/common/lxd/storage-pools/nfspool/virtual-machines/testvm/root.img: exit status 1 (qemu-img: /var/snap/lxd/common/lxd/storage-pools/nfspool/virtual-machines/testvm/root.img: error while converting raw: Could not create '/var/snap/lxd/common/lxd/storage-pools/nfspool/virtual-machines/testvm/root.img': Permission denied)
The strange part is LXD does create directories on the NFS share during initial pool creation,
so it seems to have some write access to the shares, but despite this, launching of VMs fails.