Removing ZFS volume from running VM without rebooting and without the LXD agent installed

rg68 · August 24, 2024, 3:47pm

I’m working on a project where I am implementing glue between a VM orchestrator and LXD (BOSH, if you’re curious).

I generally have things working, but I’m running into some issues around managing the configuration disk. This disk gets updated periodically with configuration data. To do that, I am creating a small ISO image and using the API to create a storage volume from an ISO image, and that works fine.

The issue arises when I need to update that image. Right now, I’m creating a fresh image, detaching the old configuration image and attaching the new configuration image. Obviously, after doing that, I then want to remove that image from the storage pool since it’s no longer useful. However, I get the “cannot destroy ‘<name>’: dataset is busy” error.

From my experimenting, I need to reboot the VM in question to get the dataset cleared. Obviously, rebooting a VM outside it’s normal lifecycle causes issues. In this case, I suspect that the orchestrator’s installation process gets aborted. (When a VM is created, it gets created without the persistent disk… that gets added once the VM is live.)

My storage pool is ZFS, since my original ext4 couldn’t keep up with some of the automatic VM creation. (One of the projects created ~10 VMs at once, and the I/O went through the roof… ZFS seems better for these operations.)

Things that complicate this:

BOSH has a hardened OS image (which does not have the LXD agent)
If I were to install the LXD agent, these disk attach/detach events will occur at least once before it even gets installed.
Using lxc, the storage volume in question is not used by anything. And yet, it is busy.
The orchestration tool itself runs in a VM, so no host access.

I’m curious if there is some approach I can use that I’m missing? (My suspicions are that this is the end of the road for this project.)

Thanks for your time!
-Rob

pedro-rib · September 5, 2024, 4:30am

Not sure if I understood the problem correctly, if the ISO volume is being used as a root disk device it should not be possible to update it without restarting the VM, but we do support hotplugging additional disk devices. Also we support creating disks directly from ISO files with lxc config device add vm diskName disk source=/path/to/file.iso.

So if that is the use case it should be possible to update the disk using the new image by unplugging the old disk, adding the disk using the new volume, mounting the new disk accordingly and then deleting the old ISO volume. Although this process would require any processes depending on the image to stop the new disk mount.

If I am misunderstanding some aspect of your problem please clarify so I can provide a better answer. My main confusion is: What exactly is the configuration disk and is it the VM root disk?

The behavior of not showing the volume as still in use may be a problem though, could you provide your reproducer so I can investigate this in more depth?

tomp · September 5, 2024, 8:40am

Please can you log an issue with full reproducer steps (commands) here

Thanks!

rg68 · September 5, 2024, 4:32pm

Thanks for the reply! The ISO is not the boot volume, it’s likely the 3rd disk in the system. (Root/boot disk is based on the image, ephemeral disk is optional, but likely.) Via the LXD API, I can add and remove that configuration disk… but you’re correct. Something is holding that disk (even though it’s not mounted).

rg68 · September 5, 2024, 4:34pm

Yeah, I’ll try to setup a reproducer. The challenge is simply due to the hardened image that is being used comes from the Cloud Foundry BOSH ecosystem and not the LXD community. But, I think I may be able to recreate it simply from the command-line.

rg68 · September 5, 2024, 4:36pm

… and finally, I do have a work around. I just setup a scheduled process that checks the configuration disks which are not mounted (used_by is empty) and try to delete it. Log the error or a success message.

rg68 · September 5, 2024, 5:55pm

I tried to recreate the issue, and everything works as expected. (Except for that dumb “LXD VM agent isn’t currently running” error – should just be a warning and I wrote code to explicitly ignore that error). Anyway, I’m going to close this out. But, for anyone who may following along, I did capture CLI variant of the processing. Most likely, TL;DR, but for posterity.

A lot of this is, again, based on the fact that BOSH sits on top of some cloud infrastructure. In this case, LXD.

Generate an ISO image: (genisoimage happened to be pre-installed on Ubuntu, so I used it.)

$ genisoimage -o config.iso -V config-2 isocontents/
I: -input-charset not specified, using utf-8 (detected in locale settings)
Total translation table size: 0
Total rockridge attributes bytes: 0
Total directory bytes: 112
Path table size(bytes): 10
Max brk space used 0
175 extents written (0 MB)

Downloaded the current openstack stemcell (aka image) from bosh.io and extracted the QEMU image; renamed to rootfs.img since I think LXD expects that name.

$ tar xOzf ~/Downloads/bosh-stemcell-1.531-openstack-kvm-ubuntu-jammy-go_agent.tgz image | tar xzf -
$ file root.img
root.img: QEMU QCOW Image (v2), 5368709120 bytes (v2), 5368709120 bytes
$ mv root.img rootfs.img

Upload the image – create metadata.yaml tarball.

$ cat > metadata.yaml << EOF
architecture: x86_64
creation_date: 1725555591
properties:
  architecture: x86_64
  description: bosh-openstack-kvm-ubuntu-jammy-go_agent
  os: linux
  root_device_name: /
  root_disk_size: 5120MiB
EOF
$ tar czf metadata.tgz metadata.yaml
$ lxc image import metadata.tgz rootfs.img 
Image imported with fingerprint: 2334a0463f0ec22e6292d54c99d5a84cd388e4b76f3f5d84801a4034331a2ac3

Create the VM (note we need the BIOS since BOSH does not use a UEFI boot).

$ lxc init 2334a0463f0e boshvm --vm --type c2-m4 --config raw.qemu="-bios bios-256k.bin"
Creating boshvm

Import the ISO image created earlier.

$ lxc storage volume import default config.iso config-2
$ lxc storage volume list       
+---------+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
|  POOL   |      TYPE       |                               NAME                               | DESCRIPTION | CONTENT-TYPE | USED BY |
+---------+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
| default | custom          | config-2                                                         |             | iso          | 0       |
+---------+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
| default | image           | 2334a0463f0ec22e6292d54c99d5a84cd388e4b76f3f5d84801a4034331a2ac3 |             | block        | 1       |
+---------+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
| default | virtual-machine | boshvm                                                           |             | block        | 1       |
+---------+-----------------+------------------------------------------------------------------+-------------+--------------+---------+

This is where I expected things to start to deviate…

First, I just attached, detached, and then removed without starting the VM:

# Good case - unused
$ lxc config device add boshvm config-2 disk pool=default source=config-2
Device config-2 added to boshvm
$ lxc config device list boshvm
config-2
$ lxc config device remove boshvm config-2
Device config-2 removed from boshvm

Then I did the same but launched the VM. To be certain it saw everything somewhat as expected (it’s a fake config cdrom, after all), I made certain that the agent process was finding the labeled disk (config-2) and was mounting and then unmounting it:

# Expecting this to be the bad case - device is temporarily used and the lock remains...
$ lxc config device add boshvm config-2 disk pool=default source=config-2
Device config-2 added to boshvm
$ lxc start boshvm
$ lxc console boshvm
<boots...>
<disk gets mounted/unmounted as agent tries to get configuration>
/:~$ mount -l | grep config-2
<empty, so not currently mounted, looking at logs for evidence of the mount...>
<piece of logs from /var/vcap/bosh/log/current>
2024-09-05_17:27:35.92516 ********************
2024-09-05_17:27:35.92516 [Cmd Runner] 2024/09/05 17:27:35 DEBUG - Running command 'mount /dev/disk/by-label/config-2 /tmp/diskutil1195713740'
2024-09-05_17:27:35.92516 [DelayedAuditLogger] 2024/09/05 17:27:35 DEBUG - Starting logging to syslog...
2024-09-05_17:27:35.94255 [Cmd Runner] 2024/09/05 17:27:35 DEBUG - Stdout: 
2024-09-05_17:27:35.94261 [Cmd Runner] 2024/09/05 17:27:35 DEBUG - Stderr: mount: /tmp/diskutil1195713740: WARNING: source write-protected, mounted read-only.
2024-09-05_17:27:35.94267 [Cmd Runner] 2024/09/05 17:27:35 DEBUG - Successful: true (0)
2024-09-05_17:27:35.94270 [diskUtil] 2024/09/05 17:27:35 DEBUG - Mounted disk path '/dev/disk/by-label/config-2' to '/tmp/diskutil1195713740'
2024-09-05_17:27:35.94304 [diskUtil] 2024/09/05 17:27:35 DEBUG - Reading contents of '/tmp/diskutil1195713740/ec2/latest/user-data'
2024-09-05_17:27:35.94308 [File System] 2024/09/05 17:27:35 DEBUG - Reading file /tmp/diskutil1195713740/ec2/latest/user-data
2024-09-05_17:27:35.94480 [diskUtil] 2024/09/05 17:27:35 DEBUG - Unmounting disk path '/tmp/diskutil1195713740'
2024-09-05_17:27:35.94484 [File System] 2024/09/05 17:27:35 DEBUG - Reading file /proc/mounts
2024-09-05_17:27:35.94487 [File System] 2024/09/05 17:27:35 DEBUG - Read content
2024-09-05_17:27:35.94488 ********************
<logout>
<back on the desktop>
$ lxc config device remove boshvm config-2
Error: LXD VM agent isn't currently running
$ lxc config device list boshvm

# ^^ empty
$ lxc storage volume list
+---------+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
|  POOL   |      TYPE       |                               NAME                               | DESCRIPTION | CONTENT-TYPE | USED BY |
+---------+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
| default | custom          | config-2                                                         |             | iso          | 0       | <== NOT IN USE
+---------+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
| default | image           | 2334a0463f0ec22e6292d54c99d5a84cd388e4b76f3f5d84801a4034331a2ac3 |             | block        | 1       |
+---------+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
| default | virtual-machine | boshvm                                                           |             | block        | 1       |
+---------+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
$ lxc storage volume rm default config-2
Storage volume config-2 deleted
# ^^ This is where I expected it to fail.

As you can see, it worked as expected. Uncertain if I “fixed” something along the way, or if I’m too slow as a human, or the fact that it’ not a real configuration (as in the disk gets mounted/unmounted that quickly).

Thanks for checking in!