Unable to launch multiple VMs at once

Project Discussion LXD Support
1

I’m using automation to manage VMs in a small LXD cluster. The cluster comprises 3 hosts running Ubuntu 24.04 and running LXD 5.21/stable.

With the latest update (5.21.5-f2a1a0e), I can no longer launch multiple VMs at once. I’m not certain anything else really changed.

This is using the Ceph backend. Reading through https://github.com/canonical/lxd/issues/15315, I realized that as long as I “pre-create” the image, everything else is ok. And that was true for 5.21.4. Under 5.21.5, this no longer works.

In my automation, I get the following errors:

Failed creating instance from image: Error inserting volume "35e2a917321479d8e7eeac7beeb656a640b6ea387d9aae6c484bc7e707c4e422" for project "default" in pool "remote" of type "images" into database "UNIQUE constraint failed: index 'storage_volumes_unique_storage_pool_id_node_id_project_id_name_type'

Failed creating instance from image: Failed to run: rbd --id admin --cluster ceph --image-feature layering --image-feature striping --image-feature exclusive-lock --image-feature object-map --image-feature fast-diff --image-feature deep-flatten clone lxd_remote/image_35e2a917321479d8e7eeac7beeb656a640b6ea387d9aae6c484bc7e707c4e422_ext4.block@readonly lxd_remote/virtual-machine_hydradev_vm-ccee85c0-c1d5-4ad6-52f9-011bd9b62963.block: exit status 2 (rbd: clone error: (2) No such file or directory)

In this particular instance, 15 VMs are being created. 7 fail and 8 succeed. The VMs that succeeded are across all 3 hosts.

Trying to recreate from the command-line, if I create a VM (to be certain the image is ready), and then create 3 VMs concurrently, I get the following:

$ lxc init ubuntu:26.04 --vm v1 --target hydra1 & lxc init ubuntu:26.04 --vm v2 --target hydra2 & lxc init ubuntu:26.04 --vm v3 --target hydra3 &
[1] 1833750
[2] 1833751
[3] 1833752
Creating v3
Creating v1
Creating v2
Retrieving image: Unpacking image: 7%Error: Failed instance creation: Failed creating instance from image: Failed to run: rbd --id admin --cluster ceph --pool lxd_remote rm image_4ba007f1a43e46b5dfa3d315911f4b96b3caeb77bcb2fbef77748e28d31103b0_ext4.block: exit status 16 (2026-07-05T23:26:59.406+0000 7310e09ea6c0 -1 librbd::image::PreRemoveRequest: 0x5652a21606b0 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.)
Error: Failed instance creation: Failed creating instance from image: Failed to run: rbd --id admin --cluster ceph --pool lxd_remote rm image_4ba007f1a43e46b5dfa3d315911f4b96b3caeb77bcb2fbef77748e28d31103b0_ext4.block: exit status 16 (2026-07-05T23:26:59.618+0000 71f90affd6c0 -1 librbd::image::PreRemoveRequest: 0x6188873c85d0 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.)
                                                     
[1]   Exit 1                     lxc init ubuntu:26.04 --vm v1 --target hydra1
[2]-  Exit 1                     lxc init ubuntu:26.04 --vm v2 --target hydra2
[3]+  Done                       lxc init ubuntu:26.04 --vm v3 --target hydra3
$ lxc list
+------+---------+------+------+-----------------+-----------+----------+
| NAME |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS | LOCATION |
+------+---------+------+------+-----------------+-----------+----------+
| v3   | STOPPED |      |      | VIRTUAL-MACHINE | 0         | hydra3   |
+------+---------+------+------+-----------------+-----------+----------+

(I used the same strategy as the ticket linked above in assigning each VM to a different host to make certain it is distributed.)

Thoughts? Can I safely “downgrade” or am I locked in (aka is there a database upgrade involved here)?

Thanks!

2

Hi, sorry to hear you are having trouble.

This sounds like a variant of an existing long standing bug in LXD:

Albeit it perhaps the recent update made it more likely to occur on your system due to changes in the underlying snap core or Go versions. Its racy and so can be liable to affect different use-cases in a variable way.

1 Like
3

Right, that’s the ticket I was using as a reference.

With the older version (5.21.4), as long as a VM had been created (and just one needed to be created), subsequent concurrent creates were fine. Hence the strategy of creating a VM once the image was loaded – I didn’t have to implement any locking mechanism.

Now (5.21.5), concurrent creates are never ok. Even if a VM/container has been created and is active on all hosts. Definitely got changed, and to the negative… impacts any automation folks want to do.

I guess I’m back to creating the locking mechanism for the automation myself. :expressionless:

4

@markylaing is going to look into this

5

Thank you! That would be great.

For my part, as I was writing code I realized I have a loose throttle capacity that I could twist a little bit and just reuse (that is, start using a mutex and set to 1 for how many active requests there are).

FWIW, that throttle was in place to hold creation of VMs down to ~4 (there was no mutex, so it was a bit wiggly). Mostly because when using it in a home lab, you’re 99% going to be on a single server. Launching 10+ VMs at once can timeout the automation capability. :man_shrugging:

6

Hi @rg68 ,

If pre-loading the image I was not able to replicate this issue. I could only reproduce if the image was not preloaded.

Since concurrent instance creation on ceph is already broken, I suggest creating the instances sequentially for now. We are actively working on a mechanism for distributed locking, so this overall issue will be solved in 6.10.

Thanks, Mark.

7

Ok, thank you. I swear that yesterday I was seeing multiple unpacking events with the same image but not today after 10 creates. I did get the database error, so I definitely need to get my locking mechanism to work. Fortunately, the controller code is on one machine, so I think that simplifies it somewhat.

I do think 5.21.5 is more “touchy” than 5.21.4.

When will 6 become LTS? I see from this page that it “follows the Canonical” release schedule. So I thought it would be April 2026. Obviously not!

Thanks!

8

Just a final summary: I believe what is happening is that the image gets cleaned up -immediately- in 5.21.5.

Why?

What I was doing was to load the custom image, create a VM (to preload/preprocess the image), and then delete that VM. Then the automation sets up compilation VMs, and once software has been compiled, it creates the deployment VMs. (Installs software and all that “magic” stuff.)

In 5.21.4, this was ok and didn’t take too long.

In 5.21.5, the image gets unpacked 3 times (!!!). When the “predeploy” VM was created, when the compilation VMs were created, and when the deployment VMs are created. Why? Because the VMs were deleted each time. So it was like starting new for each phase, not to mention it takes ~3 minutes each time. I was able to see that happening in the web UI, thankfully.

So, the obvious change is that I never delete that predeploy VM. Until the image has been replaced and it gets cleaned up.

9

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.