Cannot remove cluster member because “Image not found”

LXD Support
1

The cluster has six members.
Storage system is LVM.
I want to remove one cluster member.
I have moved all instances to other cluster members.
I ran lxc cluster evacuate memberX

Then I wanted to remove memberX from the cluster. It failed with

root@ijssel:~# lxc cluster remove luts
Error: Failed getting image info for "39a93d0b355279d430e8ce21c689aa88515212ee99874276e77f7f31ad7bf810": Image not found

In the mean time I have deleted all images that showed up at lxc image ls.

root@roer:~# lxc image ls
+-------+-------------+--------+-------------+--------------+------+------+-------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCHITECTURE | TYPE | SIZE | UPLOAD DATE |
+-------+-------------+--------+-------------+--------------+------+------+-------------+
root@roer:~#

root@roer:~# lxc storage volume ls local -cnLt -fcsv | grep -v container | grep -v virtual-machine
root@roer:~#

What I noticed is that there are still 23 LVM volumes present across the six cluster members with a name that starts with image_. Each fingerprint is not present on all cluster members, some are there on just one or two members.

Also interestingly, there are 24 entries in the “Global SQL” table storage_volumes with names like fingerprints.

root@roer:~# lxd sql global 'select id,name from storage_volumes where length(name) > 30 order by name'
+-----+------------------------------------------------------------------+
| id  |                               name                               |
+-----+------------------------------------------------------------------+
| 202 | 04a91c976cfc99c1480164728523cdfe450cdbd19d9040abb3d0198684482a02 |
| 254 | 0dbe67ba549060052a6abcc2b39c0767a0de8f377c0744ee24bb837b601ef02b |
| 163 | 1c49c35aba7f5a7d776a01bf3f6b3a5ca2adfe17c9e4728473b78d1d2bacd6a0 |
| 269 | 3834d4de8ace179c47aed676a18feb7985bf852ef6fa21af0f5cbe6fac56a627 |
| 217 | 39a93d0b355279d430e8ce21c689aa88515212ee99874276e77f7f31ad7bf810 |
| 226 | 39a93d0b355279d430e8ce21c689aa88515212ee99874276e77f7f31ad7bf810 |
| 271 | 3baa3f20af01daa35cc0ede82c1acce5e6a40965e692c47b184a78741ba40a0d |
| 260 | 3e9403fe7645000fc49ec89bca056c7fd53e9a142a3a9054ee02c13a2f14b6d0 |
| 158 | 4720f324a72e5d767d4d7b2bceea22408edeb329e7566d982a40af7ea2a0531a |
| 233 | 4cfdb6e4f871176d6fd535395eadb034428acaf83da9b55086f3808787e610b8 |
| 96  | 4e15a9bde9a8d5d5e96b722c32b047f78aa0bd686a2755b2d428bd665c6a37de |
| 243 | 531fa29f4b60139fb8963b8943e24c55f3a050b5406c5d613038d847f86b1d8e |
| 196 | 5503f699e02fea10825c988d293103ba5e0782fc0f98f96299b29583c12eb8dc |
| 240 | 6063f2011ba10d2dfdb2518ae02a1ddb621334a002bdf3def111dfd54dce2abd |
| 258 | 6a26c611488b9018dec0f2f84a36a69e16b27a4bcd4b08ff4a2bddb67b1a5fcb |
| 237 | 98f390c5e26031f059d6f49c48c12e3c245cb475fe7424af3728a0dc88e60064 |
| 274 | 9ae1602e1976c4ecb4ec7eacaa33260824df5cc6dd077ce794c68b567cd2a848 |
| 238 | a840c9970bf470624cb176b6b7561b7df1143e9d44de734df19f664584035336 |
| 236 | b789b81c7261b971e45b904c372b19b9a245172a50c6b88780554efbc582dab6 |
| 204 | ba19a8689440d4078e951ec08f094433dd3432d993ff5588b2846343bc6d996f |
| 246 | bdb77be1287ea5816a26bd3b3355e482012e7c99cceeece57da08fe50e8a2d71 |
| 245 | e1ab4249f4cc68b47c4c4c8ce2981deb4d5bb2ad2e81f237bf10bd57120f4615 |
| 252 | f42eab18aa248c4caeebc4dd053c8fccd5830589dff4365612b96931275a2989 |
+-----+------------------------------------------------------------------+

If I list all lvm volumes on all cluster members with name prefix image_ then I see 23 of them. From a quick glance the fingerprints match with those.

How can I get rid of these “hidden” images?

2

Is it possible for those images to be in other projects?

for p in $(lxc project list -f csv | awk -F, '{print $1}' | sed 's/ (current)$//'); do
    echo "images in ${p} project:"
    lxc image list --project="${p}"
done
3

No, there is only the default project.

4

You can do

sudo lxd sql global 'delete from storage_volumes where name = "39a93d0b355279d430e8ce21c689aa88515212ee99874276e77f7f31ad7bf810"'