LXD disks get cross-wired... somehow

LXD Support
1

I’m trying to track down an issue that is pretty rare, but I’ve seen it at least twice. My setup is a little bit odd, in that I’m using LXD and BOSH together. BOSH itself is VM agnostic, and I wrote the CPI (aka translation layer) to interface with LXD. It seems to work very well.

However… There are times where one of the disks attached to a VM “becomes” the other disk (so I end up with two disks with the same partition scheme, etc). I don’t think the entire disk gets cloned – the subdirectories are empty as far as I could tell. At this point, my only recourse is to destroy and recreate, which isn’t optimal. (Fortunately, it’s a home lab, and so far mostly a big annoyance).

Typical setup of a VM is boot disk + ephemeral disk (software gets installed here) + persistent disk (data that “persists” goes here). In this last case, the MySQL database for Cloud Foundry lost the persistent disk – I had two disks mounted, and both were copies of the ephemeral disk.

I’m not doing any operations with BOSH, so I don’t think it’s code, per se… But I do think both times I was rebooting the hosts. (There are two – both Ubuntu – and I tend to update/reboot them together. Most of the time this is fine.)

Uncertain if there is logging that I can look at or turn on that might help? Or any suggestion would be most welcome! Part of the challenge is recreating whatever it is that is going on.

Versions:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 24.04.2 LTS
Release:	24.04
Codename:	noble
$ lxc version
Client version: 5.21.3 LTS
Server version: 5.21.3 LTS

Thanks!
-Rob

2

I just found that a dev machine is actually in this state. All the dates are 6/11, so I don’t have a clue, but presumably I did reboot.

Interestingly, when I lsblk a working host versus the broken one, you can see the change:

$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   10G  0 disk 
└─sda1   8:1    0   10G  0 part /home
                                /
sdb      8:16   0   32G  0 disk 
├─sdb1   8:17   0  7.7G  0 part [SWAP]
└─sdb2   8:18   0 24.2G  0 part /var/tmp
                                /tmp
                                /opt
                                /var/opt
                                /var/log
                                /var/vcap/data
sdc      8:32   0   64G  0 disk <=== ******* GOOD DISK ********
└─sdc1   8:33   0   64G  0 part /var/vcap/store
sr0     11:0    1    5M  0 rom

And the bad disk. It’s mounted because I was poking around!

# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   10G  0 disk 
└─sda1   8:1    0   10G  0 part /home
                                /
sdb      8:16   0   32G  0 disk 
├─sdb1   8:17   0  7.7G  0 part [SWAP]
└─sdb2   8:18   0 24.2G  0 part /var/tmp
                                /tmp
                                /opt
                                /var/opt
                                /var/log
                                /var/vcap/data
sdc      8:32   0   64G  0 disk <=== ******* BAD DISK ******** 
├─sdc1   8:33   0  7.7G  0 part 
└─sdc2   8:34   0 56.2G  0 part /mnt
sr0     11:0    1    5M  0 rom