What options do I have to access a container’s LVM thin volume if the host is not able to boot anymore?
From https://discuss.linuxcontainers.org/t/resize-lxd-container/9424/3 I understand using the OS tools might break things, but in this case I’m only interested in recovering the data, so I don’t care much.
Also I have another question: setting lvm.external=true is valid only for snap LXD versions?
Thank you!
Are you accessing the LVM partition from another machine? Can you see logical volumes using the lvdisplay command?
That is correct, its set using snap set lxd
If you boot your host with another OS (such as a recovery CD) then, depending on what backing block device you used for your lvm storage pool, they should appear in lvs and you can then activate them and mount them.
Thanks for all the replies. Sorry my question was not clear: I already have all LVM container volumes seen booting from PXE. The problem is that they cannot be activated “lvchange -ay %lvm_path%” does nothing even though exit with 0. I suspect that the OS is blind to what is the real status of the LXD volumes.
Can it be that your partition is mounted as read-only? Any logs or traces you can share?
I assume from your answer that this is unexpected, and I should be able to activate the volumes booting from another OS. That’s what I expected too, and since this isn’t happening I blamed LXD that did something obscure to me and that’s why I’m here 
root@ubuntu-server:/# lvscan
ACTIVE '/dev/vg0/lv_swap' [4.00 GiB] inherit
ACTIVE '/dev/vg0/lv_root' [50.00 GiB] inherit
ACTIVE '/dev/vg0/LXDThinPool' [<783.89 GiB] inherit
inactive '/dev/vg0/containers_ca' [800.00 GiB] inherit
inactive '/dev/vg0/containers_dns--pat' [800.00 GiB] inherit
inactive '/dev/vg0/containers_git' [800.00 GiB] inherit
inactive '/dev/vg0/containers_git-backup_test' [800.00 GiB] inherit
inactive '/dev/vg0/containers_mx2' [800.00 GiB] inherit
inactive '/dev/vg0/containers_mx2-before_upgrade' [800.00 GiB] inherit
inactive '/dev/vg0/containers_mx2-al311' [800.00 GiB] inherit
inactive '/dev/vg0/containers_mx2-al312' [800.00 GiB] inherit
root@ubuntu-server:/# lvchange -ay /dev/vg0/containers_ca
root@ubuntu-server:/# echo $?
0
root@ubuntu-server:/# lvscan
ACTIVE '/dev/vg0/lv_swap' [4.00 GiB] inherit
ACTIVE '/dev/vg0/lv_root' [50.00 GiB] inherit
ACTIVE '/dev/vg0/LXDThinPool' [<783.89 GiB] inherit
inactive '/dev/vg0/containers_ca' [800.00 GiB] inherit
inactive '/dev/vg0/containers_dns--pat' [800.00 GiB] inherit
inactive '/dev/vg0/containers_git' [800.00 GiB] inherit
inactive '/dev/vg0/containers_git-backup_test' [800.00 GiB] inherit
inactive '/dev/vg0/containers_mx2' [800.00 GiB] inherit
inactive '/dev/vg0/containers_mx2-before_upgrade' [800.00 GiB] inherit
inactive '/dev/vg0/containers_mx2-al311' [800.00 GiB] inherit
inactive '/dev/vg0/containers_mx2-al312' [800.00 GiB] inherit
root@ubuntu-server:/# dmesg | tail
[ 74.687249] audit: type=1400 audit(1706685477.420:13): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.subiquity.hook.post-refresh" pid=3247 comm="apparmor_parser"
[ 100.094369] irq 4: Affinity broken due to vector space exhaustion.
[ 100.096584] irq 4: Affinity broken due to vector space exhaustion.
[ 100.106178] irq 4: Affinity broken due to vector space exhaustion.
[ 100.106328] irq 4: Affinity broken due to vector space exhaustion.
[ 100.109633] irq 4: Affinity broken due to vector space exhaustion.
[ 101.326543] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[ 101.329160] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[ 101.330254] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[ 101.330257] cfg80211: failed to load regulatory.db
root@ubuntu-server:/# tail /var/log/syslog
Jan 31 07:18:23 ubuntu-server /subiquity: Final localedir is /snap/subiquity/1966/share/locale
Jan 31 07:18:23 ubuntu-server /subiquity: switch_language en_US
Jan 31 07:18:24 ubuntu-server kernel: [ 101.326543] cfg80211: Loading compiled-in X.509 certificates for regulatory database
Jan 31 07:18:24 ubuntu-server kernel: [ 101.329160] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
Jan 31 07:18:24 ubuntu-server kernel: [ 101.330254] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Jan 31 07:18:24 ubuntu-server kernel: [ 101.330257] cfg80211: failed to load regulatory.db
Jan 31 07:22:34 ubuntu-server systemd-networkd[1748]: eth0: DHCP: No gateway received from DHCP server.
Jan 31 07:22:34 ubuntu-server systemd-timesyncd[1536]: Network configuration changed, trying to establish connection.
Jan 31 07:24:29 ubuntu-server systemd-networkd[1748]: eth1: DHCP: No gateway received from DHCP server.
Jan 31 07:24:29 ubuntu-server systemd-timesyncd[1536]: Network configuration changed, trying to establish connection.
root@ubuntu-server:/# ls /dev/vg0/
lv_root lv_swap
root@ubuntu-server:/# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
LXDThinPool vg0 twi-aotz-- <783.89g 8.14 13.82
containers_ca vg0 Vwi---tz-k 800.00g LXDThinPool
containers_dns--pat vg0 Vwi---tz-k 800.00g LXDThinPool
containers_git vg0 Vwi---tz-k 800.00g LXDThinPool
containers_git-backup_test vg0 Vri---tz-k 800.00g LXDThinPool containers_git
containers_mx2 vg0 Vwi---tz-k 800.00g LXDThinPool
containers_mx2-al311 vg0 Vri---tz-k 800.00g LXDThinPool containers_mx2
containers_mx2-al312 vg0 Vri---tz-k 800.00g LXDThinPool containers_mx2
containers_mx2-before_upgrade vg0 Vri---tz-k 800.00g LXDThinPool containers_mx2
lv_root vg0 -wi-a----- 50.00g
lv_swap vg0 -wi-a----- 4.00g
root@ubuntu-server:/# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=32914908k,nr_inodes=8228727,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6591912k,mode=755)
/ubuntu-20.04.1-live-server-amd64.iso (deleted) on /cdrom type iso9660 (ro,relatime,nojoliet,check=s,map=n,blocksize=2048)
/dev/loop1 on /rofs type squashfs (ro,noatime)
/cow on / type overlay (rw,relatime,lowerdir=/installer.squashfs:/filesystem.squashfs,upperdir=/cow/upper,workdir=/cow/work)
/cdrom/casper/extras/modules.squashfs-generic on /usr/lib/modules type squashfs (ro,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=41070)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/loop1 on /media/filesystem type squashfs (ro,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime)
/var/lib/snapd/seed/snaps/snapd_8542.snap on /snap/snapd/8542 type squashfs (ro,nodev,relatime,x-gdu.hide)
/var/lib/snapd/seed/snaps/core18_1880.snap on /snap/core18/1880 type squashfs (ro,nodev,relatime,x-gdu.hide)
/var/lib/snapd/seed/snaps/subiquity_1966.snap on /snap/subiquity/1966 type squashfs (ro,nodev,relatime,x-gdu.hide)
I cannot even deactivate the LXDThinPool. It seems that anything configured by LXD is untouchable.
root@ubuntu-server:/# lvchange -an /dev/vg0/LXDThinPool
root@ubuntu-server:/# lvscan
ACTIVE '/dev/vg0/lv_swap' [4.00 GiB] inherit
ACTIVE '/dev/vg0/lv_root' [50.00 GiB] inherit
ACTIVE '/dev/vg0/LXDThinPool' [<783.89 GiB] inherit
inactive '/dev/vg0/containers_ca' [800.00 GiB] inherit
inactive '/dev/vg0/containers_dns--pat' [800.00 GiB] inherit
inactive '/dev/vg0/containers_git' [800.00 GiB] inherit
inactive '/dev/vg0/containers_git-backup_test' [800.00 GiB] inherit
inactive '/dev/vg0/containers_mx2' [800.00 GiB] inherit
inactive '/dev/vg0/containers_mx2-before_upgrade' [800.00 GiB] inherit
inactive '/dev/vg0/containers_mx2-al311' [800.00 GiB] inherit
inactive '/dev/vg0/containers_mx2-al312' [800.00 GiB] inherit
root@ubuntu-server:/# dmesg | tail -n1
[19228.640896] device-mapper: thin: Data device (dm-3) discard unsupported: Disabling discard passdown.
Try doing: lvchange --activate y --ignoreactivationskip <volume>
Thanks! Looks like I don’t know much about LVM thin provisioning. It’s not LXD, it looks like that LVM by default create thin pool volumes with the activationskip attribute.