Duplication of container instances in two places in the system

tadeuszk · October 10, 2023, 1:57pm

Hi, I have two storage pools. One is the default one and the other one is created in a different path

$ lxc storage list
+---------+--------+------------------------------------+-------------+---------+---------+
|  NAME   | DRIVER |               SOURCE               | DESCRIPTION | USED BY |  STATE  |
+---------+--------+------------------------------------+-------------+---------+---------+
| default | dir    | /var/lib/lxd/storage-pools/default |             | 1       | CREATED |
+---------+--------+------------------------------------+-------------+---------+---------+
| pool2   | dir    | /home/tk/Pobrane/lxd               |             | 2       | CREATED |
+---------+--------+------------------------------------+-------------+---------+---------+

I am creating a container in pool2. however, it occupies space in two locations. In the pool2 and default pool locations. What is it about? Thanks in advance for your help

$ sudo ls -l /var/lib/lxd/storage-pools/pool2/containers
razem 8
d--x------ 4 165536 root 4096 06-22 17:02 pihole
d--x------ 4 root   root 4096 10-10 15:33 piwigo

$ sudo du -sh /var/lib/lxd/storage-pools/pool2/containers
2,7G	/var/lib/lxd/storage-pools/pool2/containers


$ sudo ls -l Pobrane/lxd/containers/
razem 8
d--x------ 4 165536 root 4096 06-22 17:02 pihole
d--x------ 4 root   root 4096 10-10 15:33 piwigo

$ sudo du -sh Pobrane/lxd/containers/
2,7G	Pobrane/lxd/containers/

ru-fu · October 10, 2023, 2:25pm

Can you post some instructions to reproduce?
How did you set up the storage, how did you launch the container, what containers are running?

tadeuszk · October 10, 2023, 4:26pm

My environment is Debian 12. LXD installed from repositories.
$ lxc version
Client version: 5.0.2
Server version: 5.0.2

I create a second pool, pool2.

$ lxc storage create pool2 dir source=/home/tk/Pobrane/lxd

I have a pool list

$ lxc storage list
+---------+--------+------------------------------------+-------------+---------+---------+
|  NAME   | DRIVER |               SOURCE               | DESCRIPTION | USED BY |  STATE  |
+---------+--------+------------------------------------+-------------+---------+---------+
| default | dir    | /var/lib/lxd/storage-pools/default |             | 12      | CREATED |
+---------+--------+------------------------------------+-------------+---------+---------+
| pool2   | dir    | /home/tk/Pobrane/lxd               |             | 0       | CREATED |
+---------+--------+------------------------------------+-------------+---------+---------+

I am creating a container that I will place on the new pool pool2

$ lxc launch images:debian/12 test1 -c limits.memory=1GiB -s pool2

$ lxc ls
+-----------+---------+---------------------+------+-----------------+-----------+
|   NAME    |  STATE  |        IPV4         | IPV6 |      TYPE       | SNAPSHOTS |
+-----------+---------+---------------------+------+-----------------+-----------+
| freescout | STOPPED |                     |      | CONTAINER       | 0         |
+-----------+---------+---------------------+------+-----------------+-----------+
| guacamole | STOPPED |                     |      | CONTAINER       | 0         |
+-----------+---------+---------------------+------+-----------------+-----------+
| librenms  | STOPPED |                     |      | CONTAINER       | 0         |
+-----------+---------+---------------------+------+-----------------+-----------+
| mssql     | STOPPED |                     |      | CONTAINER       | 0         |
+-----------+---------+---------------------+------+-----------------+-----------+
| nag-log   | STOPPED |                     |      | CONTAINER       | 0         |
+-----------+---------+---------------------+------+-----------------+-----------+
| pihole    | STOPPED |                     |      | CONTAINER       | 0         |
+-----------+---------+---------------------+------+-----------------+-----------+
| pure-ftpd | STOPPED |                     |      | CONTAINER       | 0         |
+-----------+---------+---------------------+------+-----------------+-----------+
| roundcube | STOPPED |                     |      | CONTAINER       | 0         |
+-----------+---------+---------------------+------+-----------------+-----------+
| syncting1 | STOPPED |                     |      | CONTAINER       | 0         |
+-----------+---------+---------------------+------+-----------------+-----------+
| syncting2 | STOPPED |                     |      | CONTAINER       | 0         |
+-----------+---------+---------------------+------+-----------------+-----------+
| test1     | RUNNING | 10.77.51.235 (eth0) |      | CONTAINER       | 0         |
+-----------+---------+---------------------+------+-----------------+-----------+
| winbox    | STOPPED |                     |      | VIRTUAL-MACHINE | 0         |
+-----------+---------+---------------------+------+-----------------+-----------+


$ lxc storage list
 +---------+--------+------------------------------------+-------------+---------+---------+
 |  NAME   | DRIVER |               SOURCE               | DESCRIPTION | USED BY |  STATE  |
 +---------+--------+------------------------------------+-------------+---------+---------+
 | default | dir    | /var/lib/lxd/storage-pools/default |             | 12      | CREATED |
 +---------+--------+------------------------------------+-------------+---------+---------+
 | pool2   | dir    | /home/tk/Pobrane/lxd               |             | 1       | CREATED |
 +---------+--------+------------------------------------+-------------+---------+---------+

I check where the container was created and I see that it is in two places

$ sudo du -sh /var/lib/lxd/storage-pools/pool2/containers/test1
420M	/var/lib/lxd/storage-pools/pool2/containers/test1

$ sudo du -sh /home/tk/Pobrane/lxd/containers/test1
420M	/home/tk/Pobrane/lxd/containers/test1

ru-fu · October 11, 2023, 9:09am

I tried reproducing this now, but I don’t see the same behaviour.
LXD does create the second folder (/var/snap/lxd/common/lxd/storage-pools/pool2 in my case), but it remains empty when I start a container, and the container is placed in my local folder as expected.

So this might be due to snap vs. package, or Ubuntu vs. Debian.

Hoping for some of the devs with more insight to jump in here.

tomp · November 21, 2023, 11:07am

Please show output of:

lxd sql global 'select * from storage_pools_config'
ls -la /var/lib/lxd/storage-pools/pool2
ls -la /home/tk/Pobrane/lxd
findmnt

tadeuszk · November 28, 2023, 9:15am

Hi,
Thank you for joining the conversation

$ lxd sql global 'select * from storage_pools_config'
+----+-----------------+---------+--------+------------------------------------+
| id | storage_pool_id | node_id |  key   |               value                |
+----+-----------------+---------+--------+------------------------------------+
| 1  | 1               | 1       | source | /var/lib/lxd/storage-pools/default |
| 2  | 2               | 1       | source | /home/tk/Pobrane/lxd               |
+----+-----------------+---------+--------+------------------------------------+


$ ls -la /var/lib/lxd/storage-pools/pool2
razem 40
drwxr-xr-x 10 tk   tk   4096 10-10 18:05 .
drwx--x--x  4 root root 4096 10-10 18:05 ..
drwx--x--x  2 root root 4096 10-10 18:05 buckets
drwx--x--x  7 root root 4096 11-08 18:57 containers
drwx--x--x  2 root root 4096 10-10 18:05 containers-snapshots
drwx--x--x  2 root root 4096 10-10 18:05 custom
drwx--x--x  2 root root 4096 10-10 18:05 custom-snapshots
drwx--x--x  2 root root 4096 10-10 18:05 images
drwx--x--x  2 root root 4096 10-10 18:05 virtual-machines
drwx--x--x  2 root root 4096 10-10 18:05 virtual-machines-snapshots


$ ls -la /home/tk/Pobrane/lxd
razem 40
drwxr-xr-x 10 tk   tk   4096 10-10 18:05 .
drwxr-xr-x  6 tk   tk   4096 11-24 18:33 ..
drwx--x--x  2 root root 4096 10-10 18:05 buckets
drwx--x--x  7 root root 4096 11-08 18:57 containers
drwx--x--x  2 root root 4096 10-10 18:05 containers-snapshots
drwx--x--x  2 root root 4096 10-10 18:05 custom
drwx--x--x  2 root root 4096 10-10 18:05 custom-snapshots
drwx--x--x  2 root root 4096 10-10 18:05 images
drwx--x--x  2 root root 4096 10-10 18:05 virtual-machines
drwx--x--x  2 root root 4096 10-10 18:05 virtual-machines-snapshots


$ findmnt 
TARGET                                                  SOURCE                                FSTYPE          OPTIONS
/                                                       /dev/mapper/dell--vg-root             ext4            rw,relatime,errors=remount-ro
├─/sys                                                  sysfs                                 sysfs           rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security                                securityfs                            securityfs      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup                                      cgroup2                               cgroup2         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/pstore                                      pstore                                pstore          rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware/efi/efivars                           efivarfs                              efivarfs        rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf                                         bpf                                   bpf             rw,nosuid,nodev,noexec,relatime,mode=700
│ ├─/sys/kernel/debug                                   debugfs                               debugfs         rw,nosuid,nodev,noexec,relatime
│ │ └─/sys/kernel/debug/tracing                         tracefs                               tracefs         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/tracing                                 tracefs                               tracefs         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/config                                  configfs                              configfs        rw,nosuid,nodev,noexec,relatime
│ └─/sys/fs/fuse/connections                            fusectl                               fusectl         rw,nosuid,nodev,noexec,relatime
├─/proc                                                 proc                                  proc            rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/fs/binfmt_misc                            systemd-1                             autofs          rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=547
│   └─/proc/sys/fs/binfmt_misc                          binfmt_misc                           binfmt_misc     rw,nosuid,nodev,noexec,relatime
├─/dev                                                  udev                                  devtmpfs        rw,nosuid,relatime,size=16083864k,nr_inodes=4020966,mode=755,inode64
│ ├─/dev/pts                                            devpts                                devpts          rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ ├─/dev/shm                                            tmpfs                                 tmpfs           rw,nosuid,nodev,inode64
│ ├─/dev/mqueue                                         mqueue                                mqueue          rw,nosuid,nodev,noexec,relatime
│ └─/dev/hugepages                                      hugetlbfs                             hugetlbfs       rw,relatime,pagesize=2M
├─/run                                                  tmpfs                                 tmpfs           rw,nosuid,nodev,noexec,relatime,size=3223864k,mode=755,inode64
│ ├─/run/lock                                           tmpfs                                 tmpfs           rw,nosuid,nodev,noexec,relatime,size=5120k,inode64
│ ├─/run/credentials/systemd-sysusers.service           ramfs                                 ramfs           ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/credentials/systemd-tmpfiles-setup-dev.service ramfs                                 ramfs           ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/credentials/systemd-sysctl.service             ramfs                                 ramfs           ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/credentials/systemd-tmpfiles-setup.service     ramfs                                 ramfs           ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/rpc_pipefs                                     sunrpc                                rpc_pipefs      rw,relatime
│ └─/run/user/1000                                      tmpfs                                 tmpfs           rw,nosuid,nodev,relatime,size=3223864k,nr_inodes=805966,mode=700,uid=1000,gid=1000,inode64
│   ├─/run/user/1000/gvfs                               gvfsd-fuse                            fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
│   └─/run/user/1000/doc                                portal                                fuse.portal     rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
├─/boot                                                 /dev/nvme0n1p2                        ext2            rw,relatime
│ └─/boot/efi                                           /dev/nvme0n1p1                        vfat            rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro
├─/var/lib/lxcfs                                        lxcfs                                 fuse.lxcfs      rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
├─/var/lib/lxd/shmounts                                 tmpfs                                 tmpfs           rw,relatime,size=100k,mode=711,inode64
├─/var/lib/lxd/devlxd                                   tmpfs                                 tmpfs           rw,relatime,size=100k,mode=755,inode64
├─/var/lib/lxd/storage-pools/pool2                      /dev/mapper/dell--vg-root[/home/tk/Pobrane/lxd]
│                                                                                             ext4            rw,relatime,errors=remount-ro
└─/media/tk/bck                                         /dev/mmcblk0p1                        btrfs           rw,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/

tomp · November 28, 2023, 10:42am

Thanks for that.
OK so we can see what is happening here.
When creating a dir based pool from an external source directory, the source directory gets bind-mounted into the LXD storage pool directory.

We can see this in the code here:

https://github.com/canonical/lxd/blob/main/lxd/storage/drivers/driver_dir.go#L121-L143

So the storage isn’t being duplicated, its just a bind mount.

tadeuszk · November 28, 2023, 11:56am

I don’t really understand what you mean when you write:
the source directory gets bind-mounted into the LXD storage pool directory

I don’t see any symbolic links for pool2. I have two pool2 locations in 1 system. One is here
/var/lib/lxd/storage-pools/pool2

Second one here
/home/tk/Pobrane/lxd/containers/

Both indicate the same size.

tomp · November 28, 2023, 12:08pm

They are one and the same, the output of findmnt confirms it with the line:

TARGET                                                  SOURCE
├─/var/lib/lxd/storage-pools/pool2                      /dev/mapper/dell--vg-root[/home/tk/Pobrane/lxd]

Its not a symlink, its a bind mount:

https://itslinuxfoss.com/bind-mount-linux/

LXD does this so that even external storage pools are represented inside its /var/snap/lxd/common/lxd/storage-pools directory.

tadeuszk · November 28, 2023, 3:51pm

Thank you, I understand now