Running containerd containers inside lxd instance backed by ZFS storage

LXD Support
1

Hi all,

As title says. I have storage pool backed up by ZFS.

name: default
description: ""
driver: zfs
status: Created
config:
  source: default
  volatile.initial_source: /dev/sdb
  zfs.pool_name: default

I am trying to setup k8s inside lxd here. But it seems I am running into issues. I get error while creating 1st container manually.

ctr: failed to create shim task: failed to mount rootfs component: invalid argument: unknown

dmesg -T shows below
filesystem on '/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/48/fs' not supported as upperdir

So looks like zfs backed lxd storage may not work. Is there anyway I coud make this work? Will using other driver like btrf etc will help? I have 2 unsed bloack disks on this host.

Possible workaround?
After lot of reading I stumbled upon this - https://scvalex.net/posts/49/

To get his k8s PODs (CRI containerd)
Basically he did - create a zfs dataset for mounting /var/lib/containerd/io.containerd.snapshotter.v1.zfs
and edit the config for containerd to include this.

I am wondering if I can use zfs commands with lxd storage? To -

  1. to create this dataset
  2. them somehow mount it inside the instances for spefic containerd snapashotter to work.

Possible?

2

Yes overlayfs2 doesnt work on ZFS datasets.

It works on ext4, and LXD supports creating an ext4 filesystem volume backed by ZFS pool by using:

lxc storage volume create <ZFS pool> <volume name> zfs.block_mode=true block.filesystem=ext4
lxc config device add <instance> <device name> disk pool=<ZFS pool> source=<volume name> path=<path to mount volume>

And you should end up with the volume mounted as ext4 inside your guest, e.g.

/dev/zd0 on /mnt type ext4 (rw,relatime,discard,stripe=2)

You can then use this for your docker images.

See also https://documentation.ubuntu.com/lxd/en/latest/faq/#how-can-i-run-docker-inside-a-lxd-container for required syscall interception features needed for overlay2 inside a container.

You can also do this for the entire container’s root filesystem using the initial volume config settings:

E.g.

lxc launch ubuntu:24.04 c1 --storage local \
    -d root,initial.zfs.block_mode=true \
    -d root,initial.zfs.block_mode=true

Giving:

/dev/zd0 on / type ext4 (rw,relatime,idmapped,discard,stripe=2)
3

@tomp

zfs 2.2 + do support overlayfs used by container runtimes.

root@a-master1:~# nerdctl ps
CONTAINER ID    IMAGE                             COMMAND                   CREATED               STATUS    PORTS                 NAMES
29c3e38ac8bc    docker.io/library/nginx:alpine    "/docker-entrypoint.…"    About a minute ago    Up        0.0.0.0:80->80/tcp    nginx
root@a-master1:~# df -h
Filesystem                  Size  Used Avail Use% Mounted on
zdata/containers/a-master1  386G  903M  385G   1% /
none                        492K  4.0K  488K   1% /dev
tmpfs                       100K     0  100K   0% /dev/incus
/dev/root                   388G  3.3G  385G   1% /dev/kmsg
overlay                     386G  903M  385G   1% /run/containerd/io.containerd.runtime.v2.task/default/29c3e38ac8bc95e40f88fc5cdf0f414876c6bd29e87c309b9aa480e0ed1a0885/rootfs

root@incus-node2:~#zfs version
zfs-2.2.5-1
zfs-kmod-2.2.5-1

You have to install zfs 2.2+ for the same. None of ubuntu repo have zfs 2.2 (for 20 & 22). So I used Stéphane Graber’s zabbly repo to get zfs installed on my ubuntu 20

Voila!. Just works out of box

1 Like
4

Yeah Ubuntu 24.04 should work, and lxd 5.21 onwards has zfs 2.2 support in the snap.