Hello,
I have an issue with my Ceph Cluster running on Ubuntu 20.04 plus OpenStack Victoria from Ubuntu Cloud Archive as LXD containers (Compute and OSDs).
So, to be clear, my Ceph OSDs are hosted within LXD containers, and everything functions correctly if the host is Ubuntu 22.04.3 with Linux 6.2.
I currently use LXD version 5.20 installed via SNAP, which works fine on Ubuntu 22.04.3 with Linux 6.2.
However, after upgrading the host from Ubuntu 22.04.3 to 22.04.4, notably the package linux-generic-hwe-22.04 is present, the Ceph OSD daemon (still unchanged with Ubuntu 20.04 + UCA Victoria) within the LXD container fails to start.
Here is a portion of the LXD Profile (which works in Ubuntu 22.04.3 w/ Linux 6.2):
config:
raw.lxc: |-
lxc.apparmor.profile = unconfined
lxc.cgroup2.devices.allow = b 253:* rwm
lxc.mount.entry = /proc/sys/vm proc/sys/vm proc bind,rw 0 0
lxc.mount.entry = /proc/sys/fs proc/sys/fs proc bind,rw 0 0
security.privileged: "true"
description: osds
devices:
...
Here is a portion of the LXD Container for Ceph OSD (which works in Ubuntu 22.04.3 w/ Linux 6.2):
...
devices:
mapper-control:
path: /dev/mapper/control
type: unix-char
sda:
path: /dev/sda
source: /dev/disk/by-id/ata-Kingston_SSD_XYZ
type: unix-block
sdc:
path: /dev/sdc
source: /dev/disk/by-id/ata-Seagate_HDD_XYSA
type: unix-block
sdd:
path: /dev/sdd
source: /dev/disk/by-id/ata-Seagate_HDD_XYCZ
type: unix-block
sys-fs:
path: /proc/sys/fs
source: /proc/sys/fs
type: disk
sys-vm:
path: /proc/sys/vm
source: /proc/sys/vm
type: disk
...
Since the host has been upgraded, the Ceph OSD inside the container (Ubuntu 20.04 + UCA Victoria) no longer starts. The following errors are encountered:
[ceph_volume.process][INFO ] Running command: /usr/sbin/ceph-volume lvm trigger 1-<REMOVED>
[ceph_volume.process][INFO ] Running command: /usr/sbin/ceph-volume lvm trigger 4-<REMOVED>
[ceph_volume.process][INFO ] stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-999
/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-block-<REMOVED>/osd-block-<REMOVED> --path /var/lib/ceph/osd/ceph-999 --no-mon-config
abel for /dev/ceph-block-<REMOVED>/osd-block-<REMOVED>: (1) Operation not permitted
400 <STRING> -1 bluestore(/dev/ceph-block-<REMOVED>/osd-block-<REMOVED>) _read_bdev_label failed to open /dev/ceph-block-<REMOVED>/osd-block-<REMOVED>: (1) Operation not permitted
d returned non-zero exit status: 1
[ceph_volume.process][INFO ] stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-9999
/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-block-<REMOVED>/osd-block-<REMOVED> --path /var/lib/ceph/osd/ceph-9999 --no-mon-config
abel for /dev/ceph-block-<REMOVED>/osd-block-<REMOVED>: (1) Operation not permitted
400 <STRING> -1 bluestore(/dev/ceph-block-<REMOVED>/osd-block-<REMOVED>) _read_bdev_label failed to open /dev/ceph-block-<REMOVED>/osd-block-<REMOVED>: (1) Operation not permitted
d returned non-zero exit status: 1
[systemd][WARNING] command returned non-zero exit status: 1
[systemd][WARNING] failed activating OSD, retries left: 1
[systemd][WARNING] command returned non-zero exit status: 1
[systemd][WARNING] failed activating OSD, retries left: 1
As a result, the /var/lib/ceph/osd/ceph-XYZ isn’t being mounted inside the LXD Container, as before the upgrade to Ubuntu 22.04.4 in the host. And Ceph OSD doesn’t show up online in the Ceph Mon controllers.
To debug, I ran:
root@osd-1:~# dd if=/dev/ceph-block-<REMOVED>/osd-block-<REMOVED> of=/tmpdata bs=1024 count=1000
dd: failed to open '/dev/ceph-block-<REMOVED>/osd-block-<REMOVED>': Operation not permitted
It’s like in the Ceph Volume logs above!
NOTE: I’m running /sbin/lvm vgmknodes --refresh as a systemd service in the Ceph OSD container; otherwise, the LVM utilities won’t work, and Ceph Ansible doesn’t even deploy anything, to begin with. I tuned a few options in /etc/lvm/lvm.conf so LVM2 (vgcreate, lvdisplay, etc.) works inside LXD containers.
I intend to continue running my Ceph OSDs as LXD containers on Ubuntu 22.04.4 w/ Linux 6.5 (and also on the next Ubuntu 24.04) while ensuring they function correctly. Currently, the other nodes in the cluster are working as expected (Ceph OSD Inside LXD Containers with host Ubuntu 22.04.3 w/ Linux 6.2) since I’m holding the upgrade to Ubuntu 22.04.4 w/ Linux 6.5.
Please note that I’m running Ceph OSDs inside LXD for my OpenStack Cloud. I’m not using Ceph or LVM for LXD’s backend storage! I’m using LXD containers as physical/virtual machines to host something, such as Ceph OSD in one container and OpenStack Nova Compute/Network in another.
So, given that LXD is the same SNAP package on Ubuntu 22.04.3 and 22.04.4, I expected no issues since the SNAP package itself was not modified, nor were the containers. Only the kernel at the host is new.
If I remove the package linux-generic-hwe-22.04 and all Linux-6.5 packages and reboot back into Linux 6.2, the Ceph OSD inside LXD will work again! So it’s clear that something changed in Linux 6.5, breaking advanced LXD profiles with low-level device access.
A final note: If possible, I would like to run Ceph OSD as unprivileged containers! Even better, without any raw. lxc, if possible, to leverage only LXD infrastructure.
I kindly request your advice, as it prevents me from upgrading my entire infrastructure to Ubuntu 22.04.4 or newer.
NOTE: I also tried ideas from this post: https://chris-sanders.github.io/2018-05-11-block-device-in-containers/ - It didn’t work. Also, I’ve never touched UDEV in my environment.
Reference: https://discuss.linuxcontainers.org/t/ceph-osd-fails-to-start-in-lxd-container-after-upgrading-host-from-ubuntu-20-04-to-22-04/17290 - The same issue happened when I upgraded the LXD host from Ubuntu 20.04 to Ubuntu 22.04, in which people helped and suggested to replace cgroup to cgroup2, which worked! But now the issue is back. Is it cgroup3 now? 
I also requested help again on that thread, but nobody answered anything, and I’m guessing that it’s because LXD Support has moved here now. So, help! lol
Thank you for any assistance you can provide.
Cheers!