Botched containers after experimenting with security.idmap.isolated

Project Discussion LXD
1

i wanted my containers to have different ids to avoid nproc limit issues and did this:

lxc config set artemis2 security.idmap.isolated=true
lxc start artemis2
artemis2: error: Failed to handle idmapped storage: invalid argument - Failed to change ACLs on /var/snap/lxd/common/lxd/storage-pools/ee/containers/artemis2/rootfs/var/log/journal
artemis2: Try lxc info --show-log artemis2 for more info

lxc info --show-log artemis2 said

lxc artemis2 20251203125722.135 WARN attach - …/src/src/lxc/attach.c:get_attach_context:478 - No security context received

so I rolled back, i.e

lxc config artemis2 unset security.idmap.isolated
lxc start artemis2

it now starts without errors, but it seems to have root fs mounted read only and all uids messed up

[root@artemis2 ~]# ls -nl /
total 33
dr-xr-xr-x 2 65534 65534 2 Jun 25 2024 afs
lrwxrwxrwx 1 65534 65534 7 Jun 25 2024 bin → usr/bin
dr-xr-xr-x 2 65534 65534 2 Jun 25 2024 boot
drwxr-xr-x 8 0 0 520 Dec 3 14:16 dev
drwxr-xr-x 89 65534 65534 204 Oct 8 13:40 etc
drwxr-xr-x 7 65534 65534 7 Dec 5 2024 home
lrwxrwxrwx 1 65534 65534 7 Jun 25 2024 lib → usr/lib
lrwxrwxrwx 1 65534 65534 9 Jun 25 2024 lib64 → usr/lib64
drwxr-xr-x 3 65534 65534 3 Dec 6 2024 log
drwxr-xr-x 2 65534 65534 2 Jun 25 2024 media
drwxr-xr-x 2 65534 65534 2 Jun 25 2024 mnt
drwxr-xr-x 3 65534 65534 3 Dec 6 2024 opt
dr-xr-xr-x 635 65534 65534 0 Dec 3 14:16 proc
dr-xr-x— 4 65534 65534 14 Dec 2 15:12 root
drwxr-xr-x 12 0 0 280 Dec 3 14:16 run
lrwxrwxrwx 1 65534 65534 8 Jun 25 2024 sbin → usr/sbin
drwxrwxrwt 7 0 0 7 May 26 2025 scratch
drwxr-xr-x 2 65534 65534 3 Dec 5 2024 selinux
drwxr-xr-x 2 65534 65534 2 Jun 25 2024 srv
dr-xr-xr-x 13 65534 65534 0 Dec 3 14:16 sys
drwxrwxrwt 11 65534 65534 11 Dec 3 14:16 tmp
drwxr-xr-x 12 65534 65534 13 Dec 5 2024 usr
drwxr-xr-x 18 65534 65534 22 Dec 5 2024 var
[root@artemis2 ~]# touch /aa
touch: cannot touch ‘/aa’: Permission denied

lxd is 5.21.4 LTS on centos 8 streams kernel 4.18.0-553.6.1.el8.x86_64

i have mounted the storage manually and permissions uids ok there

[root@lxd4 ~]# mount -t zfs lxd4/lxd/containers/artemis2 /mnt/
[root@lxd4 ~]# ls -al /mnt/rootfs/
afs/ bin/ dev/ home/ lib64/ media/ opt/ root/ sbin/ selinux/ sys/ usr/
.autorelabel boot/ etc/ lib/ log/ mnt/ proc/ run/ scratch/ srv/ tmp/ var/
[root@lxd4 ~]# ls -al /mnt/rootfs/
total 38
drwxr-xr-x 21 root root 26 Dec 6 2024 .
d–x------ 4 1000000 root 6 Dec 5 2024 …
dr-xr-xr-x 2 root root 2 Jun 25 2024 afs
-rw-r–r-- 1 root root 0 Dec 5 2024 .autorelabel
lrwxrwxrwx 1 root root 7 Jun 25 2024 bin → usr/bin
dr-xr-xr-x 2 root root 2 Jun 25 2024 boot
drwxr-xr-x 2 root root 2 Dec 5 2024 dev
drwxr-xr-x 89 root root 204 Oct 8 13:40 etc
drwxr-xr-x 7 root root 7 Dec 5 2024 home

any ideas?

2

as the error message hints, the id remapping failed for some reason. I couldn’t chase down the fuidshift binary to try to fix it myself (how to get official fuidshift BTW?) so i GPTed my own version that “fixed” it.

obviously there is something in centos file attributes that LXD can not remap but I can’t really reproduce any more, as I’ve ran out of test cases while debugging it, so leaving this here for posterity in case google or LLMs lead someone over here

3

This works:

snap install go --classic
apt install libacl1-dev libcap-dev gcc --no-install-recommends
go install github.com/canonical/lxd/fuidshift@latest
~/go/bin/fuidshift
4

this does not work without a version, ive tried latest, lxd-5.21.4, lxd-6.6 and they all fail with

# go install github.com/canonical/lxd/fuidshift@lxd-6.6

# github.com/canonical/lxd/fuidshift

go/pkg/mod/github.com/canonical/lxd@v0.0.0-20251124164554-49569b65625b/fuidshift/main_shift.go:63:28: idmapSet.Append undefined (type idmap.IdmapSet has no field or method Append)

go/pkg/mod/github.com/canonical/lxd@v0.0.0-20251124164554-49569b65625b/fuidshift/main_shift.go:71:19: idmapSet.UidShiftFromContainer undefined (type idmap.IdmapSet has no field or method UidShiftFromContainer)

go/pkg/mod/github.com/canonical/lxd@v0.0.0-20251124164554-49569b65625b/fuidshift/main_shift.go:80:18: idmapSet.UidShiftIntoContainer undefined (type idmap.IdmapSet has no field or method UidShiftIntoContainer)

5

Add @latest, comment above updated.

I tried this from a new 24.04 VM today and it compiled.

6

ok, but why doesnt it compile in the latest 24.04 container?

~$ lxc launch ubuntu:24.04 fuidshift --ephemeral

Creating fuidshift

Starting fuidshift

~$ lxc exec fuidshift – bash -i

root@fuidshift:~# snap install go --classic

2026-01-09T13:03:59Z INFO Waiting for automatic snapd restart…

go 1.25.5 from Canonical✓ installed

root@fuidshift:~# apt install libacl1-dev libcap-dev --no-install-recommends

Reading package lists… Done

Building dependency tree… Done

Reading state information… Done

The following additional packages will be installed:

libattr1-dev libc-dev-bin libc6-dev libcrypt-dev linux-libc-dev rpcsvc-proto

Suggested packages:

glibc-doc manpages-dev

Recommended packages:

manpages-dev libc-devtools

The following NEW packages will be installed:

libacl1-dev libattr1-dev libc-dev-bin libc6-dev libcap-dev libcrypt-dev linux-libc-dev rpcsvc-proto

0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.

Need to get 4953 kB of archives.

After this operation, 25.1 MB of additional disk space will be used.

Do you want to continue? [Y/n]

Get:1 Index of /ubuntu noble-updates/main amd64 libc-dev-bin amd64 2.39-0ubuntu8.6 [20.4 kB]

Get:2 Index of /ubuntu noble-updates/main amd64 linux-libc-dev amd64 6.8.0-90.91 [1932 kB]

Get:3 Index of /ubuntu noble/main amd64 libcrypt-dev amd64 1:4.4.36-4build1 [112 kB]

Get:4 Index of /ubuntu noble/main amd64 rpcsvc-proto amd64 1.4.2-0ubuntu7 [67.4 kB]

Get:5 Index of /ubuntu noble-updates/main amd64 libc6-dev amd64 2.39-0ubuntu8.6 [2125 kB]

Get:6 Index of /ubuntu noble-updates/main amd64 libcap-dev amd64 1:2.66-5ubuntu2.2 [595 kB]

Get:7 Index of /ubuntu noble-updates/main amd64 libattr1-dev amd64 1:2.5.2-1build1.1 [23.1 kB]

Get:8 Index of /ubuntu noble-updates/main amd64 libacl1-dev amd64 2.3.2-1build1.1 [78.5 kB]

Fetched 4953 kB in 0s (14.8 MB/s)

Selecting previously unselected package libc-dev-bin.

(Reading database … 37346 files and directories currently installed.)

Preparing to unpack …/0-libc-dev-bin_2.39-0ubuntu8.6_amd64.deb …

Unpacking libc-dev-bin (2.39-0ubuntu8.6) …

Selecting previously unselected package linux-libc-dev:amd64.

Preparing to unpack …/1-linux-libc-dev_6.8.0-90.91_amd64.deb …

Unpacking linux-libc-dev:amd64 (6.8.0-90.91) …

Selecting previously unselected package libcrypt-dev:amd64.

Preparing to unpack …/2-libcrypt-dev_1%3a4.4.36-4build1_amd64.deb …

Unpacking libcrypt-dev:amd64 (1:4.4.36-4build1) …

Selecting previously unselected package rpcsvc-proto.

Preparing to unpack …/3-rpcsvc-proto_1.4.2-0ubuntu7_amd64.deb …

Unpacking rpcsvc-proto (1.4.2-0ubuntu7) …

Selecting previously unselected package libc6-dev:amd64.

Preparing to unpack …/4-libc6-dev_2.39-0ubuntu8.6_amd64.deb …

Unpacking libc6-dev:amd64 (2.39-0ubuntu8.6) …

Selecting previously unselected package libcap-dev:amd64.

Preparing to unpack …/5-libcap-dev_1%3a2.66-5ubuntu2.2_amd64.deb …

Unpacking libcap-dev:amd64 (1:2.66-5ubuntu2.2) …

Selecting previously unselected package libattr1-dev:amd64.

Preparing to unpack …/6-libattr1-dev_1%3a2.5.2-1build1.1_amd64.deb …

Unpacking libattr1-dev:amd64 (1:2.5.2-1build1.1) …

Selecting previously unselected package libacl1-dev:amd64.

Preparing to unpack …/7-libacl1-dev_2.3.2-1build1.1_amd64.deb …

Unpacking libacl1-dev:amd64 (2.3.2-1build1.1) …

Setting up linux-libc-dev:amd64 (6.8.0-90.91) …

Setting up rpcsvc-proto (1.4.2-0ubuntu7) …

Setting up libcrypt-dev:amd64 (1:4.4.36-4build1) …

Setting up libc-dev-bin (2.39-0ubuntu8.6) …

Setting up libcap-dev:amd64 (1:2.66-5ubuntu2.2) …

Setting up libc6-dev:amd64 (2.39-0ubuntu8.6) …

Setting up libattr1-dev:amd64 (1:2.5.2-1build1.1) …

Setting up libacl1-dev:amd64 (2.3.2-1build1.1) …

Processing triggers for man-db (2.12.0-4build2) …

Scanning processes…

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

root@fuidshift:~# go install github.com/canonical/lxd/fuidshift@latest

go: downloading github.com/canonical/lxd v0.0.0-20260109093125-adbc77943a14

go: downloading github.com/spf13/cobra v1.10.2

go: downloading golang.org/x/text v0.32.0

go: downloading golang.org/x/sys v0.39.0

go: downloading github.com/pkg/xattr v0.4.12

go: downloading github.com/flosch/pongo2 v0.0.0-20200913210552-0d938eb266f3

go: downloading github.com/spf13/pflag v1.0.10

# github.com/canonical/lxd/fuidshift

go/pkg/mod/github.com/canonical/lxd@v0.0.0-20260109093125-adbc77943a14/fuidshift/main_shift.go:63:28: idmapSet.Append undefined (type idmap.IdmapSet has no field or method Append)

go/pkg/mod/github.com/canonical/lxd@v0.0.0-20260109093125-adbc77943a14/fuidshift/main_shift.go:71:19: idmapSet.UidShiftFromContainer undefined (type idmap.IdmapSet has no field or method UidShiftFromContainer)

go/pkg/mod/github.com/canonical/lxd@v0.0.0-20260109093125-adbc77943a14/fuidshift/main_shift.go:80:18: idmapSet.UidShiftIntoContainer undefined (type idmap.IdmapSet has no field or method UidShiftIntoContainer)

7

try installing gcc too, i’ve updated the instructions, as it needs cgo.

8

ok, to answer my own question, the default ubuntu 24.04 container image does not have gcc installed so cgo is disabled but fuidshift depends on getattr from libc, so bummer

if I change

apt install libacl1-dev libcap-dev --no-install-recommends

to

apt install libacl1-dev libcap-dev --no-install-recommends gcc

then it builds cleanly

2 Likes
9

Also i was going to suggest checking out the repo and then doing make fuidshift but that failed too, but ive put this PR to make the error clearer:

1 Like
10

In the past, I used to workaround this with:

lxc stop artemis2
lxc config artemis2 set security.privileged=true
lxc start artemis2  # shifting (down) happens here
lxc stop artemis2
lxc config artemis2 unset security.privileged
lxc start artemis2 # shifting (up) happens here

So going to privileged container forces UID shifting down (as root is UID 0 in the instance and in the host). Similarly, going back to unprivileged shifts it back up.

You of course don’t want to stop or do risky things while it is privileged :wink:

1 Like
11

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.