Ubuntu 22.04 and 20.04 containers stuck in "System is booting" up state

LXD Support
6

Cross-posting our current workaround from the linked askubuntu thread, in case it helps others. This should not be marked as the “Solution” though, since this is much more of a workaround than anything else. :neutral_face:

Workaround

When you do apt-get purge snapd and apt-get install snapd, these snaps are not pre-seeded any more. So that’s why it “works”. But it would sure be interesting to understand more about why this happens and why only on some of our machines…

Some more details from our debugging effort

It seems clear that this is the snapd.seeded.service which is blocking the startup:

root@emerging-caiman:~# journalctl -u snapd.seeded.service
Oct 18 11:02:19 emerging-caiman systemd[1]: Starting Wait until snapd is fully seeded...

snap debug seeding gives us this:

seeded:  false
seed-error: |
  cannot perform the following tasks:
  - Start snap "lxd" (24322) services (systemctl command [start snap.lxd.activate.service] failed
  with exit status 1: Job for snap.lxd.activate.service failed because the control process exited
  with error code.
  See "systemctl status snap.lxd.activate.service" and "journalctl -xeu snap.lxd.activate.service"
  for details.
  )
preseeded:         true
image-preseeding:  4.005s
seed-completion:   –
preseed-system-key: {
  "apparmor-features": [
    "caps",
    "dbus",
    "domain",
    "file",
    "mount",
    "namespaces",
    "network",
    "network_v8",
    "policy",
    "ptrace",
    "query",
    "rlimit",
    "signal"
  ],
  "apparmor-parser-features": [
    "cap-audit-read",
    "cap-bpf",
    "include-if-exists",
    "mqueue",
    "qipcrtr-socket",
    "snapd-internal",
    "unsafe",
    "userns",
    "xdp"
  ],
  "apparmor-parser-mtime": 1692983915,
  "build-id": "55447a37514c4a317439786251326b5f762d31392f6f6b7835704d635279724779346e4d6e78414a6a2f6d647a5247354a536e6e38616e6c31636c5954612f38496435624e72466c744770475332794967704c",
  "cgroup-version": "2",
  "nfs-home": false,
  "overlay-root": "",
  "seccomp-compiler-version": "0a51bc642597bb018aeaaeea931b5cf033bb47d9 2.5.4 c3c9b282ef3c8dfcc3124b2aeaef62f56b813bfd21f8806b30a6c9dbc2e6e58d bpf-actlog",
  "seccomp-features": [
    "allow",
    "errno",
    "kill_process",
    "kill_thread",
    "log",
    "trace",
    "trap",
    "user_notif"
  ],
  "version": 10
}
seed-restart-system-key: {
  "apparmor-features": [
    "caps",
    "domain",
    "file",
    "mount",
    "namespaces",
    "network_v8",
    "policy",
    "ptrace",
    "query",
    "rlimit",
    "signal"
  ],
  "apparmor-parser-features": [
    "cap-audit-read",
    "cap-bpf",
    "include-if-exists",
    "mqueue",
    "qipcrtr-socket",
    "snapd-internal",
    "unsafe",
    "userns",
    "xdp"
  ],
  "apparmor-parser-mtime": 1692983915,
  "build-id": "55447a37514c4a317439786251326b5f762d31392f6f6b7835704d635279724779346e4d6e78414a6a2f6d647a5247354a536e6e38616e6c31636c5954612f38496435624e72466c744770475332794967704c",
  "cgroup-version": "2",
  "nfs-home": false,
  "overlay-root": "",
  "seccomp-compiler-version": "0a51bc642597bb018aeaaeea931b5cf033bb47d9 2.5.4 c3c9b282ef3c8dfcc3124b2aeaef62f56b813bfd21f8806b30a6c9dbc2e6e58d bpf-actlog",
  "seccomp-features": [
    "allow",
    "errno",
    "kill_process",
    "kill_thread",
    "log",
    "trace",
    "trap",
    "user_notif"
  ],
  "version": 10
}

systemctl status snap.lxd.activate.service

root@settling-monster:~# systemctl status snap.lxd.activate.service
× snap.lxd.activate.service
     Loaded: not-found (Reason: Unit snap.lxd.activate.service not found.)
     Active: failed (Result: exit-code) since Wed 2023-10-18 11:10:52 UTC; 1min 2s ago
   Main PID: 608 (code=exited, status=1/FAILURE)
        CPU: 36ms

Oct 18 11:10:52 settling-monster systemd[1]: Starting Service for snap application lxd.activate...
Oct 18 11:10:52 settling-monster systemd[1]: snap.lxd.activate.service: Main process exited, code=exited, status=1/FAILURE
Oct 18 11:10:52 settling-monster systemd[1]: snap.lxd.activate.service: Failed with result 'exit-code'.
Oct 18 11:10:52 settling-monster systemd[1]: Failed to start Service for snap application lxd.activate.

journalctl -xeu snap.lxd.activate.service

-- Boot 068d18bbbcca4cd5b6c10fafcaadde33 --
Oct 19 05:43:35 adequate-foal systemd[1]: Starting Service for snap application lxd.activate...
░░ Subject: A start job for unit snap.lxd.activate.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit snap.lxd.activate.service has begun execution.
░░ 
░░ The job identifier is 534.
Oct 19 05:43:36 adequate-foal systemd[1]: snap.lxd.activate.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ An ExecStart= process belonging to unit snap.lxd.activate.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Oct 19 05:43:36 adequate-foal systemd[1]: snap.lxd.activate.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit snap.lxd.activate.service has entered the 'failed' state with result 'exit-code'.
Oct 19 05:43:36 adequate-foal systemd[1]: Failed to start Service for snap application lxd.activate.
░░ Subject: A start job for unit snap.lxd.activate.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit snap.lxd.activate.service has finished with a failure.
░░ 
░░ The job identifier is 534 and the job result is failed.

Anything else we can run to get more info on this? :thinking: It’s incredibly frustrating to not be able to use the ubuntu:22.04 upstream image. There must be something on these particular desktop workstations (one Ubuntu 22.04, one Debian 12) which triggers this…

I even tried reinstalling and reconfiguring lxd from scratch (on that Debian 12 machine), using the default settings, to no avail.