Failed to start Service for snap application lxd.activate

chrome0 · November 24, 2025, 6:46pm

Hey hey

I have an intermittent fault in a container where I can see the lxd activation failing

Nov 20 19:22:33 juju-4dfde6-6 systemd[1]: snap.lxd.activate.service: Main process exited, code=exited, status=1/FAILURE
Nov 20 19:22:33 juju-4dfde6-6 systemd[1]: snap.lxd.activate.service: Failed with result 'exit-code'.
Nov 20 19:22:33 juju-4dfde6-6 systemd[1]: Failed to start Service for snap application lxd.activate.
Nov 20 19:22:33 juju-4dfde6-6 systemd[1]: Cannot find unit for notify message of PID 156292, ignoring.
Nov 20 19:22:33 juju-4dfde6-6 systemd[1]: snap.lxd.user-daemon.unix.socket: Deactivated successfully.
Nov 20 19:22:33 juju-4dfde6-6 systemd[1]: Closed Socket unix for snap application lxd.user-daemon.
Nov 20 19:22:33 juju-4dfde6-6 systemd[1]: snap.lxd.daemon.unix.socket: Deactivated successfully.
Nov 20 19:22:33 juju-4dfde6-6 systemd[1]: Closed Socket unix for snap application lxd.daemon.
Nov 20 19:22:33 juju-4dfde6-6 systemd[1]: Reloading.
Nov 20 19:22:33 juju-4dfde6-6 systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Nov 20 19:22:33 juju-4dfde6-6 systemd[1]: /lib/systemd/system/snapd.service:23: Unknown key name 'RestartMode' in section 'Service', ignoring.
Nov 20 19:22:33 juju-4dfde6-6 snapd[155772]: taskrunner.go:304: Change 180 task (Start snap "lxd" (35819) services) failed: systemctl command [start snap.lxd.activate.service] failed with exit status 1: stderr:
Nov 20 19:22:33 juju-4dfde6-6 snapd[155772]: Job for snap.lxd.activate.service failed because the control process exited with error code.
Nov 20 19:22:33 juju-4dfde6-6 snapd[155772]: See "systemctl status snap.lxd.activate.service" and "journalctl -xeu snap.lxd.activate.service" for details.
Nov 20 19:22:35 juju-4dfde6-6 snapd[155772]: backend.go:285: reloading profiles of snap-confine provided by the system snap
Nov 20 19:22:36 juju-4dfde6-6 systemd[1]: run-snapd-ns-lxd.mnt.mount: Deactivated successfully.
Nov 20 19:22:36 juju-4dfde6-6 snapd[155772]: services.go:1128: RemoveSnapServices - socket snap.lxd.daemon.unix.socket
Nov 20 19:22:36 juju-4dfde6-6 snapd[155772]: services.go:1152: RemoveSnapServices - disabling snap.lxd.daemon.service
Nov 20 19:22:36 juju-4dfde6-6 snapd[155772]: services.go:1128: RemoveSnapServices - socket snap.lxd.user-daemon.unix.socket
Nov 20 19:22:36 juju-4dfde6-6 snapd[155772]: services.go:1152: RemoveSnapServices - disabling snap.lxd.user-daemon.service
Nov 20 19:22:36 juju-4dfde6-6 snapd[155772]: services.go:1152: RemoveSnapServices - disabling snap.lxd.activate.service
Nov 20 19:22:36 juju-4dfde6-6 systemd[1]: Reloading.
Nov 20 19:22:36 juju-4dfde6-6 systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Nov 20 19:22:36 juju-4dfde6-6 systemd[1]: /lib/systemd/system/snapd.service:23: Unknown key name 'RestartMode' in section 'Service', ignoring.
Nov 20 19:22:37 juju-4dfde6-6 snapd[155772]: copydata.go:98: Cannot remove common data directories for "lxd": unlinkat /var/snap/lxd/common/var/lib/lxcfs/proc/cpuinfo: function not implemented
Nov 20 19:22:37 juju-4dfde6-6 snapd[155772]: taskrunner.go:304: Change 180 task (Copy snap "lxd" data) failed: unlinkat /var/snap/lxd/common/var/lib/lxcfs/proc/cpuinfo: function not implemented
Nov 20 19:22:37 juju-4dfde6-6 systemd[1]: snap-lxd-35819.mount: Deactivated successfully.
Nov 20 19:22:37 juju-4dfde6-6 systemd[1]: snap-lxd-35819.mount: Unit process 156020 (snapfuse) remains running after unit stopped.
Nov 20 19:22:37 juju-4dfde6-6 systemd[1]: Reloading.
Nov 20 19:22:37 juju-4dfde6-6 systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Nov 20 19:22:37 juju-4dfde6-6 systemd[1]: /lib/systemd/system/snapd.service:23: Unknown key name 'RestartMode' in section 'Service', ignoring.
Nov 20 19:22:40 juju-4dfde6-6 systemd[1]: snap-core20-2682.mount: Deactivated successfully.
Nov 20 19:22:40 juju-4dfde6-6 systemd[1]: snap-core20-2682.mount: Unit process 155927 (snapfuse) remains running after unit stopped.
Nov 20 19:22:40 juju-4dfde6-6 systemd[1]: Reloading.
Nov 20 19:22:40 juju-4dfde6-6 systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Nov 20 19:22:40 juju-4dfde6-6 systemd[1]: /lib/systemd/system/snapd.service:23: Unknown key name 'RestartMode' in section 'Service', ignoring.
Nov 20 19:22:44 juju-4dfde6-6 snapd[155772]: backend.go:285: reloading profiles of snap-confine provided by the system snap
Nov 20 19:22:44 juju-4dfde6-6 snapd[155772]: daemon.go:556: gracefully waiting for running hooks
Nov 20 19:22:44 juju-4dfde6-6 snapd[155772]: daemon.go:558: done waiting for running hooks
Nov 20 19:22:47 juju-4dfde6-6 snapd[155772]: overlord.go:532: Released state lock file
Nov 20 19:22:47 juju-4dfde6-6 systemd[1]: snapd.service: Deactivated successfully.
Nov 20 19:22:47 juju-4dfde6-6 systemd[1]: snapd.service: Consumed 26.577s CPU time.
Nov 20 19:22:47 juju-4dfde6-6 systemd[1]: snapd.service: Scheduled restart job, restart counter is at 180.
Nov 20 19:22:47 juju-4dfde6-6 systemd[1]: Stopped Snap Daemon.
Nov 20 19:22:47 juju-4dfde6-6 systemd[1]: snapd.service: Consumed 26.577s CPU time.
Nov 20 19:22:47 juju-4dfde6-6 systemd[1]: Starting Snap Daemon...
Nov 20 19:22:47 juju-4dfde6-6 snapd[156644]: overlord.go:288: Acquiring state lock file
Nov 20 19:22:47 juju-4dfde6-6 snapd[156644]: overlord.go:293: Acquired state lock file
Nov 20 19:22:48 juju-4dfde6-6 snapd[156644]: daemon.go:276: started snapd/2.72+ubuntu22.04 (series 16; classic) ubuntu/22.04 (amd64) linux/6.14.0-35-generic.
Nov 20 19:22:48 juju-4dfde6-6 systemd[1]: tmp-syscheck\x2dmountpoint\x2d3210170349.mount: Deactivated successfully.
Nov 20 19:22:48 juju-4dfde6-6 snapd[156644]: daemon.go:370: adjusting startup timeout by 35s (pessimistic estimate of 30s plus 5s per snap)
Nov 20 19:22:48 juju-4dfde6-6 snapd[156644]: backends.go:70: AppArmor status: apparmor is enabled and all features are available
Nov 20 19:22:48 juju-4dfde6-6 snapd[156644]: backend.go:145: reloading profiles for snap-confine
Nov 20 19:22:48 juju-4dfde6-6 systemd[1]: Started Snap Daemon.
Nov 20 19:22:50 juju-4dfde6-6 systemd[1]: Reloading.
Nov 20 19:22:51 juju-4dfde6-6 systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Nov 20 19:22:51 juju-4dfde6-6 systemd[1]: /lib/systemd/system/snapd.service:23: Unknown key name 'RestartMode' in section 'Service', ignoring.
Nov 20 19:22:51 juju-4dfde6-6 systemd[1]: Mounting Mount unit for core20, revision 2682...
Nov 20 19:22:51 juju-4dfde6-6 systemd[1]: Mounted Mount unit for core20, revision 2682.
Nov 20 19:22:54 juju-4dfde6-6 systemd[1]: Reloading.
Nov 20 19:22:54 juju-4dfde6-6 systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Nov 20 19:22:54 juju-4dfde6-6 systemd[1]: /lib/systemd/system/snapd.service:23: Unknown key name 'RestartMode' in section 'Service', ignoring.
Nov 20 19:22:54 juju-4dfde6-6 systemd[1]: Mounting Mount unit for lxd, revision 35819...
Nov 20 19:22:54 juju-4dfde6-6 systemd[1]: Mounted Mount unit for lxd, revision 35819.
Nov 20 19:22:56 juju-4dfde6-6 systemd[1]: Reloading.
Nov 20 19:22:56 juju-4dfde6-6 systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Nov 20 19:22:56 juju-4dfde6-6 systemd[1]: /lib/systemd/system/snapd.service:23: Unknown key name 'RestartMode' in section 'Service', ignoring.
Nov 20 19:22:58 juju-4dfde6-6 snapd[156644]: backend.go:285: reloading profiles of snap-confine provided by the system snap
Nov 20 19:22:59 juju-4dfde6-6 systemd[1]: Started snap.lxd.hook.install-6288ece5-11a9-4f98-bbac-f33f19d06d34.scope.
Nov 20 19:22:59 juju-4dfde6-6 systemd[1]: tmp-snap.rootfs_qROFjw.mount: Deactivated successfully.
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: snap.lxd.hook.install-6288ece5-11a9-4f98-bbac-f33f19d06d34.scope: Deactivated successfully.
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: Reloading.
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: /lib/systemd/system/snapd.service:23: Unknown key name 'RestartMode' in section 'Service', ignoring.
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: Listening on Socket unix for snap application lxd.daemon.
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: Listening on Socket unix for snap application lxd.user-daemon.
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: Starting Service for snap application lxd.activate...
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: snap.lxd.activate.service: Main process exited, code=exited, status=1/FAILURE
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: snap.lxd.activate.service: Failed with result 'exit-code'.
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: Failed to start Service for snap application lxd.activate.
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: Cannot find unit for notify message of PID 157165, ignoring.
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: snap.lxd.user-daemon.unix.socket: Deactivated successfully.
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: Closed Socket unix for snap application lxd.user-daemon.
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: snap.lxd.daemon.unix.socket: Deactivated successfully.
Nov 20 19:23:00 juju-4dfde6-6 systemd[1]: Closed Socket unix for snap application lxd.daemon.

Any ideas what the lxd service might be unhappy about? Fwiw details are in the juju crashdump here: https://github.com/canonical/ceph-charms/actions/runs/19473395591/artifacts/4632076230 (the failing unit is the mysql/0 one)

sdeziel1 · November 24, 2025, 7:20pm

@chrome0 it’s a bit of a long shot but is it possible that it happens on a runner with kernel 6.14?

There is a known problem when using this kernel with a 22.04 container. To check if it’s the same issue, please check if systemctl --failed reports the snapd.seeded.service as failed in the container.

chrome0 · November 27, 2025, 6:27pm

Yes that seems to be the case, the GH runner has 6.14.0-36-generic

In the container I see

lxc exec juju-03d877-1 -- sudo systemctl --failed
  UNIT                      LOAD   ACTIVE SUB    DESCRIPTION                              
● snap.lxd.activate.service loaded failed failed Service for snap application lxd.activate

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
1 loaded units listed.

lxc exec juju-03d877-1 -- sudo systemctl status snapd.seeded.service
● snapd.seeded.service - Wait until snapd is fully seeded
     Loaded: loaded (/lib/systemd/system/snapd.seeded.service; enabled; vendor preset: enabled)
     Active: activating (start) since Thu 2025-11-27 17:37:31 UTC; 48min ago
   Main PID: 342 (snap)
      Tasks: 14 (limit: 77062)
     Memory: 11.5M
        CPU: 3.696s
     CGroup: /system.slice/snapd.seeded.service
             └─342 /usr/bin/snap wait system seed.loaded

Nov 27 17:37:31 juju-03d877-1 systemd[1]: Starting Wait until snapd is fully seeded...

Any ideas on how to work around this? I unfort. can’t really change the GH runner kernel. Would running this in a VM instead of a container help?

sdeziel1 · November 27, 2025, 7:15pm

I think deploying to a 24.04 container or a 22.04 VM is worth a try if you can. It seems like we’ve only got issues with that 6.14 kernel so if you find a way to avoid it that sounds ideal

chrome0 · November 28, 2025, 7:52am

Ack, will give 22.04 VM a try (need to keep it on jammy for this test). For my tracking, do you happen to have a bug # at hand for this issue? cheers!

tomp · November 28, 2025, 1:45pm

This is the one I believe:

https://bugs.launchpad.net/snapd/+bug/2127244/