Cloud-init fails to complete in ubuntu 22.04 and is completely broken in 20.04

My cloud-init works fine with Ubuntu 24.04, Debian 12/13, Alma 8 and 9, Fedora 43.

4-node LXD 6.5 cluster

But in Ubuntu 22.04 LXCs cloud-init cannot finish because the cloud-config.service and cloud-final.service depend on snapd.seeded.service and that service is failing

root@ubuntu-2204:~# systemctl status snapd.seeded.service
● snapd.seeded.service - Wait until snapd is fully seeded
     Loaded: loaded (/lib/systemd/system/snapd.seeded.service; enabled; vendor preset: enabled)
     Active: activating (start) since Sun 2025-10-26 08:56:14 UTC; 7min ago
   Main PID: 226 (snap)
      Tasks: 14 (limit: 18570)
     Memory: 21.3M
        CPU: 1.077s
     CGroup: /system.slice/snapd.seeded.service
             └─226 /usr/bin/snap wait system seed.loaded

Oct 26 08:56:14 ubuntu-2204 systemd[1]: Starting Wait until snapd is fully seeded...

My instances have working network connectivity/dns. cloud-init in ubuntu 24.04, debian 12/13, alma 8/9 and fedora works fine.

Hoping @sdeziel1 can help here :slight_smile:

I wanted to confirm if it works on 20.04 but something else is very broken there.

The cloud-init data isn’t even made available inside the instance.

When I launch an ubuntu 20.04 LXC with cloud-init data inherited from the default project, the data isn’t inserted into the instance. While testing I’ve had occasions where all of a sudden the cloud-init data appeared inside the instance after modifying the cloud-init config specified on the instance itself (not inherited).
Unfortunately I’ve not been able to reproduce this. I tried modifying the cloud-init user and vendor data in the 20.04 instance and it’s still not found inside the instance.

root@diamond:/home/vos# lxc exec ubuntu-2004 --project sandbox -- cat /var/lib/cloud/instance/vendor-data.txt
#cloud-config
{}

root@diamond:/home/vos# lxc config show ubuntu-2004 --project sandbox
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 20.04 LTS amd64 (release) (20250624)
  image.label: release
  image.os: ubuntu
  image.release: focal
  image.serial: "20250624"
  image.type: squashfs
  image.version: "20.04"
  user.cluster: unknown
  user.fqdn: ubuntu-2004.sandbox.lab.example.com
  user.project: sandbox
  volatile.apply_template: create
  volatile.base_image: 94b4b0055aca5c16e801aa20c08aba22f387af228a3c578ab5ef9c8a342d5832
  volatile.cloud-init.instance-id: 0c9e95db-89d5-4d16-b7ef-faa42c5c563e
  volatile.eth2.hwaddr: 00:16:3e:49:f2:7b
  volatile.eth2.name: eth0
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.uuid: a0945ff5-ae3d-4123-925b-b85dc9f954f7
  volatile.uuid.generation: a0945ff5-ae3d-4123-925b-b85dc9f954f7
devices: {}
ephemeral: false
profiles:
- default
- storage-local
- network-sandbox
stateful: false
description: ""
root@diamond:/home/vos#
root@diamond:/home/vos#
root@diamond:/home/vos# lxc exec ubuntu-2004 --project sandbox -- cat /var/lib/cloud/instance/vendor-data.txt
#cloud-config
{}

root@diamond:/home/vos# lxc config show ubuntu-2004 --project sandbox --expanded
architecture: x86_64
config:
  cloud-init.vendor-data: |-
    #cloud-config
    timezone: Europe/Amsterdam
    package_upgrade: false
    packages:
        - openssh-server
        - curl
    users:
      - name: ansible
        uid: 600
        no_user_group: true
        primary_group: nogroup
        groups: sudo
        shell: /bin/bash
        sudo: "ALL=(ALL) NOPASSWD:ALL"
        ssh_authorized_keys:
          - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMj1/LsMgJf3ci0axbBJd39AlsTTzzvJ2r0w3Uw/mAHA ansible@vos.dev
          - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJf9y9WBKU+TpLTPjkZiqCVfzA+DXRdB/Gf7dnpCCukV Semaphore
  cluster.evacuate: stop
  image.architecture: amd64
  image.description: ubuntu 20.04 LTS amd64 (release) (20250624)
  image.label: release
  image.os: ubuntu
  image.release: focal
  image.serial: "20250624"
  image.type: squashfs
  image.version: "20.04"
  user.cluster: unknown
  user.fqdn: ubuntu-2004.sandbox.lab.example.com
  user.project: sandbox
  volatile.apply_template: create
  volatile.base_image: 94b4b0055aca5c16e801aa20c08aba22f387af228a3c578ab5ef9c8a342d5832
  volatile.cloud-init.instance-id: 0c9e95db-89d5-4d16-b7ef-faa42c5c563e
  volatile.eth2.hwaddr: 00:16:3e:49:f2:7b
  volatile.eth2.name: eth0
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.uuid: a0945ff5-ae3d-4123-925b-b85dc9f954f7
  volatile.uuid.generation: a0945ff5-ae3d-4123-925b-b85dc9f954f7
devices:
  eth2:
    network: sandbox
    type: nic
  root:
    path: /
    pool: local
    size: 15GiB
    type: disk
ephemeral: false
profiles:
- default
- storage-local
- network-sandbox
stateful: false
description: ""
root@diamond:/home/vos# lxc exec ubuntu-2004 --project sandbox -- cat /var/lib/cloud/instance/vendor-data.txt
#cloud-config
{}

root@diamond:/home/vos# lxc exec ubuntu-2004 --project sandbox -- cat /var/lib/cloud/instance/user-data.txt
#cloud-config
{}

Some steps to make the issue clearly visible:
I’m launching the 3 ubuntu LTS instances with identical LXD config.
cloud-init config is empty in 20.04, never reaching done in 22.04 and works in 24.04

root@diamond:/home/vos# lxc launch ubuntu:20.04 u1 -p default -p storage-local -p network-sandbox --project sandbox
Launching u1
root@diamond:/home/vos# lxc exec u1 --project sandbox -- cloud-init status
status: done
root@diamond:/home/vos# lxc exec u1 --project sandbox -- id ansible
id: ‘ansible’: no such user


root@diamond:/home/vos# lxc launch ubuntu:22.04 u2 -p default -p storage-local -p network-sandbox --project sandbox
Launching u2
root@diamond:/home/vos# lxc exec u2 --project sandbox -- cloud-init status
status: running
root@diamond:/home/vos# lxc exec u2 --project sandbox -- id ansible
uid=600(ansible) gid=65534(nogroup) groups=65534(nogroup),27(sudo)


root@diamond:/home/vos# lxc launch ubuntu:24.04 u3 -p default -p storage-local -p network-sandbox --project sandbox
Launching u3
root@diamond:/home/vos# lxc exec u3 --project sandbox -- cloud-init status
status: done
root@diamond:/home/vos# lxc exec u3 --project sandbox -- id ansible
uid=600(ansible) gid=65534(nogroup) groups=65534(nogroup),27(sudo)

root@diamond:/home/vos# lxc exec u2 --project sandbox -- cloud-init status
status: running
root@diamond:/home/vos# lxc exec u1 --project sandbox -- cat /var/lib/cloud/instance/vendor-data.txt
#cloud-config
{}

root@diamond:/home/vos# lxc exec u2 --project sandbox -- cat /var/lib/cloud/instance/vendor-data.txt
#cloud-config
package_upgrade: false
<..>

root@diamond:/home/vos# lxc exec u3 --project sandbox -- cat /var/lib/cloud/instance/vendor-data.txt
#cloud-config
package_upgrade: false
<..>

When I create an instance and populate either user or vendor data it sometimes does work but i am no longer able to reproduce it.

root@diamond:/home/vos# lxc delete u4 --project sandbox --force
root@diamond:/home/vos# lxc init ubuntu:20.04 u4 -p default -p storage-local -p network-sandbox --project sandbox
Creating u4
root@diamond:/home/vos# lxc config edit u4 --project sandbox
root@diamond:/home/vos# lxc config show u4 --project sandbox
architecture: x86_64
config:
  cloud-init.user-data: |
    #cloud-init
    package_upgrade: true
  image.architecture: amd64
  image.description: ubuntu 20.04 LTS amd64 (release) (20250624)
  image.label: release
  image.os: ubuntu
  image.release: focal
<..>
root@diamond:/home/vos# lxc start u4 --project sandbox
root@diamond:/home/vos#
root@diamond:/home/vos# lxc exec u4 --project sandbox -- cloud-init status
status: done
root@diamond:/home/vos# lxc exec u4 --project sandbox -- cat /var/lib/cloud/instance/user-data.txt
#cloud-config
{}

Could you please provide the cloud-init status --wait --long output from an instance that is broken?

Hey thanks for the reply :slight_smile:

root@u4:~# cloud-init status --wait --long
status: done
extended_status: done
boot_status_code: enabled-by-generator
last_update: Thu, 01 Jan 1970 00:00:04 +0000
detail: DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net]
errors: []
recoverable_errors: {}

The problem is that the whole cloud-init config is never applied because it’s not in the instance.

root@u4:~# cat /var/lib/cloud/seed/nocloud-net/user-data 
#cloud-config
{}

root@u4:~# cat /var/lib/cloud/seed/nocloud-net/vendor-data 
#cloud-config
{}

root@u4:~# cat /var/lib/cloud/seed/nocloud-net/network-config 
version: 1
config:
    - type: physical
      name: eth0
      subnets:
          - type: dhcp
            control: auto

The instance:

root@emerald:/home/vos# lxc config show u4 --project sandbox -e
architecture: x86_64
config:
  cloud-init.user-data: |
    #cloud-init
    package_upgrade: false
  cloud-init.vendor-data: |-
    #cloud-config
    timezone: Europe/Amsterdam
    package_upgrade: false
    packages:
        - openssh-server
        - curl
    users:
      - name: ansible
        uid: 600
        no_user_group: true
        primary_group: nogroup
        groups: sudo
        shell: /bin/bash
        sudo: "ALL=(ALL) NOPASSWD:ALL"
        ssh_authorized_keys:
          - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMj1/LsMgJf3bBJd39AlsTTzzvJ2r0w3Uw/mAHA ansible@example.com
          - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJf9y9WBKUPjkZiqCVfzA+DXRdB/Gf7dnpCCukV Semaphore
  cluster.evacuate: stop
  image.architecture: amd64
  image.description: ubuntu 20.04 LTS amd64 (release) (20250624)
  image.label: release
  image.os: ubuntu
  image.release: focal
  image.serial: "20250624"
  image.type: squashfs
  image.version: "20.04"
  user.cluster: lab
  user.fqdn: u4.sandbox.lab.example.com
  user.project: sandbox
  volatile.base_image: 94b4b0055aca5c16e801aa20c08aba22f387af228a3c578ab5ef9c8a342d5832
  volatile.cloud-init.instance-id: 2a608a6f-994a-4fc4-9665-fb4f1b014927
  volatile.eth2.host_name: veth58d325ff
  volatile.eth2.hwaddr: 00:16:3e:f3:c0:bc
  volatile.eth2.name: eth0
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: 370d20c9-d5da-41f4-b522-da68f09bd27e
  volatile.uuid.generation: 370d20c9-d5da-41f4-b522-da68f09bd27e
devices:
  eth2:
    network: sandbox
    type: nic
  root:
    path: /
    pool: local
    size: 15GiB
    type: disk
ephemeral: false
profiles:
- default
- storage-local
- network-sandbox
stateful: false
description: ""

The DataSourceNoCloud indicates that cloud-init couldn’t get the config from LXD. Normally, you’d expect to see DataSourceLXD. This looks like doc: ubuntu-minimal VM images do not support cloud-init as claimed · Issue #14605 · canonical/lxd · GitHub

I double checked but I really am not using the minimal image for 20.04. I’m using the fullblown image.

It’s old (20.04) and comes with an old cloud-init version that doesn’t have LXD specific support iirc.

Oh right, with 20.04, I think you have to use the legacy config keys like user.user-data.

Here’s my local test profile:

$ lxc profile show test
name: test
description: ""
config:
  cloud-init.user-data: |-
    #cloud-config
    write_files:
    - content: "cloud-init.user-data\n"
      path: /root/cloud-init.user-data
  user.user-data: |-
    #cloud-config
    write_files:
    - content: "user.user-data\n"
      path: /root/user.user-data
devices: {}
used_by:
- /1.0/instances/u4
project: default

And here’s what I get:

$ lxc launch ubuntu:20.04 u4 -p test -p default
$ lxc exec u4 -- cloud-init status --wait --long
...................status: done
extended_status: done
boot_status_code: enabled-by-generator
last_update: Thu, 01 Jan 1970 00:00:09 +0000
detail: DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net]
errors: []
recoverable_errors: {}
$ lxc exec u4 -- ls -l
total 1
drwx------ 3 root root  3 Oct 29 15:54 snap
-rw-r--r-- 1 root root 15 Oct 29 15:54 user.user-data
1 Like