New LXD image server available (images.lxd.canonical.com)

@equator8848 that is correct. Ubuntu server/cloud images are available from the various builtin remotes like ubuntu:, ubuntu-daily:, ubuntu-minimal: and ubuntu-minimal-daily:.

There is an exception for Ubuntu desktop images that are available through images:.

I can use lxc launch ubuntu-daily:focal to use ubuntu cloud image, but the image has a critical bug of cloud-init. Cloud-init status always reporting running Ā· Issue #5304 Ā· canonical/cloud-init Ā· GitHub :smiling_face_with_tear:

Any way to download previous version of Ubuntu cloud image? thank you

Hi,

you can install previous versions of the image using its fingerprint/sha256.

For example:

# SHA256 for "ubuntu-focal-daily-amd64-server-20240531" container image
#
lxc launch ubuntu-daily:004d24eef2e5b4c145038dc15bdade445a57991bfc6a33262a725d509bb33ef4 c1

Unfortunately, Iā€™m not aware of a good way to quickly find the fingerprints.
You can go to the product catalog (JSON file) of the given remote (here is ubuntu-daily). The fingerprint in field combined_squashfs_sha256 is for containers and in combined_disk1-img_sha256 is for VM images.

1 Like

I would personally like to have Lubuntu, and Edubuntu. But probably it would be useful to have all the official Ubuntu Flavours.

Thank you for your reply :slight_smile:

Iā€™m using this new images server to try and launch Rocky VMs (rockylinux/8/cloud), but they seem to be missing the LXD agent:
$ lxc launch images:rockylinux/8/cloud --vm
Creating the instance
Instance name is: cosmic-wasp
Starting cosmic-wasp

(wait a few minutes)

$ lxc exec cosmic-wasp bash
Error: LXD VM agent isnā€™t currently running

I then tried injecting my ssh key via lxd profile, setting both cloud-init.user-data and user.user-data for users ā€˜rockyā€™ or ā€˜rootā€™. But neither of those seem to work either.

Anyone else trying Rocky images and having trouble accessing the newly launched (VM) instances?

lxc launch images:rockylinux/8/cloud --vm --console rocky8 works for me. The VM reboots as part of installing the lxd-agent and when it comes back, it works.

I copy and pasted your launch command. I see some failures in the consoleā€¦ but no way of knowing what is causing the failure. Any ideas?

[FAILED] Failed to start LXD - agent.
See 'systemctl status lxd-agent.service' for details.
[  OK  ] Reached target Cloud-init target.

Rocky Linux 8.10 (Green Obsidian)
Kernel 4.18.0-553.8.1.el8_10.x86_64 on an x86_64

rocky8 login:

I exported and mounted the image to inspect /var/log/messages.

Jul  9 04:52:07 rocky8 kernel: virtio-fs: tag <config> not found
Jul  9 04:52:07 rocky8 lxd-agent-setup[721]: Couldn't mount 9p or cdrom, failing
Jul  9 04:52:07 rocky8 systemd[1]: lxd-agent.service: Control process exited, code=exited status=1
Jul  9 04:52:07 rocky8 systemd[1]: lxd-agent.service: Failed with result 'exit-code'.
Jul  9 04:52:07 rocky8 systemd[1]: Failed to start LXD - agent.

some posts online suggest this might be to do with the VMā€™s kernelā€¦ but then I wouldnā€™t be the only one experiencing the issue. something strange in my config or lxd version?

What do you run as the host OS?

raf@lxd1:~$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
raf@lxd1:~$ uname -a
Linux lxd1 5.15.0-107-generic #117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
raf@lxd1:~$ snap info lxd | grep ^tracking 
tracking:     latest/stable
raf@lxd1:~$

This is the issue because LXD should be exporting the config drive that contains the keys that the lxd-agent needs for communicating over vsock to the LXD host.

CentOS clones donā€™t have 9p kernel module and so cannot use that.

Can you see any virtiofsd processes running on the host?

What does snap list | grep lxd show?

You could also try refreshing to latest/candidate as that contains the up coming LXD 6.1 release:

sudo snap refresh lxd --channel=latest/candidate

Iā€™ve just tried this on my Jammy test machine:

snap list | grep lxd
lxd        5.21.1-2d13beb  28463  latest/stable  canonical**  -

uname -a
Linux 5.15.0-113-generic
lxc launch images:rockylinux/8/cloud --vm
lxc ls
+----------------+---------+------------------------+------+-----------------+-----------+
|      NAME      |  STATE  |          IPV4          | IPV6 |      TYPE       | SNAPSHOTS |
+----------------+---------+------------------------+------+-----------------+-----------+
| equal-bonefish | RUNNING | 10.63.202.220 (enp5s0) |      | VIRTUAL-MACHINE | 0         |
+----------------+---------+------------------------+------+-----------------+-----------+

lxc shell equal-bonefish
[root@equal-bonefish ~]#

Please can you check what image fingerprint you have:

lxc image list
+-------+--------------+--------+------------------------------------+--------------+-----------------+-----------+-----------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |            DESCRIPTION             | ARCHITECTURE |      TYPE       |   SIZE    |         UPLOAD DATE         |
+-------+--------------+--------+------------------------------------+--------------+-----------------+-----------+-----------------------------+
|       | 2d70077a9173 | no     | RockyLinux 8 amd64 (20240704_0004) | x86_64       | VIRTUAL-MACHINE | 869.64MiB | Jul 9, 2024 at 8:11am (UTC) |
+-------+--------------+--------+------------------------------------+--------------+-----------------+-----------+-----------------------------+

Same image and lxd version, only difference appears to be kernel.

raf@lxd1:~$ snap list | grep lxd
lxd     5.21.1-2d13beb  28463  latest/stable  canonical**  -
raf@lxd1:~$ lxc image list | grep Rocky
|            | 2d70077a9173 | no     | RockyLinux 8 amd64 (20240704_0004) | x86_64       | VIRTUAL-MACHINE | 869.64MiB  | Jul 9, 2024 at 12:54am (UTC) |
raf@lxd1:~$

I donā€™t see any virtiofsd processes running on the host where the rocky VM is hosted, though there are other (ubuntu) VMs that function as expected with lxd agent running.

Another host in the cluster does have virtiofsd processes.

Can you run lxc monitor --pretty and then in another window stop and start the problem VM again.

I see these error lines in the monitor:

WARNING[2024-07-09T11:24:02Z] Unable to use virtio-fs for config drive, using 9p as a fallback  err="Stateful migration unsupported" instance=rocky8 instanceType=virtual-machine project=random
WARNING[2024-07-09T11:24:02Z] Unable to use virtio-fs for config drive, using 9p as a fallback  err="Stateful migration unsupported" instance=rocky8 instanceType=virtual-machine project=random

fyi i have set migration stateful in the cluster configuration.

Those messages also occur for ubuntu VMs, but presumably the fallback to 9p succeeds on the ubuntu kernels?

Ah so in that case, you cant use Rocky images with the lxd-agent because they dont have 9p kernel module. This is because virtio-fs doesnā€™t support live migration sadly.

Yes, thanks for confirming. Just to close the loop, in case you have set migration.stateful by default in your lxd configuration this works in terms of the lxd-agent starting, and getting a shell:

lxc launch images:rockylinux/8/cloud --vm rocky8 --config migration.stateful=false

Of course, it will not be able to be live migrated.

1 Like

Thank you for having addressed the image server replacement issue! I thought this was long resolved, but ran into the issue just now. Hereā€™s some feedback on a user story that just happened to me and seems suboptimal.

On a fairly recently created Ubuntu 20.04 VM (official cloud image serial 20240710) the lxd snap is installed, 4.0.9-a29c6f1 (24061) tracking 4.0/stable/ubuntu-20.04. This gave me the old images server by default, which (as you probably know) silently returns zero results.

I expected the snaps to have been updated to use the new image server by now. Since the old one will not return any results at all, I think this could happen without regressing users?

I then found Remote image servers which makes no mention of the extra steps required for older, still supported lxd snap tracks.

Finally, this post also didnā€™t contain precise instructionsā€”I had to remove the old images: remote first, which involved guessing the command (or looking it up) rather than it being presented here.

Suggestions:

  1. Older lxd snap tracks should be updated to provide the new images: server for fresh install and also when upgrading to the latest stable, since the old one doesnā€™t work at all now.
  2. Until the above is done, the Remote image servers documentation page should link to this post (or an equivalent).
  3. The post should contain exact instructions on what the user should do, including the lxc rm images step.

HTH!

1 Like

This is already in the works and is expected to happen soon. The updates are already in 4.0/edge.

2 Likes

@rbasak LXD 4.0.10 LTS has been released with support for the new image server:

1 Like