Instance not using image profiles

ibrummel · April 17, 2025, 7:35pm

I have an image that has a custom profile on it.

 # lxc image show remote:bb814da8419c443a56db7a4f86b3a9cd5f8c34e464fadd1112b79aef25304a52
auto_update: false
properties:
  description: my-custom-image
  os: ubuntu
  release: jammy
public: false
expires_at: 1969-12-31T18:00:00-06:00
profiles:
- default
- vm-bios

When I create an instance using the image, the custom vm-bios profile is not being utilized.

 # lxc init remote:bb814da8419c443a56db7a4f86b3a9cd5f8c34e464fadd1112b79aef25304a52 foo
Creating foo
 # lxc config show foo
architecture: x86_64
config:
  image.description: my-custom-image
  image.os: ubuntu
  image.release: jammy
  volatile.apply_template: create
  volatile.base_image: bb814da8419c443a56db7a4f86b3a9cd5f8c34e464fadd1112b79aef25304a52
  volatile.cloud-init.instance-id: 6b43f061-5cbe-41ff-845e-35e2fb9718cf
  volatile.eth0.hwaddr: 00:16:3e:c9:f3:f7
  volatile.uuid: 6562b131-9691-4bb1-bc18-c8f07d371ea9
  volatile.uuid.generation: 6562b131-9691-4bb1-bc18-c8f07d371ea9
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

The profile is on the local server that LXD is running on

 # lxc profile show vm-bios
name: vm-bios
description: enable VMs to boot from seabios (bios) instead of uefi
config:
  raw.apparmor: /var/lib/snapd/hostfs/usr/share/seabios/bios-256k.bin r,
  raw.qemu: -bios /var/lib/snapd/hostfs/usr/share/seabios/bios-256k.bin
  security.secureboot: "false"
devices: {}
used_by: []

Am I misreading that instances should use profiles from the image?

LXD version: 5.21.3 LTS

tomp · April 17, 2025, 7:47pm

Just as an aside, separate from the main issue, you don’t need those raw. settings as you can just use security.csm=true to get seabios boot mode.

nikita-mezhenskyi · April 22, 2025, 10:25pm

Hi @ibrummel ,

You are correct that instances should use profiles associated with the image. However, profiles are local to the LXD server and its cluster (if any).

I was able to reproduce the issue you’ve encountered using the following set up:

One LXD server is acting as the “remote” and has a locally saved image and a custom profile attached to it.
Another LXD server uses the image from the “remote” to create an instance. However, this LXD server does not have the same custom profile created locally.

To work around this issue, I had to create an identical profile on all the servers that use images from the “remote”.

Could you please provide more details about your set up? Does it look similar to what I described?

tomp · April 23, 2025, 7:11am

Yes this sounds like expected behaviour. The profile will need to exist in the local server/cluster.

Thanks for looking into it @nikita-mezhenskyi

ibrummel · April 23, 2025, 12:37pm

Thanks for the response. My setup is very similar. I have a remote LXD server that I store a lot of images on. Both the remote LXD “imageserver” and my local LXD server both have the custom profile:

[PROD]root@cihost2 ~
 # lxc profile show vm-bios
config:
  raw.apparmor: /var/lib/snapd/hostfs/usr/share/seabios/bios-256k.bin r,
  raw.qemu: -bios /var/lib/snapd/hostfs/usr/share/seabios/bios-256k.bin
  security.secureboot: "false"
description: enable VMs to boot from seabios (bios) instead of uefi
devices: {}
name: vm-bios
used_by: []

Yet when creating the instance on the LXD server that has the vm-bios profile, that profile is not attached to the instance.

root@server ~
 # lxc profile show vm-bios
config:
  raw.apparmor: /var/lib/snapd/hostfs/usr/share/seabios/bios-256k.bin r,
  raw.qemu: -bios /var/lib/snapd/hostfs/usr/share/seabios/bios-256k.bin
  security.secureboot: "false"
description: enable VMs to boot from seabios (bios) instead of uefi
devices: {}
name: vm-bios
used_by: []
root@server ~
 # lxc image info lxdimages:da8fc60fb4e294f791b2c6dd427a8b0f3b4f5e4a45a4dea7d1e62a64b3ff12a6
Fingerprint: da8fc60fb4e294f791b2c6dd427a8b0f3b4f5e4a45a4dea7d1e62a64b3ff12a6
Size: 4133.88MiB
Architecture: x86_64
Type: virtual-machine
Public: no
Timestamps:
    Created: 2025/04/23 09:56 UTC
    Uploaded: 2025/04/23 12:33 UTC
    Expires: never
    Last used: never
Properties:
    os: ubuntu
    release: jammy
    description: server-netbox.yml-jammy-latest
Aliases:
    - server-netbox.yml-jammy-latest
Cached: no
Auto update: disabled
Profiles:
    - default
    - vm-bios
root@server ~
 # lxc launch lxdimages:da8fc60fb4e294f791b2c6dd427a8b0f3b4f5e4a45a4dea7d1e62a64b3ff12a6 foo
Creating foo
Starting foo
root@server ~
 # lxc config show foo
architecture: x86_64
config:
  image.description: server-netbox.yml-jammy-latest
  image.os: ubuntu
  image.release: jammy
  volatile.base_image: da8fc60fb4e294f791b2c6dd427a8b0f3b4f5e4a45a4dea7d1e62a64b3ff12a6
  volatile.cloud-init.instance-id: 373d0f98-d50c-4ba1-8e3c-e2bfbb889099
  volatile.eth0.host_name: tap63fa7789
  volatile.eth0.hwaddr: 00:16:3e:29:73:30
  volatile.last_state.power: RUNNING
  volatile.uuid: 7a7c079f-48ef-4bd9-abef-c3432308d5f6
  volatile.uuid.generation: 7a7c079f-48ef-4bd9-abef-c3432308d5f6
  volatile.vsock_id: "1355136680"
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

I don’t believe any of my LXD daemon configs would affect this:

 # lxc config show
config:
  core.https_address: '[::]:8443'
  core.trust_password: true
  images.remote_cache_expiry: "1"
  storage.backups_volume: default/lxd-backups
  storage.images_volume: default/lxd-images

nikita-mezhenskyi · April 23, 2025, 2:50pm

Thank you for additional details, this is helpful. I’m investigating further.

nikita-mezhenskyi · April 23, 2025, 7:52pm

Yet when creating the instance on the LXD server that has the vm-bios profile, that profile is not attached to the instance.

I recreated the setup and I also experience the same issue consistently. It seems my suggested workaround was incomplete. I may have missed some other changes that I made while debugging. Sorry for the confusion.

I’ve dug into this further, and I’ve found a reliable workaround that involves an extra step before creating the instance.

You can copy the image from “remote” to your local LXD server and explicitly specify all profiles you want the image to use, like so:

lxc image copy <remote>:<image> local: -p default -p vm-bios

This will create a local copy of the image and set its profiles to default and vm-bios.

Then, when creating an instance you should specify a local image instead:

lxc launch local:<image> foo

or simply

lxc launch <image> foo

Could you please try this method (copying image to local first with the -p flags, then launching from the local copy) and let me know if it correctly applies the vm-bios profile to your new instance?

ibrummel · April 23, 2025, 8:48pm

Using -p default -p vm-bios does work. Though I was hoping to not have to explicitly define the custom profiles as the profile must only be applied to this image and none others.

This seems like a bug as the local LXD daemon should use the profiles on the remote image, without needing to explicitly define the image while copying it over? Should I open a bug report on Github?

nikita-mezhenskyi · April 23, 2025, 11:20pm

Yes, please feel free to open a GitHub bug report mentioning all the details you provided here.

I’m still trying to understand if that’s a bug or a design decision. Looking through the code, I see that we, for example, intentionally prevent copying information about profiles from remote servers. Hence, the need to use -p flags in the lxc image copy command. I’ll bring up this topic with my colleagues during our meeting.

nikita-mezhenskyi · April 23, 2025, 11:37pm

Depending on your use case, you can also try forming a cluster with your LXD “imageserver” (https://documentation.ubuntu.com/lxd/latest/howto/cluster_form/). Though, it is recommended to have at least 3 nodes in a cluster. With this setup, LXD servers will share the database and have access to the same profiles.

tomp · April 24, 2025, 8:09am

I’m not so sure, but we will evaluate and explore whether its appropriate.

as the local LXD daemon should use the profiles on the remote image

I don’t think this is necessarily always appropriate.
After all the profiles of the same name on remote server may be missing or contain completely different config than the profiles on the local server.

Images won’t import/apply profiles from the remote LXD server, so it doesn’t seem appropriate to me that LXD should automatically try and apply the profile names of a remote image using the local profiles of the same name.

@nikita-mezhenskyi is going to check whether this policy is applied consistently.

It would seem to me to be more appropriate to require the admin of the local LXD system to have to explicitly set which profiles to be used with the image (either when doing lxc image copy or via lxc image edit after it has been downloaded/cached). Its this @nikita-mezhenskyi is going to check is consistently done.

That way the admin of the local LXD system can be confident which configs are being applied.

nikita-mezhenskyi · April 24, 2025, 7:27pm

Images won’t import/apply profiles from the remote LXD server.

Yes, this is indeed the case. I did some more tests to confirm. In all cases, the cached image ignores the profiles set up on the remote image and only retains the “default” profile.

It would seem to me to be more appropriate to require the admin of the local LXD system to have to explicitly set which profiles to be used with the image (either when doing lxc image copy or via lxc image edit after it has been downloaded/cached).

Based on the tests I did, I can confirm this is currently the way it works. Profiles have to be manually specified for the local image. There is no scenario where a local image imports a remote image’s profiles.

system · April 28, 2025, 11:42am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.