Understanding cluster.evacuate default

[pmarini]

Reading this documentation page, section cluster.evacuate, I understand that by default LXD chooses auto .

If any device is not suitable for migration, the instance will not be migrated (only stopped).

In my case, if I leave cluster.evacuate to the default value, instances are not migrated. If, however I set cluster.evacuate=migrate instances are indeed migrated, that is what I’m looking for.

I’m running 5.21.5-f2a1a0e in a 4-nodes cluster. Instances are containers with a nic type=bridged attached:

eth1:
name: eth1
nictype: bridged
parent: br1
type: nic

br1 is present in all the hosts.

I’ve set cluster.healing_threshold = 60.

Questions:

1. if default cluster.evacuate just stops the instances, shouldn ‘t we expect that cluster.evacuate=migrate throws an error?
2. If I have to set cluster.evacuate=migrate can I put in a profile?

Thanks!

[tomp]

Please can you show lxc config show <instance> --expanded?

I suspect you have an instance device that does not support migration (without overriding).

If I have to set cluster.evacuate=migrate can I put in a profile?

Yes you can.

[pmarini]

Please can you show lxc config show <instance> --expanded?

architecture: x86_64
config:
  boot.autostart: "true"
  cluster.evacuate: migrate
  image.architecture: x86_64
  image.description: <Redacted>
  image.os: ubuntu
  image.release: noble
  limits.cpu: "1"
  limits.memory: 2GiB
  snapshots.expiry: 7d
  volatile.base_image: 5b3ea981cb08d6ea6c418f54d1f1fec0d59493f87132feb36f44a7a4c8e6a7d4
  volatile.cloud-init.instance-id: 85fff811-07aa-412e-9feb-92ee8c47e51f
  volatile.eth1.host_name: vethc8361d6b
  volatile.eth1.hwaddr: 00:16:3e:3d:b9:f5
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: 0164c063-0b18-405b-9f0a-ff568ebcb9fd
  volatile.uuid.generation: 0164c063-0b18-405b-9f0a-ff568ebcb9fd
devices:
  eth1:
    name: eth1
    nictype: bridged
    parent: br1
    type: nic
  root:
    path: /
    pool: remote
    size: 10GiB
    type: disk
ephemeral: false
profiles:
- prf-<redacted>
stateful: false
description: Test instance

[tomp]

Thanks.

So the reason that cluster.evacuate=auto didn’t migrate the instance is because of this statement in the docs:

If any device is not suitable for migration, the instance will not be migrated (only stopped).

Instance options

The “problem” device in this case is eth1, because it uses parent to connect to an external bridge br1.

Its not that LXD cannot migrate this instance, but rather that it cannot be certain that on the target cluster member the br1 network represents the same logical network.

So out of an abundance of caution LXD does not assume they are the same, to avoid accidentally connecting an instance to a network it shouldn’t be connected to.

You can override this concern by setting cluster.evacuate=migrate as you have found.

When using a bridged or macvlan NIC with the network= setting referring to a Managed Network instead of parent= then LXD knows you are referring to a logical network defined in its realm and so cluster.evacuate=auto will migrate it.

But currently LXD does not support modelling external bridges as managed networks, and only supports this for macvlan type networks.

There is this outstanding feature request for it here:

Managed network to reference unmanaged bridges

[pmarini]

Thanks @tomp for the clarification! I have subscribed to that GH ticket and follow up there.