LXD 5.20 has been released


The LXD team would like to announce the release of LXD 5.20!

Thank you to everyone who contributed to this release!

LXD change to AGPLv3

Canonical has decided to change the default contributions to the LXD project to AGPLv3 to align with our standard license for server-side code. All Canonical contributions have been relicensed and are now under AGPLv3. Community contributions remain under Apache 2.0. We follow the Software Freedom Law Center guidance in relation to this. Going forward, any contribution to LXD will be made under AGPLv3 by default. The author of a change remains the copyright holder of their code (no copyright assignment).

It is important to note this change does not prevent our users from using, modifying, or providing LXD-based software solutions, provided that they share the source code if they are modifying it and making it available to others. The conditions of the license are designed to encourage those looking to modify the software to contribute back to the project and the community.

New features and highlights

Create metadata and data OSD pools as part of creating a cephfs storage pool

Support for creating Ceph metadata and data OSD pools as part of creating a cephfs storage pool has been added. This new functionality is enabled using the cephfs.create_missing, cephfs.meta_pool and cephfs.data_pool pool config options.
For example:

lxc storage create mypool cephfs source=cephfs \
  cephfs.create_missing=true \
  cephfs.data_pool=xyz_data \

CSM boot device order supported in the LXD snap package

Added support to the LXD snap’s EDK2 firmware to respect disk device(s) boot.priority setting when using security.csm mode. Previously this was causing a problem when importing BIOS-based VMs that did not boot using UEFI because the VM firmware was trying to boot the UEFI devices first, and this meant that PXE network boot was tried before the BIOS-based root disk, causing long boot delays.

Debug mode for EDK2 UEFI firmware

To aid with debugging VM boot issues it is now possible to start VMs with an EDK2 UEFI firmware variant that has debug logging enabled. This can be enabled by setting boot.debug_edk2=true on the instance. The debug logs will then be available in $LXD_DIR/logs/<instance_name>/edk2.log.

Authorization restructure

LXD currently supports two forms of user authorization; TLS certificates and Canonical RBAC.
In order to make way for supporting OpenFGA as an additional authorization method there has been a significant restructing of the existing authorization code to make it modular.

Minimum Go version to build LXD raised to 1.20

We have increased the minimum supported Go version that is needed to compile LXD to Go 1.20.
This is to allow us to keep our external dependencies up to date.

Shiftfs support has been removed

Following the removal of shiftfs in the Ubuntu (from Mantic onwards) LXD has now also dropped support for shiftfs. The preferred way for container filesystems to have their UID/GID mappings be dynamically shifted is with idmapped mounts, which is now supported for ZFS and Cephfs (in addition to the long standing ext4, xfs and btrfs support).

Removal of 2MB UEFI firmware support

The 2MB VM UEFI firmware variant has been removed from the LXD snap package and existing VMs using the 2MB variant will be switched to the 4MB variant on next reboot. The 4MB variant supports more operating systems.

VM support for NVME storage (from Incus)

It is now possible to control which bus type a disk device uses inside a VM. Previously all disks used the virtio-scsi bus type. But now a new io.bus configuration key was added to disk type devices of VMs. The default is still virtio-scsi but it can now also be set to nvme in order to have the disk device appear as an NVME SSD inside the VM.


lxc storage volume create default vol1 --type=block
lxc config device add v1 vol1 disk source=vol1 pool=default io.bus=nvme
lxc exec v1 -- lsblk
sda       8:0    0   10G  0 disk 
├─sda1    8:1    0  100M  0 part /boot/efi
└─sda2    8:2    0  9.9G  0 part /
nvme0n1 259:0    0   10G  0 disk 

VM hot-plug/hot-unplug of directory passthrough disk devices (from Incus)

It is now possible to hot-plug and hot-unplug directory passthrough disk devices with VMs when LXD is using virtio-fs (which is the default if available). If the VM guest is runing the lxd-agent then the directory will also be mounted at the desired location inside the guest.

Note: At this time the directory inside the guest is not unmounted if the disk device is hot-unplugged.


lxc launch ubuntu:22.04 v1 --vm
mkdir /home/user/foo
touch  /home/user/foo/hello
lxc config device add v1 foo disk source=/home/user/foo path=/mnt/foo
lxc exec v1 -- ls -la /mnt/foo
total 8
drwxrwxr-x 2 ubuntu ubuntu 4096 Dec 11 09:21 .
drwxr-xr-x 3 root   root   4096 Dec 11 09:22 ..
-rw-rw-r-- 1 ubuntu ubuntu    0 Dec 11 09:21 hello
lxc config device remove v1 foo
lxc exec v1 -- ls -la /mnt/foo
ls: cannot access '/mnt/foo': Transport endpoint is not connected

VM LXD identifier serial device renamed to com.canonical.lxd

The ring-buffer serial device that is exposed to LXD VM guests which is used by the lxd-agent to instruct it to start, as well as being used to indicate the agent’s run status back to LXD, has been renamed from org.linuxcontainers.lxd to com.canonical.lxd. To maintain compatibility with existing images that contain systemd units that check for the presence of the old serial device the old device is still present in the guest, however it is no longer used for indicating the agent’s run status to LXD. See Rename LXD QEMU VM ring buffer to com.canonical.lxd for more information.

Complete changelog

Here is a complete list of all changes in this release:

Full commit list
  • doc/networking/firewall: add more restrictive UFW rules
  • loki: enable TLS verification if a CA cert is provided
  • test/container_devices_unix: Make unix device checks less flaky
  • gitignore: Ignore pycache
  • lxd/daemon: Initialise server name and global config before patches
  • doc: add a note about go-lxc build issue when LXC_DEVEL=1
  • lxd/patches: Only update volumes that need updating in patchStorageZfsUnsetInvalidBlockSettingsV2
  • lxd/patches: Only update volumes that need updating in patchStorageZfsUnsetInvalidBlockSettings
  • lxd/firewall: Fix nftables ACL template
  • lxd/api: replace numeric literal 301 by http.StatusMovedPermanently
  • lxd/auth/oidc: replace numeric literal 301 by http.StatusMovedPermanently
  • lxd/devlxd: replace numeric literal 401 by http.StatusUnauthorized
  • doc: update link to tool downloads
  • lxd: Update certificate cache again after cluster join.
  • lxd/patches: Add cluster check for patches fixing volumes
  • doc: update broken MAAS link
  • lxd/storage_pools: Fix etag when retrieving storage pool
  • Makefile: add staticcheck target
  • Add staticcheck config
  • golangci: sort linters list
  • doc/instances: clarify initial volume configuration
  • doc/instance options: mark limits.network.priority as deprecated
  • shared/cliconfig: Nicer error on missing socket
  • doc: remove link to topical navigation
  • doc: remove Security section from the start page
  • lxd/instance/drivers: Check running status with InitPID for cgroups
  • lxd/instance/drivers: Extend error message in deviceAddCgroupRules
  • api: Add cluster_internal_custom_volume_copy
  • shared/api: Add Location to StorageVolumeSource
  • shared/api: Add Source to StorageVolumePost
  • lxd/db: Add function to update storage volume node
  • lxd: Handle copying storage volumes with a single API call
  • lxd: Support single API custom volume rename
  • client: Set Source.Location if supported
  • doc: Update API
  • lxc: check host/port and URL path connectivity when adding a simplestreams remote
  • test: test bad connectivity when adding a simplestreams remote
  • lxd/instance/exec: Use shared.NewExecWrapper for MirrorRead in non-interactive exec
  • shared/ws/mirror: Updare Mirror*() to return error channels
  • client: shared.Mirror*() usage
  • lxd/instance/exec: Log error from ws.Mirror*() in execWs
  • readme: reference Ubuntu’s LXD security page
  • test/macaroon-identity: Adds RBAC endpoints to external auth daemon.
  • test: Rename macaroon-identiy → rbac.
  • test/includes: Update includes for external auth daemon.
  • test/includes: Rename includes file for RBAC server.
  • gitignore: Update gitignore for test rbac server binary.
  • test: Updates calls to start/stop RBAC daemon.
  • test/suites: Updates macaroon authentication test.
  • test/suites: Adds an RBAC test suite.
  • test: Runs RBAC test in standalone tests.
  • Revert “lxc: check host/port and URL path connectivity when adding a simplestreams remote”
  • Revert “test: test bad connectivity when adding a simplestreams remote”
  • doc: temporarily ignore MAAS links
  • lxc/copy: Require destination name to be provided
  • po: Update translations
  • shared/api: Add authentication method constants.
  • client: Replaces ‘oidc’ string with constant.
  • lxc/config: Replaces ‘oidc’ string with constant.
  • lxc: Replaces ‘oidc’ string with constant.
  • lxd: Replaces ‘oidc’ string with constant.
  • client: Replaces ‘candid’ string with constant.
  • lxc/config: Replaces ‘candid’ string with constant.
  • lxc: Replaces ‘candid’ string with constant.
  • lxd: Replaces ‘candid’ string with constant.
  • lxd-migrate: Replaces ‘candid’ string with constant.
  • client: Replaces ‘tls’ string with constant.
  • lxc/config: Replaces ‘tls’ string with constant.
  • lxc: Replaces ‘tls’ string with constant.
  • lxd: Replaces ‘tls’ string with constant.
  • lxd-agent: Replaces ‘tls’ string with constant.
  • lxd-migrate: Replaces ‘tls’ string with constant.
  • shared/network: remove unused args of GetTLSConfig()
  • lxd/migration_connection: drop unused args for shared.GetTLSConfig()
  • lxd/storage_volumes: drop unused args for shared.GetTLSConfig()
  • lxd/util/http: drop unused args for shared.GetTLSConfig()
  • shared/cert: drop unused args for GetTLSConfig()
  • test/suites: Unsets RBAC configuration after test.
  • lxd/instance/driver/qemu: replace sha1 by sha256 in blockNodeName()
  • shared/api: Adds constant for default project name.
  • lxd/cluster: Updates project.Default to api.ProjectDefaultName.
  • lxd/db: Updates project.Default to api.ProjectDefaultName.
  • lxd/device: Updates project.Default to api.ProjectDefaultName.
  • lxd/instance/drivers: Updates project.Default to api.ProjectDefaultName.
  • lxd/instance: Updates project.Default to api.ProjectDefaultName.
  • lxd/maas: Updates project.Default to api.ProjectDefaultName.
  • lxd/network/acl: Updates project.Default to api.ProjectDefaultName.
  • lxd/network: Updates project.Default to api.ProjectDefaultName.
  • lxd/project: Updates project.Default to api.ProjectDefaultName.
  • lxd/storage: Updates project.Default to api.ProjectDefaultName.
  • lxd: Updates project.Default to api.ProjectDefaultName.
  • lxd/project: Removes project.Default.
  • lxd/request: Exports query parameter methods and moves to lxd/request.
  • lxd: Updates calls to projectParam and queryParam.
  • shared/util/linux: Update NewExecWrapper.Read to be time based when waiting for output from a process after it has exited
  • lxd/auth: Adds entitlement, object, and permission types and constants.
  • lxd/auth: Adds functions for creating auth objects.
  • lxd/auth: Adds tests for authorization objects.
  • lxd/auth: Extends the authorizer interface.
  • lxd/auth: Update common authorizer for Authorizer interface extension.
  • lxd/auth: Implement Authorizer for TLS driver.
  • lxd/auth: Implement Authorizer for RBAC driver.
  • lxd: Do not set user access data in request context.
  • lxd: Update calls to auth package.
  • lxd: Only allow missing access handler when AllowUntrusted is true.
  • lxd: Update allowPermission function.
  • lxd: Updates allowAuthenticated function.
  • lxd/db/operationtype: Updates Permission method.
  • lxd/operations: Updates operation permissions.
  • lxd/db/cluster: Renames constants.go file.
  • lxd/db/cluster: Add storage bucket entity type.
  • lxd/db/cluster: Adds URLToEntityType function.
  • lxd/db/cluster: Adds a unit test for the URLToEntityType function.
  • lxd/project: Updates permission handling for projects.
  • lxd/project: Updates permissions tests.
  • lxd/events: Pass an auth.PermissionChecker into the event listener.
  • lxd-agent: Update call to AddListener for the LXD Agent.
  • lxd: Update authorization for the /1.0 endpoint.
  • lxd: Update authorization for cluster endpoints.
  • lxd: Update authorization for internal endpoints.
  • lxd/metrics: Adds method to filter metrics with a permission checker.
  • lxd: Update authorization for metrics.
  • lxd: Update authorization for projects API.
  • lxd: Updates authorization for certificates API.
  • lxd: Updates authorization for events API.
  • lxd: Updates authorization for image API.
  • lxd: Add/remove images and image aliases from authorizer.
  • lxd: Update authorization for instances.
  • lxd/instance/drivers: Add/remove/rename instances in authorizer.
  • lxd: Update authorization for network ACL API.
  • lxd: Update network ACLs in the authorizer.
  • lxd: Update authorization for network allocations.
  • lxd: Update authorization for network forwards.
  • lxd: Update authorization for network load balancers.
  • lxd: Update authorization for network peers.
  • lxd: Update authorization for network zones.
  • lxd: Update network zones in the authorizer.
  • lxd: Update authorization for the networks API.
  • lxd: Update networks in the authorizer.
  • lxd: Update authorization for operations.
  • lxd: Update authorization for profiles.
  • lxd: Update profiles in authorizer.
  • lxd: Update authorization for resources.
  • lxd: Update authorization for storage buckets.
  • lxd: Update storage buckets in authorizer.
  • lxd: Update authorization for storage pools.
  • lxd: Update storage pools in authorizer.
  • lxd: Update authorization for storage volumes.
  • lxd/storage: Add/Remove/Rename storage volumes in authorizer.
  • lxd: Update authorization for warnings.
  • lxd/cluster/config: Add missing bool default values
  • lxd/storage_volumes: Fix calls to QueryParam
  • lxd/patches: Ensure renaming is only done on cluster leader
  • instance/lxc: Fix swap limit handling
  • test/suites: Fixes wait_no_operations helper.
  • gomod: Remove github.com/pborman/uuid dependency
  • lxd/storage/drivers: Generate and parse UUID using github.com/google/uuid
  • lxd/instance/drivers: Generate and parse UUID using github.com/google/uuid
  • lxd/instance: Generate UUID using github.com/google/uuid
  • lxc-to-lxd: Generate UUID using github.com/google/uuid
  • lxd-migrate: Generate UUID using github.com/google/uuid
  • lxd/apparmor: Generate UUID using github.com/google/uuid
  • lxd/bgp: Generate UUID using github.com/google/uuid
  • lxd/db: Generate UUID using github.com/google/uuid
  • lxd/device: Generate UUID using github.com/google/uuid
  • lxd/events: Generate UUID using github.com/google/uuid
  • lxd/firewall/drivers: Generate UUID using github.com/google/uuid
  • lxd/operations: Generate UUID using github.com/google/uuid
  • lxd/rsync: Generate UUID using github.com/google/uuid
  • lxd/storage/s3/miniod: Generate UUID using github.com/google/uuid
  • shared/validate: Parse UUID using github.com/google/uuid
  • test/rbac: Generate UUID using github.com/google/uuid
  • lxd/auth/oidc: Generate UUID using github.com/google/uuid
  • lxd: Handler error from oidc.NewVerifier
  • metrics: Fix label merging in metric sets
  • zfs: Support zfs pools containing ‘/’ in the patch
  • lxd/auth: Adds OpenFGA model.
  • Makefile: Adds make target for generating openfga model json.
  • lxd/auth: Runs make-openfga.
  • lxd/auth: Adds constants for relations.
  • lxd/auth: Adds Resources type and load option.
  • gomod: Adds openfga dependency.
  • lxd/auth: Adds OpenFGA authorization driver.
  • lxd/cluster/config: Adds OpenFGA config keys.
  • lxd-metadata: Runs make update-metadata.
  • lxd/db: Exports StoragePoolVolumeTypeToName function.
  • lxd: Adds method to daemon to load the OpenFGA authorizer.
  • lxd: Loads OpenFGA authorizer at startup if configured.
  • lxd: Load OpenFGA authorizer on config change.
  • test/lint: Adds linter for openfga model.
  • test/includes: Adds util for getting certificate fingerprint.
  • test/includes: Adds utils for running and interacting with an openfga server.
  • test/includes: Use hostname for RBAC server endpoint.
  • test/suites: Adds OpenFGA test suite.
  • test/suites: Adds OpenFGA clustering test.
  • test: Runs OpenFGA tests in main.
  • workflows: Installs openfga server and CLI in github action.
  • doc: Adds OpenFGA to wordlist.
  • doc: Adds openfga server configuration options.
  • doc: Adds authorization explanation page.
  • doc: Updates authentication page to separate authorization.
  • doc: Update references to RBAC section.
  • doc: Adds authorization page to security related links.
  • doc: Adds authorization page to explanation section.
  • test/deps: switch to ecdsa certificate
  • github: shorten job names to improve the UI view
  • test/clustering: remove unneeded shellcheck ignore and update others
  • lxc/network forward: Fix typo port to ports.
  • i18n: Update translation templates
  • doc: add page title for related links
  • config: Ensure config key values are reset to their default
  • lxd/seccomp: Switch to path/filepath
  • lxd/seccomp: Pass correct path and fstype to IdmappedStorage
  • lxd/forksyscall: Fix idmapped mount code path
  • test: Test unsetting config keys
  • test: Check instance type in filesystem metrics
  • config: Restrict user.* keys
  • doc/configuration: review openfga.* documentation
  • doc/openfga: small fixes to documentation
  • test/includes/certificates: add gen_cert_and_key()
  • test/metrics: use gen_cert_and_key function instead of directly calling openssl
  • test/remote: use gen_cert_and_key function instead of directly calling openssl
  • test/tls_restrictions: add some double quotes
  • test/tls_restrictions: fix some comments
  • test/tls_restrictions: make sure expected failures get the expected 403
  • test/tls_restrictions: use gen_cert_and_key function instead of directly calling openssl
  • test/tls_restrictions: ensure type=metrics certificates cannot access anything besides /1.0/metrics.
  • test: Validate user.* keys
  • client: Use io.Writer for Stdout/Stderr in InstanceExecArgs
  • lxd/device/proxy: Consider routed NIC IPs for wildcard target check
  • lxd/network/driver/bridge: Improve comments for accept_ra
  • gomod: Bump to Go 1.20 and update deps
  • github: Use Go 1.20 and check for compat with that in go mod tidy
  • github: Removes whitespace
  • lxd/main: Stop seeding the RNG
  • lxd/network/utils: use crypto/rand instead of math/rand
  • lxd/lxd-metadata: Remove noisy log line
  • doc: Ensure readthedocs build uses Go 1.20
  • doc: Removes readthedocs whitespace
  • shared/cert: Replace x509.ParseCRL() by x509.ParseRevocationList()
  • lxd/util/http: Use x509.RevocationList
  • lxc/network_load_balancer: fix typo port to ports
  • i18n: Update translations
  • test/lint: Removes openfga model linter.
  • test/basic: always use – with lxc exec
  • test/basic: test with and without “–” separator
  • test/clustering: always use – with lxc exec
  • test/config: always use – with lxc exec
  • test/devlxd: always use – with lxc exec
  • test/image_acl: always use – with lxc exec
  • test/storage_snapshots: always use – with lxc exec
  • doc/howto/network_ovn_setup: always use – with lxc exec
  • doc/howto/instances_troubleshoot: always use – with lxc exec
  • lxd/devlxd: always use – with lxc exec
  • api: disk_io_bus
  • doc: Add io.bus to disk devices
  • doc: Reformat disk option table
  • lxd/device/disk: Add io.bus
  • lxd/instance/qemu: Add NVME disk support
  • lxd/devices/disk: Always apply the disk options
  • incusd/auth: Fix handling of trusted certs in CA mode
  • Makefile: Rename update to update-gomod
  • doc/requirements: allow linking to Go requirements
  • doc/installing: link to Go requirements and update Ubuntu instructions
  • doc/howto/benchmark_performance: link to Go requirements
  • doc/howto/migrate_from_lxc: link to Go requirements
  • doc/requirements: Go 1.20 is now the minimum version
  • shared/cert: Update code comments about CRL
  • lxd/util/http: Check if the CRL was signed by the CA before using it
  • btrfs: Add function to check subvolumes in a given path
  • btrfs: Use hasSubvolumes when creating a new pool
  • test: Btrfs pool with a subvolume as its source
  • lxc/delete: Include instance name in error message
  • Update translations
  • client: Add function to open webbrowser
  • client: Allow overriding web browser
  • client: Cleanup OIDC login
  • tests: Add mini-oidc
  • tests/lint: Ignore test/mini-oidc
  • tests: Add oidc helpers
  • tests: Add OpenID Connect tests
  • lxc: Use volume copy when moving to target project
  • shared/network: Only skip TLS verification if no remote certificate is available
  • lxd/daemon_images: fix typo
  • test: Restructure local volume handling
  • test: Add storage volume move between projects
  • tests: Properly test core.trust_ca_certificates
  • lxd: Enforce users to be authenticated before running the access handler.
  • lxd/instance/exec: Only use keepalives on TCP sockets
  • client: Use io.Reader for Stdin in InstanceExecArgs
  • lxd/task/group: Code style
  • lxd/task/group: Handle nil group
  • lxd/storage/filesystem/fs: Rename parseMountinfo()
  • lxd/storage/filesystem/fs: Add GetMountinfo()
  • lxd/storage/drivers/btrfs: Skip nodatacow on compressed pools
  • lxd/storage/drivers/btrfs: Check for datacow mount option
  • Makefile: Force Go 1.20
  • Makefile: remove toolchain directive from go.mod for backward compat
  • go.mod: update deps with make update-gomod
  • github: try up to 3 times to download Go tip tarball
  • github: remove Go tip tarball after extraction
  • config: Fix acme.ca_url short description
  • Update metadata
  • lxd/storage/drivers: Add new cephfs create keys
  • lxd/instance/drivers/driver_qemu: factor out config volume mounting from setupNvram
  • lxd/instance/drivers/driver_qemu: force 4MB UEFI firmware in snap
  • shared: Read system certs from /etc instead of snapd/hostfs
  • lxd/instance/drivers/driver_qemu: use OVMF_CODE.fd in a non-snap environment
  • lxd/instance/drivers/driver_qemu: add the boot.debug_edk2 option
  • shared/instance: correct volatile.apply_nvram type
  • lxd/state: Add local server UUID
  • client/lxd/instances: Close websocket as soon as channel mirror finishes in ExecInstance
  • lxc/exec: No need to use io.ReadCloser anymore
  • shared/ws/mirror: No need for defer in MirrorWrite and MirrorRead
  • Revert “lxd/instance/exec: Only use keepalives on TCP sockets”
  • client/lxd/instances: Consume ping messages from server for exec control and stdin channels
  • lxd/instance/drivers/qemu: Tweak systemd/udev units of lxd agent
  • lxd/instance/qemu: Support LXD virtual machines
  • lxd-agent/main_agent: Update virtio-port path
  • lxd/daemon: Setup and load UUID file on init
  • doc/installing: LXC_DEVEL needs to be fixed on 22.04+
  • lxd/instance/drivers: Add comments for lxd-agent udev rules, systemd unit, and serial devices
  • lxd/instance/drivers/qemu: consistently rely on $PATH to find binaries
  • lxd/instance/drivers/qemu: mount the config drive as readonly
  • lxd/instance/drivers/qemu: reduce the size of /run/lxd_agent tmpfs and set nodev,nosuid,noatime
  • lxd/instance/drivers/qemu: do not preserve the ownership during the cp to avoid chown
  • github: instruct dependabot to also look after the stable-5.0 branch
  • lxd/instance/drivers: Cleanup old lxd-agent symlink in install script
  • lxc/move: Prevent pool migration to block project migration
  • lxd/instance_post: Determine root device from profiles in target project
  • lxc/move: Throw an error when unsupported move flags are used
  • i18n: Update translations
  • github: instruct dependabot to also look after the stable-4.0 branch
  • github: dependabot knows which branch to target
  • lxd/storage/drivers: Update cephfs entity helpers
  • lxd/instance/drivers/driver_qemu: support bootorder in edk2 CSM mode
  • lxd/storage/drivers: Add DefaultVMBlockFilesystemSize to driver Info struct
  • lxd/storage/drivers/btrfs: Set drivers DefaultVMBlockFilesystemSize
  • lxd/storage/drivers/ceph: Set drivers DefaultVMBlockFilesystemSize
  • lxd/storage/drivers/cephfs: Set drivers DefaultVMBlockFilesystemSize
  • lxd/storage/drivers/dir: Set drivers DefaultVMBlockFilesystemSize
  • lxd/storage/drivers/lvm: Set drivers DefaultVMBlockFilesystemSize
  • lxd/storage/drivers/mock: Set drivers DefaultVMBlockFilesystemSize
  • lxd/storage/drivers/zfs: Set drivers DefaultVMBlockFilesystemSize
  • lxd/storage/backend: Use drivers default VM block volume size for config filesystem
  • lxd/storage/drivers/volume: Use drivers default VM block size for filesystem volume
  • lxd/project: Fix typo in comment
  • lxd/instance/drivers: Use the pools default VM block filesystem size
  • lxd/storage: Use the pools default VM block filesystem size
  • lxd/project: Add TODO for instance limits accounting
  • lxd/instance: Use stable random generator for temporary instance name
  • lxd/instance: Improve error message
  • lxd/instance/drivers/qemu: Run specific remote config only for Ceph backends
  • lxd/storage/drivers: Create cephfs entities if keys specified
  • lxd/storage/drivers: Revert osd/fs creation
  • doc/reference: Add doc reference for new config keys
  • shared/version: Add storage_cephfs_create_missing extension
  • test/suites: Add cephfs create_missing test
  • lxd/instances: Properly detect unfiltered
  • lxd/images: Properly detect unfiltered
  • shared/filter: Support string slices
  • lxd/storage_volumes: Allow filtering based on UsedBy
  • docs: update iso import in instances_create
  • lxd/storage/drivers: Collect subvolumes via filepath traversal if in nested container.
  • doc/howto: Make pool name consistent in iso tutorial.
  • lxd/instance_post: Retain root disk device if not explicitly changed
  • doc/instances: change pool name to be consistent
  • test: Add tests for server-side instance move
  • lxd/instance/drivers/qemu_cmd: Return clean EOF error
  • api: Add API extension for improved server-side move
  • shared/api/instance: Expand InstancePost structure
  • lxc/move: Respect all flags on server-side move
  • lxd/instance_post: Respect provided config, device and profile overwrites on move
  • tests: Add server-side move tests
  • doc: Update API
  • i18n: Update translations
  • doc/support: add information about Ubuntu Pro
  • github: enable dependabot gomod checking for 4.0/5.0 stable branches
  • github: have curl fail instead of feeding bogus data on download error
  • lxd/instance/drivers/qemu: Send device notifications
  • lxd-agent/events: Properly forward device events
  • lxd/instance/drivers/qmp/commands: Add CharDevice commands
  • lxd/device/disk: Allow virtiofs hotplug/hotremove
  • lxd/device/disk: Don’t spawn 9p proxy for hotplug
  • lxd/instance/drivers/qemu: Add support for hotplug/hotremove of virtiofs
  • lxd-agent/events: Add support for mounting hot-plugged paths
  • shared: remove shiftfs
  • shared/idmap: Remove shiftfs
  • lxd/sys: Remove shiftfs
  • lxd/storage/drivers: Remove shiftfs
  • lxd/seccomp: Remove shiftfs
  • lxd/instance/drivers: Remove shiftfs
  • lxd/device/config: Remove shiftfs
  • lxd: Remove shiftfs
  • tests: Rename shiftfs test with idmapped_mount
  • .github/workflows: remove shiftfs
  • doc: remove shiftfs
  • lxd/metadata: remove shiftfs
  • Revert “doc: Adds authorization page to explanation section.”
  • Revert “doc: Adds authorization page to security related links.”
  • Revert “doc: Update references to RBAC section.”
  • Revert “doc: Updates authentication page to separate authorization.”
  • Revert “doc: Adds authorization explanation page.”
  • Revert “doc: Adds openfga server configuration options.”
  • Revert “doc: Adds OpenFGA to wordlist.”
  • Revert “workflows: Installs openfga server and CLI in github action.”
  • Revert “test: Runs OpenFGA tests in main.”
  • Revert “test/suites: Adds OpenFGA clustering test.”
  • Revert “test/suites: Adds OpenFGA test suite.”
  • Revert “test/includes: Use hostname for RBAC server endpoint.”
  • Revert “test/includes: Adds utils for running and interacting with an openfga server.”
  • Revert “test/includes: Adds util for getting certificate fingerprint.”
  • Revert “lxd: Load OpenFGA authorizer on config change.”
  • Revert “lxd: Loads OpenFGA authorizer at startup if configured.”
  • Revert “lxd: Adds method to daemon to load the OpenFGA authorizer.”
  • Revert “lxd/db: Exports StoragePoolVolumeTypeToName function.”
  • Revert “lxd-metadata: Runs make update-metadata.”
  • Revert “lxd/cluster/config: Adds OpenFGA config keys.”
  • Revert “lxd/auth: Adds OpenFGA authorization driver.”
  • Revert “gomod: Adds openfga dependency.”
  • Revert “lxd/auth: Adds Resources type and load option.”
  • Revert “lxd/auth: Adds constants for relations.”
  • Revert “lxd/auth: Runs make-openfga.”
  • Revert “Makefile: Adds make target for generating openfga model json.”
  • Revert “lxd/auth: Adds OpenFGA model.”
  • doc: Remove openfga server options.
  • test: if backend is ZFS (also through random backend) and version less than 2.2, skip idmapped mount
  • lxd/instance/drivers: Set correct RBD content type for qemu drives
  • lxc/move: Overwrite profiles only if explicitly provided by the user
  • lxd/instance_post: Retain previous profiles on instance move
  • tests: Improve tests for instance move
  • lxd/db/instances: Fix instance names from project not retrieved
  • gomod: Update dependencies
  • lxd/cluster: Retry cluster join if cluster is busy
  • github: stop dependabot from looking after 4.0 gomods
  • build(deps): bump actions/labeler from 4 to 5
  • lxd/apparmor: Allow qemu access to microceph conf
  • lxd/state: Add new ServerClustered field
  • lxd: Use ServerClustered
  • github: Update for new labeler
  • lxd/cluster/config: Add missing description default values
  • lxd/node: Add missing description default values
  • Update metadata
  • lxd/storage/s3/miniod: Discover port using IPv4 address family
  • lxd-agent: Prevent panic when devlxd server is stopped
  • lxd/storage/drivers: Always copy Ceph VMs filesystem volume
  • doc/cloud-init: overwrite link text to make spell checker happy
  • internal/server/response: Don’t re-send headers when streaming
  • incusd/operations: Use ManualResponse to send headers early
  • client: Always use event listener for operations.
  • lxd/instance/drivers/qemu: Load storage pool before accessing it
  • golangci: Updates the metalinter configuration.
  • lxd/firewall/drivers: Removes unnecessary break statements from switch.
  • test/lint: Add script to invoke golangci-lint with ‘–new’.
  • Makefile: Remove invocation of golangci-lint from Makefile.
  • incusd/storage: Fix size check for ISO volumes
  • incus: Fix typo in comment
  • incusd/storage: Use Shutdown context for import from backup
  • incusd/instance/qemu: Properly set cdrom type
  • doc: fix broken links
  • build(deps): bump actions/setup-go from 4 to 5
  • client/lxd/instances: Treat nil args as empty InstanceExecArgs in ExecInstance
  • client/lxd/instances: Always consume pings from control socket if established in ExecInstance
  • client/lxd/instances: Discard non-interactive stdout/stderr output if writer(s) not supplied in ExecInstance
  • client/lxd/instances: Remove unnecessary args nil check
  • doc/storage/cephfs: specify that you can automatically create pools
  • Change license to AGPLv3
  • doc/rest-api: Refresh swagger YAML
  • doc: Adds AGPLv to word list


The release tarballs can be found on our download page.

Binary builds are also available for:

  • Linux: snap install lxd
  • MacOS: brew install lxc
  • Windows: choco install lxc

LXD 5.20 is now available in the latest/candidate snap channel and will be rolled out to stable users in the new year.


It appears the snap is not showing the correct license: Incorrect license information for the LXD snap - snap - snapcraft.io

Spdx is designed to support complicated cases like LXD, so the snap’s license should be able to reflect this complexity.

At the very least, having the snap show the most restrictive license is a good start, imo.

Nice to see that, FINALLY, the open source community remembers the AGPL-v3.
MySQL could have been saved if this idea would have spread earlier…

LXD 5.20 is now being progressively rolled out to the last/stable channel.

Since the initial release notes above, it should be noted that due to the upstream EDK2 project dropping support for CSM mode then LXD security.csm mode now instead switches QEMU to boot via Seabios directly rather than through EDK2.


Just out of interest, this update has broken my server setup. I have

  • Ubuntu 22.04 LTS
  • ZFS for my storage, which is where I put the volumes for data for the containers.
  • LXD to run the containers.

LXD has upgraded to 5.20, which has removed shiftfs support. Ubuntu 22.04 LTS has ZFS 2.1.5 which doesn’t have idmapped mounts support. Some containers (which shared volumes) now fail to start with:

$ lxc start backups
Error: Failed to setup device mount "paperlessng": idmapping abilities are required but aren't supported on system
Try `lxc info --show-log backups` for more info

I suspect I can’t downgrade to 5.19 which has shiftfs support. Maybe I should have pinned it to 5.19, but then I had no idea this would happen and LXD also didn’t check whether my system supported it before it upgraded, which is user hostile.

What can I do to solve this problem, please?

As an interim thought, it might be useful to add shiftfs support back in for all the LTS users for the next 8 years?

Hi Alex,

As an interim thought, it might be useful to add shiftfs support back in for all the LTS users for the next 8 years?

The LXD 5.0.x LTS series is associated with the Ubuntu 22.04 release, and indeed this does not have shiftfs support removed for the reason you suggest. This is available in the 5.0/stable channel, and is supported until June 2027. Unfortunately you cannot switch back from latest/stable to 5.0/stable due to DB schema changes.

If you’re getting refreshed onto LXD 5.20 it suggests you are following the latest/stable which has a new release approximately once a month, with the previous version only receiving support until the next version is released. So we wouldn’t recommend pinning an unsupported monthly version on a server.

See here for more info on snap channels:

Historically if LXD was installed manually, it defaulted to the latest/stable track. Whereas the version pre-seeded into Ubuntu LTS releases was set to the associated LTS track.

I’d be interested to know if you made a conscious decision to upgrade to the latest/stable track or whether it occurred implicity via manual installation at some point.

Going forward for our next LTS, 5.21.x, we are going to make this the default track for manual installs to avoid this scenario in the future (although we will still make the non-LTS Ubuntu releases install the latest/stable track by default to show off the new features).

Let me try to reproduce the issue locally and ill come back to you with a workaround.

Hi Tom

Thanks for your reply:

Historically if LXD was installed manually, it defaulted to the latest/stable track. Whereas the version pre-seeded into Ubuntu LTS releases was set to the associated LTS track.

This server has been around since 18.04, when LXD was a package, rather than a snap. I guess it got auto-upgraded to the snap and ended up on latest/stable. I don’t think I ever changed it though. I would definitely have pinned it, though, had I understood the implications.

Let me try to reproduce the issue locally and ill come back to you with a workaround.

Yes, please. I’m pondering my options for this as currently bit of my server aren’t currently running!


If you need to temporarily pin to LXD 5.19 you can do this:

snap refresh lxd --channel=5.19/stable

I’ve tested and it seems to work for downgrading from LXD 5.20 (but wont work from LXD 5.21 as there will be schema changes that prevent it).

I just tried installing LXD 5.19, enabling shiftfs, creating a container on a ZFS pool, and then refreshing up to latest/stable and on next container restart the root filesystem was manually shifted (because shiftfs was unavailable) but it started fine.

So I suspect from your error you have a custom disk device attached to the instance with the shift setting enabled.

Please can you show output of lxc config show <instance> --expanded?

I think if you want to switch to the 5.0/stable channel you’ll need to export your instances and custom volumes to tarballs using lxc export and lxc storage volume export then setup a fresh installation of LXD, and then import them back in.

Please can you show output of lxc config show <instance> --expanded ?

This is the output; there are some volatile.idmap.* entries that may be relevant? The volume paperless-ng is shared with a paperless-ngx instance that did start, did some id remapping at start up, and it’s relevant items are shown below this:

$ lxc config show backups --expanded
architecture: x86_64
  image.architecture: amd64
  image.description: ubuntu 20.04 LTS amd64 (release) (20210510)
  image.label: release
  image.os: ubuntu
  image.release: focal
  image.serial: "20210510"
  image.type: squashfs
  image.version: "20.04"
  user.network-config: |
    version: 1
      - type: physical
        name: eth0
          - type: static
            ipv4: true
            control: auto
      - type: nameserver
  user.user-data: |
    package_update: true
    package_upgrade: true
    package_reboot_if_required: true
  volatile.base_image: 52c9bf12cbd3b06d591c5f56f8d9a185aca4a9a7da4d6e9f26f0ba44f68867b7
  volatile.eth0.hwaddr: 00:16:3e:87:81:3c
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: STOPPED
  volatile.uuid: 20fba133-fdaa-46f5-aa76-d5433ec407dd
  volatile.uuid.generation: 20fba133-fdaa-46f5-aa76-d5433ec407dd
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
    path: /srv/paperless-ng
    pool: default
    source: paperless-ng-data
    type: disk
    path: /
    pool: default
    type: disk
ephemeral: false
- backups
stateful: false
description: ""

from paperless-ngx:

$ lxc config show paperless-ngx --expanded
architecture: x86_64                
  volatile.idmap.base: "0"
  volatile.idmap.current: '[]'
  volatile.idmap.next: '[]'

Not sure if this helps?

re: fixing it by exporting, re-installing, and importing; that’s going to be fairly tricky for my syncthing container as it contains around 100GiB of data in the volume.

Thanks again.

Please can you show output of lxc storage show default and lxc storage volume show default paperless-ng-data thanks

$ lxc storage show default
  source: pool4t/lxd
  volatile.initial_source: pool4t/lxd
  zfs.pool_name: pool4t/lxd
description: ""
name: default
driver: zfs
- /1.0/images/97b9236df59497b28eebeb91eee7a2bd815e428613e49e478837ffa401d39da0
- /1.0/instances/backups
- /1.0/instances/focal-builder
- /1.0/instances/paperless-ngx
- /1.0/instances/plexd
- /1.0/instances/postgresql
- /1.0/instances/smb-timemachine
- /1.0/instances/syncthing
- /1.0/instances/taskd
- /1.0/profiles/backups
- /1.0/profiles/default
- /1.0/profiles/paperless-ngx
- /1.0/profiles/plexd
- /1.0/profiles/postgresql
- /1.0/profiles/smb-timemachine
- /1.0/profiles/syncthing
- /1.0/profiles/taskd
- /1.0/storage-pools/default/volumes/custom/paperless-ng-data
- /1.0/storage-pools/default/volumes/custom/paperless-ng-syncthing
- /1.0/storage-pools/default/volumes/custom/postgresql-data
- /1.0/storage-pools/default/volumes/custom/syncthing-folders
- /1.0/storage-pools/default/volumes/custom/taskd
status: Created
- none


$ lxc storage volume show default paperless-ng-data
  security.shifted: "true"
  volatile.idmap.last: '[]'
  volatile.idmap.next: '[]'
description: ""
name: paperless-ng-data
type: custom
- /1.0/profiles/backups
- /1.0/profiles/paperless-ngx
location: none
content_type: filesystem
project: default
created_at: 0001-01-01T00:00:00Z

Note that the paperless-ng-data volume has been ‘touched’ by LXD 5.20 and thus the idmap entries may have disappeared due to the idmapping exercise. CF with syncthing-folders which does have entries:

$ lxc storage volume show default syncthing-folders
  volatile.idmap.last: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
description: ""
name: syncthing-folders
type: custom
- /1.0/profiles/syncthing
location: none
content_type: filesystem
project: default
created_at: 0001-01-01T00:00:00Z

Again, this may be me misreading/misunderstanding the situation!

The the issue is with this setting, as that is no longer supported in LXD 5.20 unless you’re using a filesystem with a kernel that supports idmapped mounts.

Try setting it to false with lxc storage volume set default paperless-ng-data security.shifted false and see if that allows you to start the instance. You will likely find that the share is effectively read only in the container as the dynamic mappings won’t apply.

I need the r/w shared volumes. I’ve pinned LXD to 5.20/stable to ensure it doesn’t go any further whilst I try to sort this out.

How would you react to the idea of upgrading the kernel to HWE (to get Linux 6.2) and then upgrading ZFS to 2.2? This would enable VFS idmaps and then LXD 5.20 would work then be okay? My nervousness is around doing the ZFS upgrade and whether LXD would 5.20 would be happy with ZFS 2.2?

LXD 5.20 supports ZFS 2.2 in the kernel fine. It bundles several versions of ZFS userland tooling and switches to the correct version as needed.

So a bit of a surprising outcome. I upgrade the kernel to the hwe one, 6.5, and on a whim decided to start the containers just to see what happened. The zfs version on the host is still (AFAICT) 2.1.5. And the containers have started, and everything seems to be working okay. Everything seems to be rw. The only difference is that Linux 6.5 supports VFS idmaps.

1 Like

Yeah the lxd snap bundles several versions of the zfs user land including 2.2 so if the host kernel provides the right zfs module version it will use it.