Instance volume configuration through disk device

LXD Specifications
1
Project LXD
Status Implemented
Author(s) @dinmusic
Approver(s) @tomp
Release 5.19
Internal ID LX048

Abstract:

We would like to introduce new initial configuration for instance disk devices. Initial configuration will be applied only to the root disk devices when the instance is created. The configuration settings will have initial. prefix, that is trimmed during instance creation.

Rationale:

The instance volume settings are influenced by the pool’s configuration. In order to create a new instance with a specific volume configuration, the pool’s configuration has to be changed before the instance is created. Initial configuration would allow new instances to be created with the specific configuration without the need to modify the pool’s configuration.

Specification

Design

Initial configuration will be saved in profile device config and in instance device config. While changes to the profile device configuration will have no effect on the existing devices, changes to the instance device configuration will not be allowed.

Initial configuration will have precedence over the default pool configuration. Similarly, the initial configuration provided when creating an instance has precedence over existing configuration in the profile.

  1. Extend disk validation rules
    New initial configuration settings should be added and validated along the existing disk configuration validation rules.

  2. Trim initial. prefix
    Before passing configuration values to the specific storage driver, initial configuration key prefix must be trimmed (e.g. initial.zfs.block_mode -> zfs.block_mode).

  3. Apply the configuration
    The initial configuration must be applied to the instance root volume only when the instance is being created.

Restrictions

  1. Initial configuration can be applied only to the root disk devices.

  2. Initial configuration is applied to the instance’s volume only when the instance is being created (empty instance or instance created from image).

  3. Initial configuration has precedence over the default pool configuration.

  4. Initial configuration provided at instance creation time has precedence over the initial configuration in profile.

  5. Certain settings, such as size, will not be configurable using initial configuration.

Validation

  1. Profile: When applying initial configuration to the profile, configuration keys that contain initial. prefix will be trimmed and validated against the device’s storage driver.

  2. Instance: When initial configuration is applied at the instance creation, configuration keys that contain initial. prefix will be trimmed and validated against the device’s storage driver.

Images

When a new instance is created from an image, LXD first attempts to create an optimized image volume based on the pool’s default settings. The optimized image is reused for all new instances, as long as the default pool settings do not change. If the default pool settings do change, the optimized image volume has to be recreated to match the new pool settings, and the old image volume is deleted (or retained in /deleted if any instance relies on it). If optimized image volume cannot be created (e.g. the image volume cannot be shrunk) , LXD unpacks the image archive directly into the instance volume.

The initial configuration will impact optimized image volumes, because providing a custom configuration will require an image volume to be recreated. To avoid potential performance degradation that this change can cause, we have to decide how to handle optimized image volumes when initial configuration is applied. Currently, the following approaches are being considered:

  1. Image switching

The default approach described above can cause switching between image volumes. Whenever a new instance without the initial configuration is created, an optimized image volume that matches the pool’s default settings is created. If a new instance is created with the initial configuration that differs from the pool’s default settings, a new optimized image volume has to be created. If a new instance without the initial configuration is created again, the optimized image volume will have to be replaced again (adding the time of unpacking the image).

For example:

# Storage pool with zfs.block_mode disabled.
lxc storage create test zfs
lxc storage set test volume.zfs.block_mode=false

# Creates an optimized image volume that matches the default pool settings.
lxc launch ubuntu:22.04 c1 --storage test

# Recreates an optimized image volume that match new volume settings.
lxc launch ubuntu:22.04 c2 --storage test —device root,initial.zfs.block_mode=true

# Recreates an optimized image volume (again) that match the default pool settings.
lxc launch ubuntu:22.04 c3 --storage test
  1. Use optimized image if recreate is not required

We can solve the image volume switching problem by preventing new instances that contain initial configuration from being able to recreate an existing optimized image volume. This means, when a new instance with initial configuration is being created, the LXD will check whether an image with the given fingerprint already exists. If the image volume exists and matches the instance volume requirements, or does not exist at all then the optimized image will be used. If the image volume already exists, but does not match the instance volume requirements, the image archive will be unpacked directly into the instance volume.

For example:

 # Storage pool with zfs.block_mode disabled.
lxc storage create test zfs
lxc storage set test volume.zfs.block_mode=false

# No optimized image exists at this point, therefore an optimized image volume is created
lxc launch ubuntu:22.04 c1 —storage test —device root, initial.zfs.block_mode=true

# Recreates an optimized image volume that matches the default pool settings.
lxc launch ubuntu:22.04 c2—storage test

# Because an optimized image volume already exists and does not match the instance volume
# requirements, the image archive is unpacked directly into the instance volume.
lxc launch ubuntu:22.04 c3 —storage test —device root, initial.zfs.block_mode=true

# The optimized image volume already exists and matches the instance volume requirements.
# Therefore the image volume is reused (from instance c2).
lxc launch ubuntu:22.04 c4 —storage test —device root, initial.zfs.block_mode=false
  1. Retain an optimized image volume for each volume configuration.

Finally, we could create an optimized image volume specific to each volume configuration. We would then preserve this image volume for as long as it remains in use by at least one instance.

For example:

# Storage pool with zfs.block_mode disabled.
lxc storage create test zfs
lxc storage set test volume.zfs.block_mode=false

# An optimized image is created.
lxc launch ubuntu:22.04 c1 —storage test —device root, initial.zfs.block_mode=true

# A new optimized image is created.
lxc launch ubuntu:22.04 c2—storage test

# An optimized image is reused (from instance c1)
lxc launch ubuntu:22.04 c3 —storage test —device root, initial.zfs.block_mode=true

# An optimized image is reused (from instance c2)
lxc launch ubuntu:22.04 c4 —storage test —device root, initial.zfs.block_mode=false

Examples

Applying initial configuration through profile:

lxc profile device set my-profile root initial.zfs.block_mode=true
lxc launch ubuntu:22.04 c1 —profile my-profile

Applying initial configuration when instance is being created:

lxc launch ubuntu:22.04 c1 —device root,initial.zfs.block_mode=true

API Changes:

  • None

CLI Changes:

  • None

Database Changes:

  • None

Upgrade Handling:

  • Upgrade handling is not required, as initial configuration is a new feature.
1 Like