Background
This post demonstrates how to customize Ubuntu LXD cloud image metadata, templates and rootfs in LXD to provide custom cloud-init datasources or meta-data directives on first boot. From time to time, the images produced by your favorite vendor don’t quite meet your needs.
In this post you will learn:
- Where to download Ubuntu supported cloud-images
- How LXD uses metadata templates to configure cloud-init
- How to alter LXD metadata to customize your deployments
Why changes are needed
Some reasons to customize images before launch may include:
- securing applications before any third party code runs
- removing undesirable image content or packages
- custom config before services are started
- manipulating the default initial boot or network configuration performed by cloud-init
In my case, cloud-init v.21.4 release introduced a new LXD datasource which provides better integration with LXD than the existing NoCloud datasource.
Since cloud-init doesn’t typically change default behavior on stable Ubuntu releases: 18.04, 20.04, 21.10, the NoCloud datasource will continue to be detected by cloud-init instead of the new LXD datasource. Since we prefer LXD datasouce over NoCloud, we’ll need to roll our own custom images.
LXD image metadata
Canonical produces daily builds of cloud-images for each Ubuntu release at a URLs like:
Among many other compressed formats, LXD cloud image support can be provided by split files which deliver LXD metadata templates (focal-server-cloudimg-amd64-lxd.tar.xz) and the base root filesystem (focal-server-cloudimg-squashfs).
The compressed LXD metadata and templates contains a metadata.yaml and templates subdirectory. The LXD metadata.yaml describes which files to write during instance create, copy or start. Current Ubuntu images, and many other Linux flavors, provide NoCloud seed files to the /var/lib/cloud/seed/nocloud-net/ directory as defined in metadata.yaml under the “templates” key like the following:
templates:
/var/lib/cloud/seed/nocloud-net/meta-data:
when:
- create
- copy
template: cloud-init-meta.tpl
The YAML above tells LXD to create the file /var/lib/cloud/seed/nocloud-net/meta-data when the instance is either created or copied based on the template file templates/cloud-init-meta.tpl. These templates will be dropped in LXD 2.24.
Downloading official images
Obtain both the lxd compressed metadata and the squashfs image from latest official daily Ubuntu cloud-images for your release of choice.
$ export RELEASE=jammy
$ wget https://cloud-images.ubuntu.com/daily/server/${RELEASE}/current/${RELEASE}-server-cloudimg-amd64-lxd.tar.xz
$ wget https://cloud-images.ubuntu.com/daily/server/${RELEASE}/current/${RELEASE}-server-cloudimg-amd64.squashfs
Note: Cloud-init 21.4 has not yet published to 18.04, 20.04 or 21.10 so LXD datasource is not yet present. Once the cloud-init SRU bug LP: #1949521 is complete, then hack away at Bionic, Focal, Hirsute and Impish.
Changing LXD metadata for official Ubuntu images
Given LXD sources image metadata and rootfs files, we have a couple of ways to “fix” our stable Ubuntu images to prefer LXD datasource over NoCloud.
- Adapt LXD image metadata templates to inject /etc/cloud/cloud.cfg.d/90-use-lxd.cfg during instance launch to tell cloud-init to prefer LXD before NoCloud.
- Adapt LXD image metadata templates, dropping nocloud seed files from the metadata so nocloud will not be discovered.
Note: This assumes you already have LXD configured on your system. In not; time to set it up.
Option 1: Add custom /etc/cloud/cloud.cfg.d/90-use-lxd.cfg via LXD metadata.yaml
This option alters only the metadata.yaml to lay down the 90-use-lxd.cfg file on the filesystem to tell cloud-init to discover the LXD datasource is before NoCloud. The unaltered upstream squashfs in this case could still be used with other container launches (yet a different metadata.yaml) and still retain cloud-init’s NoCloud datasource.
# Uncompress original LXD metadata
$ tar xf ${RELEASE}-server-cloudimg-amd64-lxd.tar.xz
# Add directives to create /etc/cloud/cloud.cfg.d/95-use-lxd.cfg
$ cat > templates/cloud-init-use-lxd.tpl <<EOF
# Added by LXD metadata.yaml
datasource_list: [ LXD, NoCloud ]
EOF
$ cat > add-lxd.yaml <<EOF
/etc/cloud/cloud.cfg.d/95-use-lxd.cfg:
when:
- create
- copy
template: cloud-init-use-lxd.tpl
EOF
$ cat add-lxd.yaml >> metadata.yaml
# Compress LXD metadata and templates
$ tar -czf ${RELEASE}-server-cloudimg-amd64-prefer-lxd.tar.xz metadata.yaml templates/
$ lxc image import ${RELEASE}-server-cloudimg-amd64-prefer-lxd.tar.xz ${RELEASE}-server-cloudimg-amd64.squashfs --alias prefer-lxd-md-${RELEASE}
$ lxc launch prefer-lxd-md-${RELEASE} test-extend-lxd-metadata
# Check LXD is aware of the new template
$ lxc config template list test-extend-lxd-metadata
...
| cloud-init-use-lxd.tpl |
...
$ lxc config template show test-extend-lxd-metadata cloud-init-use-lxd.tpl
# Added by LXD metadata.yaml
datasource_list: [ LXD, NoCloud ]
# Confirm LXD datasource is using the LXD socket API
$ lxc exec test-extend-lxd-metadata cloud-init query subplatform
LXD socket API v. 1.0 (/dev/lxd/sock)
Note: When defining templates in LXD metadata.yaml, the lxd command line will allow the caller to update the values of those template files at container launch via lxc config template edit <YOUR_CONTAINER> cloud-init-use-lxd.tpl. The updated value will be written to the container during launch.
Option 2: Redact NoCloud seed templates from LXD metadata.yaml and templates
This option will no longer create NoCloud seed files at container launch time, cloud-init will not detect NoCloud and consequently fallback to discovery of other datasources with less precedence, such as LXD.
# Redact all /var/lib/cloud/seed/nocloud-net/* template definitions
$ head -n 13 metadata.yaml > redacted-metadata.yaml
$ mv redacted-metadata.yaml metadata.yaml
# Remove the template files from templates subdirectory
$ rm templates/cloud-init-*
# Recreate the LXD metadata gz
$ tar -czf ${RELEASE}-server-cloudimg-amd64-redacted-lxd.tar.xz metadata.yaml templates/
# Import the new LXD image ${RELEASE}-redacted-md
$ lxc image import ${RELEASE}-server-cloudimg-amd64-redacted-lxd.tar.xz ${RELEASE}-server-cloudimg-amd64.squashfs --alias ${RELEASE}-redacted-md
# Launch our new container
$ lxc launch ${RELEASE}-redacted-md test-redacted-md
# Confirm metadata templates provided by LXD to the launched container
# Confirm minimal templates listed
$ lxc config template list test-redacted-md
+--------------+
| FILENAME |
+--------------+
| hostname.tpl |
+--------------+
# Check the LXD API is detected
$ lxc exec test-redacted-md cloud-init query subplatform
LXD socket API v. 1.0 (/dev/lxd/sock)
Conclusion
In either of these cases cloud-init will detect the LXD datasource instead of NoCloud. Option 1 is preferable as it still leaves the option for cloud-init to fallback to use NoCloud datasource in the event that the LXD datasource is not discoverable due to LXD configuration settings security.devlxd = False. By defining or extending LXD templates in metadata.yaml custom system configuration can be written to the filesystem before the container is first launched. This allows the first container launch to be exactly the custom image content you desire.
Thanks for your time and happy hacking.
Cross posted from blackbox.github.io: customize lxd metadata templates