Spec: Make ISO in livefs build

Project Discussion Foundations
1

Abstract

We will modify livecd-rootfs to build the ISO in the livefs build rather than relying on ubuntu-cdimage running debian-cd later.

Rationale

The fundamental reason for doing this is that having the livefs build result in the artifact we want to distribute makes everything much easier. In particular:

  • testing changes to the installer-related code in livecd-rootfs
  • it makes the whole image building and publication process much simpler for when we replace the current cdimage master:
    1. trigger a livefs build
    2. download one artifact
    3. publish it somewhere people can be download it from
  • it allows people who want to produce ISOs that resemble the official once but differ in some way (e.g. having different or no packages in the pool, having a different set of packages installed by default, etc) much easier

Specification

Quick background on what ISOs look like today

ISOs contain three main things:

  • The filesystem layers as squashfs files that are both the install source and the rootfs for the live session, and some related metadata (in /casper)
  • The package pool that packages can be pulled from during offline installs (in /dists, /pool)
  • Stuff to make the ISO bootable (grub, shim, kernel, initrd) (in /boot and /casper)

The package pool is signed with “the cdimage key” which is trusted by all apt installs (it is installed as /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg). Subiquity mounts the ISO at /cdrom in the target system during installation and temporarily adds an apt source to reference it during the install.

Proposed new structure

The above structure will not change very much. Instead of signing the pool with the cdimage key, we will:

  • Generate a (ed25519) gpg key during the livefs build
  • Sign the pool with this key
  • Embed the public part in a deb822-style apt sources file referencing the pool at /cdrom
  • Embed this sources file in the base rootfs layer at /etc/apt/sources.list.d/cdrom.sources (so both the live session and the target system will have the pool available to apt)

This implies a small change to subiquity (basically it does not need to add the source referencing /cdrom but it should remove the pre-existing sources file at the end of installation) and casper (we should stop running apt-cdrom to add the cdrom to the live session).

In addition, the “official” status of the ISO -- which ends up in .disk/info -- will need to be supplied to the ISO build from wherever is requesting the builds, which implies a small change to launchpad-buildd.

Implementation details

The existing implementation of debian-cd is a mix of shell and Makefiles. I’m going to take this chance to rewrite it in Python.

Building an ISO requires knowing:

  • The architecture and series we are building for
  • The address of the mirror to pull packages from the pool from and the components of that mirror to use
  • The list of packages to include in the pool
  • Where the squashfs files that contain the rootfs and other metadata layers are
  • Where to put the final ISO
  • All the bits of information that end up in .disk/info on the ISO and in the “volume ID” for the ISO

It’s not completely trivial to come up with a nice feeling interface between livecd-rootfs and the new tool. There are by my count about 13 parameters that are needed to build the ISO and having a tool take 13 arguments seems a bit overwhelming. In addition some steps need to run before the layers are made into squashfs files and some after. It felt nicer to have an isobuild tool with a few subcommands (7, in the end) and taking arguments relevant to each step:

isobuild --work-dir "" init --disk-id "" --series "" --arch ""

Set up the work-dir for later steps. Create the skeleton file layout of the ISO, populate .disk/info etc, create the gpg key referred to above. Store series and arch somewhere that later steps can refer to.

isobuild --work-dir "" setup-apt --mirror "" --components ""

Create an apt state directory for use by later steps, pulling the passed components from the supplied mirror. This might need to take additional architectures to enable.

isobuild --work-dir "" generate-pool --package-list-file ""

Create the pool from the passed germinate output file.

isobuild --work-dir "" generate-sources --mountpoint ""

Generate an apt deb822 source for the pool, assuming it is mounted at the passed mountpoint, and output it on stdout.

isobuild --work-dir "" add-live-filesystem --artifact-prefix ""

Copy the relevant artifacts to the casper directory (and extract the uuids from the initrds)

isobuild --work-dir "" make-bootable --project "" --capitalized-project "" --subarch ""

Set up the bootloader etc so that the ISO can boot (for now I plan to clone debian-cd and run the tools/boot/$series-$arch script but those should be folded into isobuild fairly promptly IMO).

isobuild --work-dir "" make-iso --vol-id "" --dest ""

Generate the checksum file and run xorriso to build the final ISO.

Transition plan

  1. Submit MP to launchpad-buildd to allow specifying “official”
  2. Add code to ubuntu-cdimage to allow specifying official for a build
    1. As unrecognized keys in metadata are ignored we can do these in parallel
  3. Add code to livecd-rootfs to build the ISO for all images that build an installer
  4. For all builds that produce an ISO
    1. Validate the ISO produced by the livefs works
    2. Change ubuntu-cdimage to publish the ISO from the livefs rather than running the other artifacts through debian-cd
  5. When we stop building noble ISOs we can drop debian-cd from cdimage (it’s a bit unclear when this will be; we have had to issue focal and bionic ISOs well after we would routinely stop building them to address bootability issues)
  6. Eventually, remove Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com> from the set of keys apt trusts by default for Ubuntu

Risks

  • we will be running xorriso from the target series and architecture
    • this might not behave identically to the fixed amd64 version of xorriso we use today
    • it might be very slow on riscv64
    • if this part proves to be a problem, we can have the output of the build be a tarball of the input to xorriso, but hopefully not
  • the change to the key the pool is signed with changes the security properties a bit (mostly for the better though afaics)
7 Likes
2

I have made a livecd-rootfs merge proposal now, which is the meat of this work.

2 Likes