Spec: Make ISO in livefs build

[mwhudson]

Abstract

We will modify livecd-rootfs to build the ISO in the livefs build rather than relying on ubuntu-cdimage running debian-cd later.

Rationale

The fundamental reason for doing this is that having the livefs build result in the artifact we want to distribute makes everything much easier. In particular:

• testing changes to the installer-related code in livecd-rootfs
• it makes the whole image building and publication process much simpler for when we replace the current cdimage master:
  1. trigger a livefs build
  2. download one artifact
  3. publish it somewhere people can be download it from
• it allows people who want to produce ISOs that resemble the official once but differ in some way (e.g. having different or no packages in the pool, having a different set of packages installed by default, etc) much easier

Specification

Quick background on what ISOs look like today

ISOs contain three main things:

• The filesystem layers as squashfs files that are both the install source and the rootfs for the live session, and some related metadata (in /casper)
• The package pool that packages can be pulled from during offline installs (in /dists, /pool)
• Stuff to make the ISO bootable (grub, shim, kernel, initrd) (in /boot and /casper)

The package pool is signed with “the cdimage key” which is trusted by all apt installs (it is installed as /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg). Subiquity mounts the ISO at /cdrom in the target system during installation and temporarily adds an apt source to reference it during the install.

Proposed new structure

The above structure will not change very much. Instead of signing the pool with the cdimage key, we will:

• Generate a (ed25519) gpg key during the livefs build
• Sign the pool with this key
• Embed the public part in a deb822-style apt sources file referencing the pool at /cdrom
• Embed this sources file in the base rootfs layer at /etc/apt/sources.list.d/cdrom.sources (so both the live session and the target system will have the pool available to apt)

This implies a small change to subiquity (basically it does not need to add the source referencing /cdrom but it should remove the pre-existing sources file at the end of installation) and casper (we should stop running apt-cdrom to add the cdrom to the live session).

In addition, the “official” status of the ISO -- which ends up in .disk/info -- will need to be supplied to the ISO build from wherever is requesting the builds, which implies a small change to launchpad-buildd.

Implementation details

The existing implementation of debian-cd is a mix of shell and Makefiles. I’m going to take this chance to rewrite it in Python.

Building an ISO requires knowing:

• The architecture and series we are building for
• The address of the mirror to pull packages from the pool from and the components of that mirror to use
• The list of packages to include in the pool
• Where the squashfs files that contain the rootfs and other metadata layers are
• Where to put the final ISO
• All the bits of information that end up in .disk/info on the ISO and in the “volume ID” for the ISO

It’s not completely trivial to come up with a nice feeling interface between livecd-rootfs and the new tool. There are by my count about 13 parameters that are needed to build the ISO and having a tool take 13 arguments seems a bit overwhelming. In addition some steps need to run before the layers are made into squashfs files and some after. It felt nicer to have an isobuild tool with a few subcommands (7, in the end) and taking arguments relevant to each step:

isobuild --work-dir "" init --disk-id "" --series "" --arch ""

Set up the work-dir for later steps. Create the skeleton file layout of the ISO, populate .disk/info etc, create the gpg key referred to above. Store series and arch somewhere that later steps can refer to.

isobuild --work-dir "" setup-apt --mirror "" --components ""

Create an apt state directory for use by later steps, pulling the passed components from the supplied mirror. This might need to take additional architectures to enable.

isobuild --work-dir "" generate-pool --package-list-file ""

Create the pool from the passed germinate output file.

isobuild --work-dir "" generate-sources --mountpoint ""

Generate an apt deb822 source for the pool, assuming it is mounted at the passed mountpoint, and output it on stdout.

isobuild --work-dir "" add-live-filesystem --artifact-prefix ""

Copy the relevant artifacts to the casper directory (and extract the uuids from the initrds)

isobuild --work-dir "" make-bootable --project "" --capitalized-project "" --subarch "" 

Set up the bootloader etc so that the ISO can boot (for now I plan to clone debian-cd and run the tools/boot/$series-$arch script but those should be folded into isobuild fairly promptly IMO).

isobuild --work-dir "" make-iso --vol-id "" --dest ""

Generate the checksum file and run xorriso to build the final ISO.

Transition plan

1. Submit MP to launchpad-buildd to allow specifying “official”
2. Add code to ubuntu-cdimage to allow specifying official for a build
   1. As unrecognized keys in metadata are ignored we can do these in parallel
3. Add code to livecd-rootfs to build the ISO for all images that build an installer
4. For all builds that produce an ISO
   1. Validate the ISO produced by the livefs works
   2. Change ubuntu-cdimage to publish the ISO from the livefs rather than running the other artifacts through debian-cd
5. When we stop building noble ISOs we can drop debian-cd from cdimage (it’s a bit unclear when this will be; we have had to issue focal and bionic ISOs well after we would routinely stop building them to address bootability issues)
6. Eventually, remove Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com> from the set of keys apt trusts by default for Ubuntu

Risks

• we will be running xorriso from the target series and architecture
  • this might not behave identically to the fixed amd64 version of xorriso we use today
  • it might be very slow on riscv64
  • if this part proves to be a problem, we can have the output of the build be a tarball of the input to xorriso, but hopefully not
• the change to the key the pool is signed with changes the security properties a bit (mostly for the better though afaics)

[mwhudson]

I have made a livecd-rootfs merge proposal now, which is the meat of this work.

[mwhudson]

An update here, I’ve uploaded a version of livecd-rootfs that builds an ISO alongside all the other pieces that were already built. These ISOs will be ignored by the publication machinery for now, so what we need to do is test that they work OK and then switch ubuntu-cdimage over to publishing the livefs-built ISO.

I have made a spreadsheet listing all the installers we make to track this: https://docs.google.com/spreadsheets/d/1O-8eGl3SNp2Diuk_0NaH4I9Z9d6WFkrxmaOspguy8zw/edit?usp=sharing.

If you are able to, testing these ISOs would be super appreciated! Basically for each one, head to the linked livefs, click on the most recent build, download the ISO (it will be called something like livecd.kubuntu.iso) and test it as you usually would. No testing beyond the basics is really needed (honestly, knowing that the ISO boots is probably the most interesting part!) but it might be worthwhile to test both an online and offline install as the way the package pool on the ISO is handled has changed a bit.

[eeickmeyer]

Well, I found this glaring issue:

The “RELEASE” likely happens because it’s pulling from the .desktop file in /var/lib/snapd/desktop/applications(ubuntu-desktop-bootstrap_ubuntu-desktop-bootstrap.desktop) which has the following contents (comments mine):

[Desktop Entry]
X-SnapInstanceName=ubuntu-desktop-bootstrap
Type=Application
Version=1.0
Name=Install RELEASE # ←--Here’s the problem
Comment=Install this system permanently to your hard disk
Keywords=ubiquity;
X-SnapAppName=ubuntu-desktop-bootstrap
Exec=/snap/bin/ubuntu-desktop-bootstrap
Icon=ubiquity
Terminal=false
Categories=GTK;System;Settings

So, this is likely a result of the Name= field being populated and not overridden. Maybe casper should copy the file to /usr/share/applications and edit that? Just a thought.

[mwhudson]

The thing is casper does edit that file! (in place, not by copying it). I wonder why that code is not running in this case…

[mwhudson]

Tobias pointed out some problems with the arm64 ISO which lead to finding a bit of path confusion in my new code, I have a merge proposal up to fix this at OpenID transaction in progress

[mwhudson]

OK my changes landed so it would be interesting to test the ISOs again!

[arraybolt3]

As of a few minutes ago, there is now a calamares-settings-ubuntu upload that will make Lubuntu, Kubuntu, and Ubuntu Unity work with the new ISO build mechanism.

Some bugs encountered in the process:

• Kubuntu’s ISO is too small to be right. 660.6 MiB when the squashfs file is 4.3 GiB. Ubuntu Unity seems to have the same problem (113.6 MiB ISO, 3.9 GiB squashfs). Lubuntu’s ISO is being built without immediately visible issues though.
• casper is still inserting an apt-cdrom line in /etc/apt/sources.list. In practice, this could be totally ignored and Calamares-based flavors would be fine; my code changes ensure this line is ignored, and the file doesn’t end up on the installed system at the end of the installation process.

[mwhudson]

I’ve fixed the kubuntu thing now, the ISOs are now at least a more plausible size.

Does the casper thing cause issues beyond the cosmetic? My thinking is to disable this once we’re publishing the livefs built ISOs on cdimage.

[mwhudson]

A slightly belated update, all the code has merge and we are now publishing the ISOs that livecd-rootfs makes for resolute on cdimage.ubuntu.com! There have been a few wrinkles along the way but as far as I know everything should be working on these ISOs now.