Testing Ubuntu Release Upgrades

Upgrade testing

When releasing a new Ubuntu release or point release one important QA step consists in ensuring that any valid upgrade path from a previous release to the new release works correctly, that is: completes without errors, and after a reboot the upgraded system boots and behaves as expected.

Of course, this can be done manually by installing the release to be upgraded, ensuring that it’s fully up-to-date (apt full-upgrade), running do-release-upgrade, rebooting, and checking if the state of the system corresponds to the expected one. However, other than consuming precious human time, this has other drawbacks: the test platform and the initial and final states of the system are not well defined, so setting up a reproducer after a bug has been found is not straightforward. Moreover, it’s desirable to run upgrade tests daily and not only at release time, to make sure that upgrade regressions are spotted early. There is, in other words, the need to automate the upgrade testing process.

Automated upgrade tests

The auto-upgrade-testing package automates upgrade tests leveraging the autopkgtest tools. The package has been in use internally for a long time, but wasn’t present in the Ubuntu package archive (at least, not for any supported release). The package has now been brought to more moderns standards and uploaded to the archive, and it’s available for installation on Kinetic.

The tool takes an upgrade profile as its input, which defines things like:

  • which autopkgtest virtualization server to use to run the upgrade test
  • the initial and final Ubuntu releases
  • how to prepare the initial system, which normally means installing an Ubuntu metapackage. This easily allows for testing flavors by using an upgrade profile that installs a flavour metapackage (e.g. xubuntu-desktop)
  • what to check before the upgrade is done (e.g. the release is the expected one)
  • what to check after the upgrade is done (e.g. the metapackage is still installed).

auto-upgrade-testing then generates a dummy package with an autopkgtest that does the release upgrade according to the profile, then starts an autopkgtest run against the initial release. This way almost everything is delegated to autopkgtest, and auto-upgrade-testing will only collect the test results from it.

How to use it

First of all, let’s install the package:

apt install auto-upgrade-testing

Note that it’s only available on Kinetic for now.

Several upgrade profiles are already defined in the auto-upgrade-testing-specifications Launchpad project:

git clone https://git.launchpad.net/auto-upgrade-testing-specifications

All these upgrade profiles use the qemu autopkgtest virt server (autopkgtest-virt-qemu), so make sure that the user launching auto-upgrade-testing is in the kvm group. The tool doesn’t require root.

Now an upgrade test can be launched with:

auto-upgrade-testing --config <profile.yaml> --provision

For example:

auto-upgrade-testing --config /path/to/auto-upgrade-testing-specifications/profiles/ubuntu-bionic-focal-basic-amd64_qemu.yaml --provision

By using this profile the “base” bionic image will be built using autopkgtest-build-qemu, prepared according to the upgrade profile, and cached under /var/cache/auto-upgrade-testing/. Images can safely be removed from there: they’ll be recreated as needed.

Details on how to write an upgrade profile will be covered in a follow-up post.

Corner cases

  • auto-upgrade-testing follows the do-release-upgrade logic to choose which release to upgrade to. If multiple upgrades are needed to reach the target release then auto-upgrade-testing will do so, possibly rebooting the testbed system several times.
  • When testing upgrades from an LTS release to the following LTS release, auto-upgrade-testing will always prefer the LTS-to-LTS path over the LTS-to-interim-to-LTS path, even if LTS-to-LTS upgrades are not enabled yet. This is by design and is accomplished by calling do-release-upgrade -d when necessary.
  • Testing upgrades from EOL releases requires adjusting the creation of the initial image so that old-releases.ubuntu.com is used in sources.list. Refer to the autopkgtest tooling manpages on how to do that (e.g. see $AUTOPKGTEST_APT_SOURCES_FILE in autopkgtest-build-qemu(1)).

Limitations

  • As noted above the package is available only on Kinetic for now (currently the Ubuntu development release). However it has been already tested on Jammy, and backporting it to Jammy using the Ubuntu Backports process is in the plans.
  • The tool runs the upgrade with DEBIAN_FRONTEND=noninteractive, meaning that it won’t help spotting blocking debconf questions, which are often to be considered upgrade bugs.
  • For the moment the only tested and maintained virtualization backend is qemu (autopkgtest-virt-qemu).

Contributing and reporting bugs

The tool source code is hosted on GitHub:

https://github.com/canonical/auto-upgrade-testing

while bugs are tracked on Lauchpad:

4 Likes