Testing Ubuntu Release Upgrades

Quality
1

Upgrade testing

When releasing a new Ubuntu release or point release one important QA step consists in ensuring that any valid upgrade path from a previous release to the new release works correctly, that is: completes without errors, and after a reboot the upgraded system boots and behaves as expected.

Of course, this can be done manually by installing the release to be upgraded, ensuring that it’s fully up-to-date (apt full-upgrade), running do-release-upgrade, rebooting, and checking if the state of the system corresponds to the expected one. However, other than consuming precious human time, this has other drawbacks: the test platform and the initial and final states of the system are not well defined, so setting up a reproducer after a bug has been found is not straightforward. Moreover, it’s desirable to run upgrade tests daily and not only at release time, to make sure that upgrade regressions are spotted early. There is, in other words, the need to automate the upgrade testing process.

Automated upgrade tests

The auto-upgrade-testing package automates upgrade tests leveraging the autopkgtest tools. The package has been in use internally for a long time, but wasn’t present in the Ubuntu package archive (at least, not for any supported release). The package has now been brought to more moderns standards and uploaded to the archive, and it’s available for installation on Kinetic.

The tool takes an upgrade profile as its input, which defines things like:

  • which autopkgtest virtualization server to use to run the upgrade test
  • the initial and final Ubuntu releases
  • how to prepare the initial system, which normally means installing an Ubuntu metapackage. This easily allows for testing flavors by using an upgrade profile that installs a flavour metapackage (e.g. xubuntu-desktop)
  • what to check before the upgrade is done (e.g. the release is the expected one)
  • what to check after the upgrade is done (e.g. the metapackage is still installed).

auto-upgrade-testing then generates a dummy package with an autopkgtest that does the release upgrade according to the profile, then starts an autopkgtest run against the initial release. This way almost everything is delegated to autopkgtest, and auto-upgrade-testing will only collect the test results from it.

How to use it

First of all, let’s install the package:

apt install auto-upgrade-testing

Note that it’s only available on Kinetic for now.

Several upgrade profiles are already defined in the auto-upgrade-testing-specifications Launchpad project:

git clone https://git.launchpad.net/auto-upgrade-testing-specifications

All these upgrade profiles use the qemu autopkgtest virt server (autopkgtest-virt-qemu), so make sure that the user launching auto-upgrade-testing is in the kvm group. The tool doesn’t require root.

Now an upgrade test can be launched with:

auto-upgrade-testing --config <profile.yaml> --provision

For example:

auto-upgrade-testing --config /path/to/auto-upgrade-testing-specifications/profiles/ubuntu-bionic-focal-basic-amd64_qemu.yaml --provision

By using this profile the “base” bionic image will be built using autopkgtest-build-qemu, prepared according to the upgrade profile, and cached under /var/cache/auto-upgrade-testing/. Images can safely be removed from there: they’ll be recreated as needed.

Details on how to write an upgrade profile will be covered in a follow-up post.

Corner cases

  • auto-upgrade-testing follows the do-release-upgrade logic to choose which release to upgrade to. If multiple upgrades are needed to reach the target release then auto-upgrade-testing will do so, possibly rebooting the testbed system several times.
  • When testing upgrades from an LTS release to the following LTS release, auto-upgrade-testing will always prefer the LTS-to-LTS path over the LTS-to-interim-to-LTS path, even if LTS-to-LTS upgrades are not enabled yet. This is by design and is accomplished by calling do-release-upgrade -d when necessary.
  • Testing upgrades from EOL releases requires adjusting the creation of the initial image so that old-releases.ubuntu.com is used in sources.list. Refer to the autopkgtest tooling manpages on how to do that (e.g. see $AUTOPKGTEST_APT_SOURCES_FILE in autopkgtest-build-qemu(1)).

Limitations

  • As noted above the package is available only on Kinetic for now (currently the Ubuntu development release). However it has been already tested on Jammy, and backporting it to Jammy using the Ubuntu Backports process is in the plans.
  • The tool runs the upgrade with DEBIAN_FRONTEND=noninteractive, meaning that it won’t help spotting blocking debconf questions, which are often to be considered upgrade bugs.
  • For the moment the only tested and maintained virtualization backend is qemu (autopkgtest-virt-qemu).

Contributing and reporting bugs

The tool source code is hosted on GitHub:

https://github.com/canonical/auto-upgrade-testing

while bugs are tracked on Lauchpad:

4 Likes
2

Adding a new release

To enable upgrade testing to a new development release we need to add the upgrade profiles we’re interested in. Upgrade profiles are maintained in git under the profiles/ directory of this repository:

And this is how an upgrade profile looks like:

- testname: "focal->jammy desktop upgrade"
  provisioning:
    backend: qemu
    arch: amd64
    flavor: ubuntu
    image_name: focal_jammy_ubuntu_desktop_amd64.img
    build_args:
      - "--post-command 'sudo apt-get -y update && sudo apt-get -y dist-upgrade && sudo DEBIAN_FRONTEND=noninteractive apt-get -y install ubuntu-desktop^' -v --timeout 14400"
    releases:
      - "focal"
      - "jammy"

  script_location: "file://../tests"
  pre_upgrade_scripts:
    - "check-ubuntu-desktop"

  post_upgrade_tests:
    - "null"

The interesting bits of this config file are:

  • image_name: the base image to test, i.e. an image the Ubuntu system to upgrade. The auto-upgrade-testing tool will create it automatically if it doesn’t exist.
  • build_args: arguments passed to autopkgtest-buildvm-ubuntu-cloud when building the base image (image_name). Normally used to prepare a specific image type by installing a metapackage; in the example above we’re preparing a desktop image by installing ubuntu-desktop.
  • releases: the initial release and final release to get to via (potentially multiple) do-release-upgrade runs.
  • script_location: location of the scripts (the actual tests) to run on the system under test
  • pre_upgrade_scripts: script (test) to run before beginning the upgrade process
  • post_upgrade_tests: script (test) to run after the upgrade process

Knowing this we can easily create new upgrade profile. As an example here is the commit adding Kinetic -> Lunar profiles.

3 Likes