Last October, @utkarsh announced that the Ubuntu Release Team would release Ubuntu 26.04 LTS without the ISO Tracker. Indeed, over the last 15 years, the ISO Tracker has become difficult to maintain, grown unreliable and nearly impossible to migrate to modern environments.
At the time of this bold announcement, we were planning to assess key ISO Tracker features and decide what would be the best approach to offer them to the community and flavor leads during the 26.04 cycle while we’re working on a more reliable, modern, and forward-looking solution.
These key critical functionalities needed during a testing and release week are:
- Visualizing built artifacts
- Entering manual tests results
- Setting an artifact ready for release
- Respinning an image (rebuild)
Our initial plan was to provide a stop-gap solution and track some of these needs using a combination of shared communication and Google Sheet until a better solution could go public…
Well, nobody really liked the idea to track the release week with a Google Sheet, so we worked really hard to progress our instance of an internal tool called Test Observer and close the gaps with ISO Tracker, and we are now confident enough that it can be used to track 26.04 deliverables.
Introducing tests.ubuntu.com
This new test-results tracking tool is an internal tool called Test Observer that is used in various scenarios at Canonical. We thought it was pretty close to what we needed for Ubuntu, so we got our own instance, tweaked a few workflows and UI/UX, and made it available.
Now, let’s review a few elements of the user interface that will help you transition from the ISO Tracker:
Managing Access
First, While everyone has viewer access to tests.ubuntu.com by default, you will need to log on using your Ubuntu One / Launchpad account to get extra permissions. In particular to create new tests results, you will need to be part of the ubuntu-testing team. To Log in, simply click the Log in menu on the top right corner
Listing the artifacts
This is the default view. It shows all the recent artifacts built, tested, and monitored by the release team. Various filters are available, so we’ll filter by OS version and set it to resolute:
Now, we’re only looking at 37 artifacts. These are the artifacts managed by the release team for Ubuntu and Flavors images.
Also notice the updated URL: Test Observer. It could be handy to save that one.
Artifact detail view
When you select an artifact, you will see details on the left:
- version: this drop down shows the artifact build tests you are looking at
- os: the flavor of the image (ubuntu-desktop, ubuntu, ubuntu-server, …)
- release: Ubuntu series
- owner: what team owns this artifact; this could be a flavor team or internal canonical team.
- sha256: the sha256 fingerprint of the artifact
- image link: link to the actual artifact to download
Entering manual tests results
At the moment we only display 2 test environments for each artifact: the build results from cdimage.ubuntu.com and the manual tests. The build environment will always be successful for the image to end up on tests.ubuntu.com. The focus during release weeks will be on user manual tests.
When you expand the user manual tests section, you will see a menu called “Add Test Result”
This is how we will add manual test results. These tests will be documented and accessible in GitHub under Manual test suite instructions (Work In Progress). But the Add Test Result button will pop up the dialog below.
This will be your opportunity to report manual testing against your favorite flavor. In the screenshot below, I’ve added all the tests required for the amd64 server ISO. As you see, the test name will automatically include your LaunchPad user name first.
Where to find manual testing instructions
When you decide to do manual testing against a release artifact, you will want to find test definitions for your product. We have been relocating ISO Tracker manual testing data to a new GitHub project: ubuntu/ubuntu-manual-tests The Manual test suite instructions link in tests.ubuntu.com under Manual testing will take you to this updated GitHub project.
There you should be able to identify your product within the list and see the list of mandatory tests after you select your product and follow the link to the appropriate testsuite.
To follow the testsuite link on GitHub, you have the convenient Symbolic Link button:
Make sure that the specific manual test you are looking to add has not already been done. We realize this will require some communication during release week and encourage to ask question on matrix → https://matrix.to/#/#release:ubuntu.com
Respin an image (rebuild)
Sometimes during the release week, we will hear this famous quote: “We need to respin”. This would usually mean we need to rebuild one or all the images.
Ubuntu release team members will have the ability to respin new images, but we are working on a more granular ACL allowing each flavor lead to request a respin. This work is in progress and might be ready by the 26.04 release. If not, coordination with the release team will remain necessary.
This Action is located under the cdimage.ubuntu.com build environment. You can see the Image build plan can be rerun. Clicking rerun will schedule that artifact to be rebuilt.
Setting an artifact ready for release
Once all the tests pass and an artifact owner or flavor leads is happy about the results, they are able to mark an artifact ready for release.
What’s next?
Next week, the Ubuntu release team will announce the Ubuntu 26.04 LTS Beta week, and we will be using tests.ubuntu.com for the first time to track manual testing progress. We will provide more details then on how the release team can assist every flavor lead while we experiment with this new tool.
We’re all very excited to hear your feedback as we introduce this new tool to the Ubuntu release process. There are still a few quirks, and we will continue to touch up the tool until the Beta next week. (For example, the Manual testing GitHub project needs to be rearranged for better visibility.)
Thanks in advance for your support.
