Retiring the ISO Tracker! Introducing tests.ubuntu.com for 26.04 LTS

[mclemenceau]

Last October, @utkarsh announced that the Ubuntu Release Team would release Ubuntu 26.04 LTS without the ISO Tracker. Indeed, over the last 15 years, the ISO Tracker has become difficult to maintain, grown unreliable and nearly impossible to migrate to modern environments.

At the time of this bold announcement, we were planning to assess key ISO Tracker features and decide what would be the best approach to offer them to the community and flavor leads during the 26.04 cycle while we’re working on a more reliable, modern, and forward-looking solution.

These key critical functionalities needed during a testing and release week are:

• Visualizing built artifacts
• Entering manual tests results
• Setting an artifact ready for release
• Respinning an image (rebuild)

Our initial plan was to provide a stop-gap solution and track some of these needs using a combination of shared communication and Google Sheet until a better solution could go public…

Well, nobody really liked the idea to track the release week with a Google Sheet, so we worked really hard to progress our instance of an internal tool called Test Observer and close the gaps with ISO Tracker, and we are now confident enough that it can be used to track 26.04 deliverables.

Introducing tests.ubuntu.com

This new test-results tracking tool is an internal tool called Test Observer that is used in various scenarios at Canonical. We thought it was pretty close to what we needed for Ubuntu, so we got our own instance, tweaked a few workflows and UI/UX, and made it available.

Now, let’s review a few elements of the user interface that will help you transition from the ISO Tracker:

Managing Access

First, While everyone has viewer access to tests.ubuntu.com by default, you will need to log on using your Ubuntu One / Launchpad account to get extra permissions. In particular to create new tests results, you will need to be part of the ubuntu-testing team. To Log in, simply click the Log in menu on the top right corner

Listing the artifacts

This is the default view. It shows all the recent artifacts built, tested, and monitored by the release team. Various filters are available, so we’ll filter by OS version and set it to resolute:

Now, we’re only looking at 37 artifacts. These are the artifacts managed by the release team for Ubuntu and Flavors images.

Also notice the updated URL: https://tests.ubuntu.com/#/images?Release=resolute. It could be handy to save that one.

Artifact detail view

When you select an artifact, you will see details on the left:

• version: this drop down shows the artifact build tests you are looking at
• os: the flavor of the image (ubuntu-desktop, ubuntu, ubuntu-server, …)
• release: Ubuntu series
• owner: what team owns this artifact; this could be a flavor team or internal canonical team.
• sha256: the sha256 fingerprint of the artifact
• image link: link to the actual artifact to download

Entering manual tests results

At the moment we only display 2 test environments for each artifact: the build results from cdimage.ubuntu.com and the manual tests. The build environment will always be successful for the image to end up on tests.ubuntu.com. The focus during release weeks will be on user manual tests.

When you expand the user manual tests section, you will see a menu called “Add Test Result”

This is how we will add manual test results. These tests will be documented and accessible in GitHub under Manual test suite instructions (Work In Progress). But the Add Test Result button will pop up the dialog below.

This will be your opportunity to report manual testing against your favorite flavor. In the screenshot below, I’ve added all the tests required for the amd64 server ISO. As you see, the test name will automatically include your LaunchPad user name first.

Where to find manual testing instructions

When you decide to do manual testing against a release artifact, you will want to find test definitions for your product. We have been relocating ISO Tracker manual testing data to a new GitHub project: ubuntu/ubuntu-manual-tests The Manual test suite instructions link in tests.ubuntu.com under Manual testing will take you to this updated GitHub project.

There you should be able to identify your product within the list and see the list of mandatory tests after you select your product and follow the link to the appropriate testsuite.

To follow the testsuite link on GitHub, you have the convenient Symbolic Link button:

Make sure that the specific manual test you are looking to add has not already been done. We realize this will require some communication during release week and encourage to ask question on matrix → https://matrix.to/#/#release:ubuntu.com

Respin an image (rebuild)

Sometimes during the release week, we will hear this famous quote: “We need to respin”. This would usually mean we need to rebuild one or all the images.

Ubuntu release team members will have the ability to respin new images, but we are working on a more granular ACL allowing each flavor lead to request a respin. This work is in progress and might be ready by the 26.04 release. If not, coordination with the release team will remain necessary.

This Action is located under the cdimage.ubuntu.com build environment. You can see the Image build plan can be rerun. Clicking rerun will schedule that artifact to be rebuilt.

Setting an artifact ready for release

Once all the tests pass and an artifact owner or flavor leads is happy about the results, they are able to mark an artifact ready for release.

What’s next?

Next week, the Ubuntu release team will announce the Ubuntu 26.04 LTS Beta week, and we will be using tests.ubuntu.com for the first time to track manual testing progress. We will provide more details then on how the release team can assist every flavor lead while we experiment with this new tool.

We’re all very excited to hear your feedback as we introduce this new tool to the Ubuntu release process. There are still a few quirks, and we will continue to touch up the tool until the Beta next week. (For example, the Manual testing GitHub project needs to be rearranged for better visibility.)

Thanks in advance for your support.

[MesterPerfect]

Thanks for the update and for the effort put into modernizing the testing workflow.

I would like to raise an accessibility concern regarding the new tool (tests.ubuntu.com).

From initial impressions, it appears to be built with Flutter, and unfortunately Flutter-based interfaces on Linux/web still have significant accessibility limitations, especially with screen readers like Orca. In many cases, UI elements are not properly exposed via AT-SPI, making navigation and interaction very difficult or sometimes impossible.

As a result, I’m concerned that this new tool may not be usable for blind or visually impaired contributors, which could unintentionally exclude part of the community from participating in testing and release processes.

Given that testing is a collaborative effort across the Ubuntu community, accessibility should be considered a core requirement, not an optional enhancement.

It would be very helpful if:

• Accessibility support (especially Orca compatibility) is evaluated early
• Proper semantic roles and keyboard navigation are ensured
• And ideally, testing with real assistive technology users is included

Otherwise, contributors who rely on screen readers may not be able to use the platform at all.

Thank you again for the work, and I hope accessibility can be taken into account as the tool evolves.

[skia]

Yes, that tool was historically built on Flutter, but this is currently annoying everybody (the Test Observer dev team, the Ubuntu Release Team that deployed it, users like you…). There are experiments to try to move away from it, but nothing is ready yet, unfortunately

[itzswirlz]

If we do stick with flutter, can we maybe have an option to have it as a desktop application?

[corradoventu]

Well, i installed successfully from dangerous-resolute-desktop-amd64.iso and i’m on this page. How can I report my test? Thank you.

[nio-wiklund]

The web site http://tests.ubuntu.com/ is ‘world wide wait’ for me: I wait for several minutes but nothing is shown. Finally an error code is printed. Is this a temporary error, or is the access limited to some internal group of users?

[leok]

I have no problem accessing the page - maybe clear cache on browser and try again?

[nio-wiklund]

Thanks @leok, your advice helped me.

[skia]

One little tiny detail we completely forgot to mention in that tutorial: people need to be part of the ~ubuntu-testing open team on Launchpad, and then logout/login from https://tests.ubuntu.com, before they get the permission to actually submit test results.

So sorry for the inconvenience, thankfully the post will be edited to mention that before the call for testing the beta is out

[diogoconstantino]

This is nice and necessary, but I want to to call the attention to the problem of test_observer not having license….

[arraybolt3]

I believe the licenses are here:

test_observer/LICENSES at main · canonical/test_observer

There doesn’t seem to be a single overarching license, but given that (some of) the code is AGPL-3.0-only, and that seems to be the strictest license there, I think it’s safe to say that’s the effective license.

[diogoconstantino]

I’m not at all confident that complies with the requirements for announcing the licence according to the AGPL.

[arraybolt3]

This probably isn’t the place to debate the details of software licensing, but it’s worth noting that the AGPL makes the license announcement conditional on creating a derivative work:

…if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software.

Assuming Canonical is the copyright owner of those parts of the code that are AGPL-licensed (which is likely given that they wrote Test Observer), they aren’t obligated to put the notice there. Anyone who creates a modified version of Test Observer and deploys it needs to put a source code link on it.

[scarlettmoore]

I was able to submit test results, but I was not able to attach issues ( I assume this is where we want to add bug reports and not comments like I originally did. I get insufficient permissions. I am with Kubuntu.

[skia]

Thanks for reporting that. We’ll make sure to fix that before the final release.

[mlueck]

I am trying to log a defect with the new tool. I already logged the defect against the package. Now I would like to note that for me the daily build failed. I try adding it as a fail. I am pasting in the URL from the defect case logged, and typing a comment. Always it states failed / not allowed.

https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/2147165

[SergioCostas]

Nice work! Just a suggestion: could you add the sticky property to the table header CSS to make it always visible, even when scrolling? I hate having a lot of columns and having to go to the top to see what each one means after I found the row I wanted.

[mclemenceau]

Thanks @SergioCostas , I have a PR for review. Hope to get it approved.

[leok]

When logging into TestObserver I get the following error: Error insufficient permissions
However I can continue onward and file results etc ..