Autopkgtest Service

Project Discussion Quality
1

This post is used to convey information about the autopkgtest service which is managed by the Ubuntu QA team and any issues with that service. Bugs about the service should be reported in Launchpad.

Current Status

  • A not fully triaged issue in the lcy02 data center is causing lower throughput in processing amd64 and i386 tests. This is being investigated. This issue will commonly manifest itself as VirtSubproc.Timeout: 1800, with a prior traceback, and test results will commonly fail with “unknown”. There are times where lcy02 functions without issue, but it is often in the state described. Note that only a minority of jobs is being run on lcy02, so the impact of this issue is limited.
  • The infrastructure is currently facing issues in spawning ppc64el testbeds. This is currently being investigated (RT: #167521).

If you think you’ve found an issue that is not already documented or for which a bug report has not been created please mention qa-help or ping @paride, @andersson123, or @hyask in #ubuntu-devel on libera.chat.

8 Likes
4

Major Environment Changes

  • 2025-01-21 - New riscv64 workers have been deployed. These run in emulation on top of amd64 hardware, which means the tests will be slow. Britney is not pushing tests for this architecture (yet), so no migration will be gated on that. Please reach out if you encounter any weird issue.
  • 2024-11-28 - A new autopkgtest version was deployed (based on upstream version 5.42, plus additional patches), which include lots of changes, both bugfixes and features. We don’t expect things to go too bad, but please keep an eye on corner cases where things might not have been completely tested.
  • 2024-11-07 - Workers in bos03-amd64 were added to spread the load across both PS5 and PS6. This helps make the infra more resilient to outages for this architecture.
  • 2024-11-06 - Changed the load-balancing strategy to have clients “stick” to a web backend. See this commit for details.
  • 2024-10-17 - We have stopped using bos01 and bos02 altogether. All arm64, ppc64el, and s390x tests now run in bos03. This means we are also no longer using s0lp4 in bos02, which was historically problematic.
  • 2024-09-11 - We got 10.255.255.1/32 and 10.255.255.2/32 blocked on the testbed to help rust-reqwest to pass and ease some transitions. Details and concerns in the MP, since this could have side effect. Additional work is ongoing to revert that change later.
  • 2024-08-12 - We recently modified both britney and security britney to not queue tests for i386 source packages which have binary packages which are all Architecture: all. This is because, for actual end users, the dependencies of an arch: all package on non-arch: all packages are satisfied by the amd64 binaries, not i386 binaries. So these i386 source packages with all Architecture: all binaries are redundant because of the amd64 tests for the same package.
  • 2024-07-23 - s390x test runners were added to PS6 (bos03) - as s390x resources are moved to bos03, we will start to use more and more runners in bos03-s390x.
  • 2024-07-03 - ppc64el test runners were added to PS6 (bos03) - as ppc64el resources get moved to bos03, we will start to use more and more runners in bos03-ppc64el.
  • 2024-06-27 - Deployed MP 468265 to fix resource leak on RabbitMQ. This should help make it even more stable.
  • 2024-06-19 - Deployed MP 467833 to fix a producer/consumer issue around RabbitMQ, that made its resource consumption explode regularly under high load. This should help make it more stable.
  • 2024-06-12 - Increased retry limit when launching an instance from 3 to 6 to help cope with arm64 instances being slow to spawn
  • 2024-05-28 - arm64 and armhf test runners were removed from scalingstack (bos01 and bos02)
  • 2024-02-09 - arm64 test runners were added to PS6 (bos03)
  • 2023-05-11 - amd64 test runners moved from scalingstack to PS5 (lcy02)
5

Service Degradation History

  • 2025-02-01 the infrastructure started having issues with s390x testbeds; as of 2025-02-04 this seems resolved.
  • 2025-01-06 - amd64 and i386 are facing an infrastructure issue in PS6 (bos03), which is out of the URM team’s hands. The issue is being tracked here. Edit: things seems to get better, we’re slowly raising the number of workers again.
    • 2025-01-17 - The problem seems to be fixed: amd64/i386 throughput is back to normal. We’ll keep monitoring the status and post an update on Monday.
    • 2025-01-27 - Healthy throughput again after a few days of the issue resurfacing, since the message above. We’re considering the issue solved for now.
  • 2024-10-03 - It was reported that some jobs from 2024-09-27 has been missing results. When requeued, the jobs worked correctly. We have no precise idea on what happened with those jobs, but if that happens again, we’ll investigate deeper. Reported again in December 2024, hopefully fixed now.
  • 2024-09-12 - The following is now fixed thanks to an hypervisor reboot on a newer kernel: Current arm64 and armhf throughput is very slow, because VMs take in average more than 1000s to spawn. This was a long standing issue that had been there since last May.
  • 2024-06-27 - The armhf workers sometimes becoming unavailable since about 2024-05-28 (LP#2067633) has been worked around completely in this MP.
  • 2024-05-25 - All the armhf test runners (arm64 hosts in bos03) were down until Monday
  • 2024-03-26 - All tests outside of armhf from around 2024-03-20 to 2024-03-26 inadvertently used the release pocket only, causing false results (notably false passes). The Noble results have been rewritten to say no-proposed=1 in the env field and become errors. This has not been done for stable releases, which may still have some false passes.
6

New Features

  • 2024-07-29 - There is now a page with the results of a specific test run. The structure of the url for a test result is autopkgtest.ubuntu.com/run/$UUID e.g. a test result.
  • 2024-06-24 - There is now a user page. It has all of your running, queued, and previous test results. It doesn’t have any data regarding ppa tests. It is available at Ubuntu Autopkgtest Results
1 Like