Autopkgtest Service

This post is used to convey information about the autopkgtest service which is managed by the Ubuntu QA team and any issues with that service. Bugs about the service should be reported in Launchpad.

Current Status

  • We have recently moved away from bos01 and bos02, and are now just using bos03 for arm64, ppc64el and s390x. If you notice any issues, please contact us.
  • A not fully triaged issue in the lcy02 data center is causing lower throughput in processing amd64 and i386 tests. This is being investigated. This issue will commonly manifest itself as VirtSubproc.Timeout: 1800, with a prior traceback.

If you think you’ve found an issue that is not already documented or for which a bug report has not been created please mention qa-help or ping @paride, @andersson123, or @hyask in #ubuntu-devel on libera.chat.

8 Likes

Major Environment Changes

  • 2024-11-07 - Workers in bos03-amd64 were added to spread the load across both PS5 and PS6. This helps make the infra more resilient to outages for this architecture.
  • 2024-11-06 - Changed the load-balancing strategy to have clients “stick” to a web backend. See this commit for details.
  • 2024-10-17 - We have stopped using bos01 and bos02 altogether. All arm64, ppc64el, and s390x tests now run in bos03. This means we are also no longer using s0lp4 in bos02, which was historically problematic.
  • 2024-09-11 - We got 10.255.255.1/32 and 10.255.255.2/32 blocked on the testbed to help rust-reqwest to pass and ease some transitions. Details and concerns in the MP, since this could have side effect. Additional work is ongoing to revert that change later.
  • 2024-08-12 - We recently modified both britney and security britney to not queue tests for i386 source packages which have binary packages which are all Architecture: all. This is because, for actual end users, the dependencies of an arch: all package on non-arch: all packages are satisfied by the amd64 binaries, not i386 binaries. So these i386 source packages with all Architecture: all binaries are redundant because of the amd64 tests for the same package.
  • 2024-07-23 - s390x test runners were added to PS6 (bos03) - as s390x resources are moved to bos03, we will start to use more and more runners in bos03-s390x.
  • 2024-07-03 - ppc64el test runners were added to PS6 (bos03) - as ppc64el resources get moved to bos03, we will start to use more and more runners in bos03-ppc64el.
  • 2024-06-27 - Deployed MP 468265 to fix resource leak on RabbitMQ. This should help make it even more stable.
  • 2024-06-19 - Deployed MP 467833 to fix a producer/consumer issue around RabbitMQ, that made its resource consumption explode regularly under high load. This should help make it more stable.
  • 2024-06-12 - Increased retry limit when launching an instance from 3 to 6 to help cope with arm64 instances being slow to spawn
  • 2024-05-28 - arm64 and armhf test runners were removed from scalingstack (bos01 and bos02)
  • 2024-02-09 - arm64 test runners were added to PS6 (bos03)
  • 2023-05-11 - amd64 test runners moved from scalingstack to PS5 (lcy02)

Service Degradation History

  • 2024-10-03 - It was reported that some jobs from 2024-09-27 has been missing results. When requeued, the jobs worked correctly. We have no precise idea on what happened with those jobs, but if that happens again, we’ll investigate deeper.
  • 2024-09-12 - The following is now fixed thanks to an hypervisor reboot on a newer kernel: Current arm64 and armhf throughput is very slow, because VMs take in average more than 1000s to spawn. This was a long standing issue that had been there since last May.
  • 2024-06-27 - The armhf workers sometimes becoming unavailable since about 2024-05-28 (LP#2067633) has been worked around completely in this MP.
  • 2024-05-25 - All the armhf test runners (arm64 hosts in bos03) were down until Monday
  • 2024-03-26 - All tests outside of armhf from around 2024-03-20 to 2024-03-26 inadvertently used the release pocket only, causing false results (notably false passes). The Noble results have been rewritten to say no-proposed=1 in the env field and become errors. This has not been done for stable releases, which may still have some false passes.

New Features

  • 2024-07-29 - There is now a page with the results of a specific test run. The structure of the url for a test result is autopkgtest.ubuntu.com/run/$UUID e.g. a test result.
  • 2024-06-24 - There is now a user page. It has all of your running, queued, and previous test results. It doesn’t have any data regarding ppa tests. It is available at Ubuntu Autopkgtest Results
1 Like