Autopkgtest Service

Quality
1

This post is used to convey information about the autopkgtest service which is managed by the Ubuntu QA team and any issues with that service. Bugs about the service should be reported in Launchpad.

Current Status

  • The s390x queue may be consumed slowly due to an issue with a specific host (s0lp4) in bos02. This can be temporarily resolved by having IS restart neutron services on the host and it has been reported in an RT submitted to Canonical IS. Neutron was last restarted on 2024-07-18. [Updated 2024-07-18]
  • Current arm64 throughput is very slow, because VMs take in average more than 1000s to spawn. An RT has been opened to fix that issue. This shouldn’t affect performance of the VM during the test. This could also affect VM performance (not sure, not always), and this is also affecting armhf tests running in that same datacenter. [Updated 2024-07-19]

If you think you’ve found an issue that is not already documented or for which a bug report has not been created please mention qa-help or ping @bdmurray, @paride, @andersson123, or @hyask in #ubuntu-devel on libera.chat.

8 Likes
4

Major Environment Changes

  • 2024-07-03 - (a few) ppc64el test runners were added to PS6 (bos03) - as ppc64el resources get moved to bos03, we will start to use more and more runners in bos03-ppc64el.
  • 2024-06-27 - Deployed MP 468265 to fix resource leak on RabbitMQ. This should help make it even more stable.
  • 2024-06-19 - Deployed MP 467833 to fix a producer/consumer issue around RabbitMQ, that made its resource consumption explode regularly under high load. This should help make it more stable.
  • 2024-06-12 - Increased retry limit when launching an instance from 3 to 6 to help cope with arm64 instances being slow to spawn
  • 2024-05-28 - arm64 and armhf test runners were removed from scalingstack (bos01 and bos02)
  • 2024-02-09 - arm64 test runners were added to PS6 (bos03)
  • 2023-05-11 - amd64 test runners moved from scalingstack to PS5 (lcy02)
5

Service Degradation History

  • 2024-06-27 - The armhf workers sometimes becoming unavailable since about 2024-05-28 (LP#2067633) has been worked around completely in this MP.
  • 2024-05-25 - All the armhf test runners (arm64 hosts in bos03) were down until Monday
  • 2024-03-26 - All tests outside of armhf from around 2024-03-20 to 2024-03-26 inadvertently used the release pocket only, causing false results (notably false passes). The Noble results have been rewritten to say no-proposed=1 in the env field and become errors. This has not been done for stable releases, which may still have some false passes.
6

New Features

  • 2024-06-24 - There is now a user page. It has all of your running, queued, and previous test results. It doesn’t have any data regarding ppa tests. It is available at Ubuntu Autopkgtest Results
1 Like