Ubuntu QA Team Weekly Status Update

Thursday 30 June 2022

Upgrade tests

  • More issue with the automated upgrade tests: some upgrades of flavors got stuck because of a high priority debconf message in encfs. That’s fixed now in the auto-upgrade-testing-profiles.
  • Spotted another issue with the automated upgrade tests: Focal to Impish profiles are not working as the version of auto-upgrade-testing doesn’t cover the case where a new LTS exists but upgrades to it are not available yet.
    • This is fixed in trunk (https://code.launchpad.net/auto-upgrade-testing), for now manually fixed on venonat as we can’t easily build a new version of the package due to missing dependencies (phablet-tools). Updating the package to drop stale code is planned.
  • Added back the Focal->Impish upgrade profiles: that’s still a valid upgrade path.

ISO testing

  • Remove the .zs-old filed zsync leaves behind after downloading the Server images. Useful in storage constrained environments (e.g. the s390x ISO testing Jenkins node).

autopkgtests

Metrics

  • Reviewed a draft spec on potential metrics to collect around Ubuntu Server (and Ubuntu in general) and on metrics collecting methodologies.
  • Jammy is not not showing up yet in the “LTS adoption” metric. I added it to the Grafana conf, but I still need to adjust a Jenkins job to parse Jammy’s data.
    • Reviewed / Merged a PR adding Jammy to the Jenkins job.

cloud-init

ubuntu-advantge-tools

  • GitHub CI is working fine (see Actions), but several integration tests we run in Jenkins are failing. Those are used for general QA and SRU validation, for which test logs are collected manually as the Jenkins jobs are too noisy. Working at fixing those.

Ubuntu Server

2 Likes

Thursday 30 June 2022

autopkgtest

  • submitted, merged an MP which fixed cache-amqp running when the queues have not yet been created
  • retested a bunch of systemd failure for @enr0n as they seem to have been due to networking issues

Ubuntu

  • sru-release of ubuntu-advantage-tools per request
  • worked on recreating an 802.11ac connection issue so that I could test the wifi regulatory domain change
  • tested some wifi regulatory domain stuff in netplan.io on a Raspberry Pi4 for @slyon
  • prepared, uploaded the SRU of a fix to allow upgrades from Groovy to Impish which fixes a bug that would leave users stranded on an EoL release
  • reviewed, approved @paride’s MP fixing the setup of automatic upgrade testing for some flavors
  • wrote an Error Tracker database query to find the version of packages which appear in the Dependencies of OOPSes collated in the same problem / bucket

Misc

  • administrative work
2 Likes

Thursday 7 July 2022

autopkgtest

  • Investigation into more systemd failures for @enr0n - the majority of the failures are in bos01 and are issues resolving us.ports.ubuntu.com. However, there are also failures building systems.
  • Retried all the systemd crashes that were due to networking issues (–log-regex “ERROR: testbed failure: cannot send to testbed”)
  • pinged IS regarding networking issues in bos01 and real resource limits
  • Redeployed lxd-armhf1 as its /srv filesystem was full and it was time for a Friday redeploy
  • Further debugging of logging into the staging environment of the autopkgtest environment not working, learned a bit about flask.
  • Merged @bdrung’s MP adding onetbb to big_packages for arm64
  • Discovered that all armhf workers were down again this time due to an inablity to write statistics to the InfluxDB. Cowboy’ed a change to the lxd cloud worker to not quit if writing fails and submitted an MP with that change. Also pinged IS about increasing the max-series-per-database setting for the ubuntu-release influx database which they did.

upgrade testing

Ubuntu

  • Special sru-release of freerdp2, gnome-shell-extension-desktop-icons-ng for Ubuntu 22.04
  • Sent an email to ubuntu-devel regarding querying the Error Tracker database for information about crashes
  • Attended the Ubuntu SRU team meeting
  • Tested the contact information missing in live environment bug again and again.
  • Tested juce/arm64, per Bryce’s +1 report, on a large instance in canonistack where it passed so I added it to big_packages and ran the test in production. Then used retry-autopkgtest-regressions to find another package, asmjit, with the same error and added it to big_packages.
  • Checked to see if there was an issue with bash-dbgsym on armhf for Jammy per @bdrung’s email to ubuntu-devel.
  • Sent an email to the ubuntu-devel mailing list regarding Ubuntu 21.10 EoL and unverified SRUs.
  • Special sru-review of python-os-brick due to an IRC ping
  • Conducted SRU verification of the upgrade from Ubuntu 20.10 to Ubuntu 21.10 bug
  • Committed changes to the ubuntu/jammy branch of ubuntu-release-upgrader which add support for upgrading from EoL releases (groovy, hirsute) to jammy
  • Reviewed, merged, and uploaded @edugomez’s ubuntu-release-upgrader change to not try and reboot after an upgrade on WSL
2 Likes

Thursday 7 July 2022

Upgrade tests

  • Landed a MP that makes auto-upgrade-testing to LTS-to-LTS upgrades if possible, instead of always using interim releases as a stepping stone (ref 99).
  • Landed a MP that bumps the autopkgtest-virt-qemu reboot timeout to 300s
  • qa-jenkins-jobs: make the upgrade tests use old-releases.ubuntu.com for EOL releases (5695839)
  • Initial doc on how the auto upgrade tests work.

autopkgtests

  • Review @bdmurray preventing workers to go down if they fail to write metrics data to InfludDB (MP 426312).

ISO testing

  • Changing the ISO download logic for the Ubuntu Server images: monitor .publish_info for changes at hourly intervals and only trigger downloads if there’s a new serial for a given arch.
  • Investigate failure in ISO static validation
    • Turns out there’s an out of date mirror, will ask IS to look into it.

cloud-init

Ubuntu

  • Weekly triage rotation for the Server Team packageset
  • Initial work at implementing a new “ssh login time” metric for Server instances

Infrastructure

  • General maintenance of the Jenkins worker nodes
  • Cleanup and monitoring of the cloud resources allocated for CI and testing of Server projects
2 Likes

Thursday 14 July 2022

autopkgtest

  • Released the autopkgtest-web charm to the stable channel and updated the production version of autopkgtest.ubuntu.com to use the new charm.
  • Investigation into and resolving an issue with mojo run failing in the staging and production version of the autopkgtest environment.
  • Updated the staging version of proposed migration to use the stable version of the autopkgtest-cloud-worker charm. (It hadn’t been published to the edge channel.)
  • Resolved an issue with static content being missing on the production version of autopkgtest.ubuntu.com. It was due to some missing symlinks.
  • Updated the documentation at readthedocs.io.
  • Discovered a number of leftover autopkgtest instances in bos01 that were causing tests to take longer because we couldn’t request new instances. Then discovered the cleanup-instances script which can be used to clean those up and cleaned up instances (after cowboy’ing some changes) in bos01 and bos02.
  • Made some changes to cleanup-instances to handle instances that are in a state which can’t be deleted and to improve logging. Then submitted an MP with those changes.
  • Investigation into why pyside2 tests take so much longer in Ubuntu than Debian.
  • Resolved an issue, two times, with an lxd armhf runner being down.
  • Investigated why liburing autopkgtests were running for so long after being pinged about it. Added liburing to never_run and reported a bug regarding my findings.
  • Reduced the number of s390x test runners in bos01 due to failures to launch new instances and added more in bos02.
  • Discovered that there was another issue with the cloud-worker-maintenance script and failing to write to influxdb, cowboy’ed a fix for that and ran the maintenance script which ended up cleaning up more old units.

Ubuntu

  • Sent an email to ubuntu-release about openjdk-lts and whether or not we should keep running its tests which take forever and are neutral. (Yes we should!)
  • Removed python-ase on armhf from long_tests because it fails there even if given more time.
  • Cleaned up some never_run entries from autopkgtest-package-configs.
  • Added pyside2/(amd64|arm64) to big_packages because it passed it canonistack.
  • Tested an upgrade from Ubuntu 20.04 to Ubuntu 22.04 and ended up reporting a bug regarding snaps being upgraded and a bug about a conf file prompt.
  • Special sru-review of gstreamer-vaapi, sru-release of opencryptoki.

Misc

  • Posted on the charmhub discourse about having the canonical-foundations-autopkgtest-web charm removed.
2 Likes

Thursday 14 July 2022

Upgrade testing

  • More tweaking autopkgtest-virt-qemu timeouts. Many of those are not configurable by the user (so I modified them in source), but we’ll have to upstream the changes somehow, most likely by adding a command line flag.
  • Documend how the auto upgrade testing works
    • General overview
    • How auto-upgrade-testing implements it
    • How we run it in Jenkins
  • Migrated https://launchpad.net/auto-upgrade-testing to git:
    • Reviewed and rejected/merged all the pending MPs
    • Performed the migration preserving history (bzr fast-export | git fast-import)
    • Updated the project settings to point to the new repository location (https://code.launchpad.net/auto-upgrade-testing)
    • Updated the Jenkins jobs to pull with git and not bzr
    • Removed the (already broken and not used) recipes
    • Added REPOSITORY_MOVED_TO_GIT flag file to the Bazaar repository (rev 103)
  • Migrated https://launchpad.net/auto-upgrade-testing-specifications to git:
    • Performed the migration preserving history (bzr fast-export | git fast-import)
    • Updated the project settings to point to the new repository location
    • Updated the Jenkins jobs to pull with git and not bzr
    • Added REPOSITORY_MOVED_TO_GIT flag file to the Bazaar repository (rev 103)
  • Announce that auto upgrade tests are maintained again
  • Backport src:reserialize to Bionic and upload it to the QA JenkaaS PPA. Planned use is parsing the upgrade profiles to extract release information in scripts.
  • Use a single Jenkins agent on venonat for the ISO tests and upgrade tests
  • Tested out /tmp on tmpfs on the upgrade testing Jenkins node. Experiment failed:
    • autopkgtest-virt-qemu already uses tricks to speed up the i/o operations (qemu storage with cache=unsafe, and unlinking the overlay image while keeping the file descriptor open, which makes the kernel disable data-to-disk sync calls).
    • The upgrade tests need more space than I thought (not immediate to see because of the “unlink” trick mentioned above).
  • Initial work in bringing the auto-upgrade-testing package back into shape.

ISO testing

  • Pin the cdimage.ubuntu.com IP to workaround the fact that the UK and US mirrors are occasionally out of sync. The root issue is being handled by IS.
  • General maintenance of the ISO testing Jenkins nodes.

pycloudlib

  • Add support for images with serial with YYYYMMDD.n format (GitHub PR)

ubuntu-advantage-tools

  • Pre-SRU review of the Kinetic MP, see MP 426755.

Ubuntu Server

Bootspeed measurements

  • Make it use the latest pycloudlib to support YYYYMMDD.n serials. All jobs are passing now.

Edit: fixed report date, thanks @guiverc

1 Like

Thursday 21 July 2022

autopkgtest

  • Resolved an issue with 2 lxd-armhf runners being down - they were paused and needed rebooting.
  • Worked with IS to resolve networking issues in bos01, then created more recent images for releases and arches which had items in the queue. Then added more workers to bos01.
  • Submitted, then merged an MP regarding cleanup-instances failing due to an inability to write to InfluxDB. [It had already been cowboy’ed.]
  • Updated autopkgtest-cloud-configs to indicate that big_packages get a longer --timeout-test.
  • Investigation into the failure to create a new lxd armhf runner in bos02. It is failing due to a lack of free space in the root filesyst
    em. This was sorted out by @paride.
  • Resolving an issue with the s390x queue not moving. It was due to an issue running xenial/s390x tests for a private PPA so I cowboy’ed a change to temporarily skip running those tests.
  • Added hilive/arm64 to big_packages because it dumps core, added nwchem/(amd64|arm64) to big_packages due to crashes.
  • Resolved an issue with lxd-armhf2 being down - it had experienced a kernel crash so I rebooted it.

Ubuntu

  • Uploaded ubuntu-release-upgrader for kinetic fixing a bug regarding seeded snaps.
  • Uploaded ubuntu-release-upgrader for jammy fixing bug, bug, and [bug]. Added SRU information to the bug reports.
  • Retried the failed ubuntu-release-upgrader autopkgtests until they ran in bos02 and passed.
  • Commented on the SRU of a calamares for jammy for flavors.
  • Discussed the ddeb-retriever service with seb128.
  • Kicked off a canary desktop image build for seb128.
  • Special sru-review of s390-tools.
  • Sponsored patches fixing a bug with zlib on IBM z15 hardware.
  • With members of the ubuntu-release team worked on the EoL process for Impish Indri.
2 Likes

Thursday 21 July 2022

Short week due to vacation.

Ubuntu

Impish EOL release team tasks

autopkgtest

ISO testing and upgrade testing

  • migrate auto-upgrade-tests to GitHub
    • Created the ubuntu-qa team under the canonical GitHub organization
    • Created canonical/auto-upgrade-testing repository with me and @bdmurray as admins, and the ubuntu-qa team having access (for future team members).
  • Redeploy the arm64 Jenkins node used for ISO testing and auto upgrade testing
    • Thanks @axino for bringing the arm64 node back online
  • Scrub of the Jammy ISO testing jobs as the Jammy point release is getting near
    • All green, but we don’t have arm64 results yet
  • Point release coordination meeting w/ extended Release Team

cloud-init

  • Regenerated COPR api keys and updated them on the Jenkins nodes
2 Likes

Thursday 28 July 2022

autopkgtest

  • Disabled all test runners in bos01 as there are issues with the network configuration.
  • Discovered that all my local autopkgtest images were using qcow2 compat level of 0.10 which is really old. Confirmed with @cpaelzer that it is really old and there are many benefits to use compat=1.1. This resulted in submitting an MP to livecd-rootfs to change the compat format we are using for image building which then autopkgtest-buildvm-ubuntu-cloud would benefit from.
  • Discovered why new lxd-armhf runners don’t have xenial images and found out where to fix it but my MP is insufficient.
  • Added bbmap to big_packages because it “Could not create the Java Virtual Machine” and it passes it Debian.
  • Submitted an MP to ubuntu-archive-tools adding a new flag to retry-autopkgtest-regressions for retrying packages which are not migrating due to failing tests of a different package (–blocked-by-tests).
  • Reenabled all the test runners in bos01 as the proxy issues there are resolved now.
  • Deleted all impish test results as the release is EoL now.

Error Tracker

  • Updated daisy to stop accepting crashes from Ubuntu 21.10 (Impish Indri) as it is EoL now. Deployed the change to staging and production.

ISO Tracker

  • Merged test cases from @alexghiti for the Nezha and VisionFive boards. Then added test suites etc for the products so that the test cases would show up in the ISO Tracker.

Ubuntu

  • Uploaded python-apt with an updated mirror list for the 22.04.1 point release.
  • Performed the Stable Release Updates verification of ubuntu-release-upgrader fixing bug 1975533 and bug 1981485 for the 22.04.1 point release.
  • Performed the Stable Release Updates verification of speech-dispatcher fixing bug 1981611.
  • Sent an email to ubuntu-devel regarding the point release process of checking for halted phased updates.
  • Investigation into a bug regarding the inability to use ubuntu-bug for snaps on the Live “CD” environment.
  • As a result of the above worked with @dbungert to get the subiquity snap updated so that it has contact information embedded in it. Verified that this worked with an updated kinetic amd64 live server image dated 20220727.1.
  • Merged an old ubuntu-archive-tools branch that had been awaiting review for some time.
  • Learning about and babysitting the ddeb-retriever as the Error Tracker has been having issues communicating with ddebs.ubuntu.com.
2 Likes

Thursday 04 August 2022

autopkgtest

  • Messed around with worker configuration so that impish autopkgtests in the queue would all get rejected and removed from the queue.
  • Updated the service-bundle in git to match changes that have been made on wendigo.

Error Tracker

  • Investigated how to have the Error Tracker workaround ddebs.ubuntu.com not being available and modified apport to ignore errors when it is unable to connect ddebs.u.c. This works because apport will fallback to downloading the ddebs from Launchpad directly. Tested the change in the staging version of the Error Tracker.
  • Submitted an RT to get the apport code updated in the production version of the Error Tracker. * Worked with IS to really get the above code change landed in production as there were some issues.

ISO Tracker

  • Added WSL product to the Ubuntu 22.04.1 point release in the ISO tracker.
  • Updated some manual test cases in the ISO tracker which had been updated in the git repository but not on the site itself.
  • Updated download links for Ubuntu Mate for Ubuntu 22.04.1 point release.
  • Performed Ubuntu Desktop testing for the 22.04.1 point release.

Ubuntu

  • Tested a bug regarding failure to unlock encrypted after upgrading to Ubuntu 22.04 LTS.
  • Merged my MP to ubuntu-archive-tools adding a --blocked-by-tests argument.
  • Reported a bug regarding the confusing --blocked-by argument of retry-autopkgtest-regressions.
  • Did some additional testing of release upgrades from EoL releases and discovered that bug is an issue that people are encountering.
  • Review of an MP regarding reinstalling nvidia meta package for people upgrading from early installs of 20.04. Created a follow on MP for the same issue.
  • Overrode some ubuntu-release-upgrader false positives which caused its Phased-Update-Percentage to be stopped.
  • Reviewed an MP which adds a quirk to stop an upgrade if a specific PAM configuration exists.
  • Discussed zlib being stuck in -proposed with @fheimes.
  • Participated in the investigation of a bug regarding OEM installs and seeded snaps.

Misc

  • Had a couple of meetings with some members of the Ubuntu Release team.
1 Like

Thursday 11 August 2022

autopkgtest

  • Redeployed lxd-armhf11, twice! once w/ trusty (see below) and once without, as it was time for a Friday redeploy.
  • Merged @paride’s branch to autopkgtest-cloud to also create xenial (and trusty) lxd images for armhf. However, there aren’t any trusty images available any more so the changes didn’t work out.
  • Created and merged my branch which prevents the scripts from trying to create trusty images.
  • Created an MP fixing a Traceback when running the autopkgtest-cloud-worker charm.
  • Rebooted lxd-armhf2 as it was down and not running any autopkgtests.
  • Restarted multiple (10) lxd-armhf autopkgtest services as they had died sometime on 2022-08-08.
  • Submitted an RT regarding bumping max-values-per-tag in the ubuntu-release influx database.
  • @xnox alerted autopkgtest maintainers to an issue with qemu-efi-noacpi trigger no longer being available and @juliank discovered that some cowboy’ed changes regarding the trigger had been overridden when the mojo service bundle was updated to use the new charmstore URL format. Since the autopkgtest-cloud-worker charm is not buildable yet (still on the Ubuntu QA team backlog) I redeployed the changes on the production servers.
  • Reran the acpi-call autopkgtest which was triggered by linux-meta-raspi and qemu-efi-noacpi and it passed.
  • Reported a bug regarding a Traceback in autopkgtest-virt-lxd which I discovered when restarting a downed lxd-armhf worker service.

Error Tracker

  • Reported a bug regarding the retracing process ack’ing every message in the retracing queue because of a swift misconfiguration and thereby draining the queue. (The crashes are still in swift though.)

ISO Tracker

  • Updated download links for Ubuntu Studio for Ubuntu 22.04.1 point release.
  • Performed Ubuntu Desktop testing for the 22.04.1 point release.

Ubuntu

  • Retriggered gdcm on kinetic/s390x with the version of zlib from -proposed for @fheimes.
  • Wrote a bug pattern for a bug regarding /boot running out of free space due to its use of zsys.
  • Overrode some ubuntu-release-upgrader false positives which caused its Phased-Update-Percentage to be stopped.
  • ISO testing for the 22.04.1 point release including testing of the OEM installs and seeded snaps bug.
  • Reported a crash report, discovered after installing Ubuntu, to the Ubuntu Error Tracker which is a duplicate of a Launchpad bug regarding seahorse and avahi. Tagged the Launchpad bug rls-jj-incoming.
  • Discovered a console message “mtd device must be supplied” when booting a fresh install of 22.04.1. Later realized this is a bug on the Foundations team radar.

Misc

  • Met with @sil2100 regarding ESM autopkgtest’ing and some other things.
  • Attended Ubuntu Developer Membership Board meeting.
  • Submitted an RT about not getting confirmation emails when submitting RTs to the ubuntu-qa queue.
2 Likes

Thursday 25 August 2022

autopkgtest

  • Restarted all the lxd armhf workers as they had gone down due to an issue with apt-source and a systemd package from a PPA.
  • The systemd tests from a PPA came back and caused the workers to fail again so this time I added a patch to the worker code to just drop the tests so we can keep churning through the armhf queue. The tests then became an issue for other arches and I had patch the worker code and kill the tests until the new worker code was picked up.
  • In the staging environment experimented with a “bad version” of systemd, thanks @enron, which would cause the worker code to continually
    run the package’s tests. Come to find out that DNS resolution is broken with that version of the systemd and the tests are unable to execut
    e ‘apt-get source’ of systemd. There is existing code to handle this type of situation and the string matching just needed to updated somet
    hing for which I created an MP.
  • Updated the autopkgtest-cloud-worker charm to revision 135 in the edge channel, then tested the updated charm in staging.
  • Released autopkgtest-cloud-worker charm revison 135 to stable and deployed it to production.
  • Reported bug 1987012 regarding a test submission request with a duplicate trigger.
  • Redeployed lxd-armhf9 - an armhf worker which had been getting long in the teeth.
  • Investigation into issues building adt images for s390x in bos01.

Error Tracker

  • Updated RT 152615 with a request to requeue more core files.

Ubuntu

  • Modified unsubscribe-team, which is a part of ubuntu-archive-tools, to have a --dry-run mode and a protected set of packages for teams. Submitted an MP with those changes. Additionally, notified the desktop team about packages which they can unsubscribed from.
  • Merged my MP from above.
  • Added a patch to bug 1985253 which is about a failure to call lspci when running a dist-upgrade quirk.
  • Special sru-review of qclib for Jammy.
  • Uploaded ubuntu-release-upgrader with mirror updates for the 20.04.5 point release.
  • Uploaded ubuntu-release-upgrader for Kinetic: changing Prompt=normal, with a fix (workaround?) for bug 1969786.
  • Reviewed a couple of britney2 merge proposals for @vorlon.
  • Reported ubuntu-release-upgrader bug 1987452 regarding a failure to upgrade from Focal or Groovy without the latest version of apt. It’s something I’d seen in the Error Tracker and with the help of @juliank was able to recreate.
  • Created a fix for the above bug and prepared the bug report for SRU’ing.
  • Uploaded an SRU of ubuntu-release-upgrader fixing bug 1987452 and bug 1985253.

Misc

  • Take home test reviews.
  • Attended Ubuntu DMB meeting.
2 Likes

Friday 26 August 2022

Server Team

Autopkgtest

Release Team / ISO testing

Misc

  • General catch-up after holiday
  • Take home test reviews
2 Likes

Thursday 01 September 2022

autopkgtest

  • Restarted the cloud-worker-maintenance service on the lxd autopkgtest coordinator as it had hung.
  • Did some further testing (to get some more detailed log files) of autopkgtest with the systemd version from the systemd-broken PPA.
  • Added 3 more lxd-armhf workers to help with the armhf autopkgtest backlog.
  • Reported bug 1988080 regarding the cloud-worker-maintenance service hanging on autopkgtest-lxd-worker.
  • Reported bug 1988223 about log files not containing strings which appear in FAIL_PKG_STRINGS any more.
  • Created an MP adding ppa.launchpadcontent.net to TEMPORARY_TEST_FAIL_STRINGS

Ubuntu

  • Reviewed and merged an MP from @Paride adding a Feature Freeze template email.
  • Pinged an AA to fully phase the updates of python-apt and ubuntu-release-upgrader for Focal and the 20.04.5 point release.
  • Pinged an SRU team member, who is also an AA, to release and fully phase base-files for the 20.04.5 point release.
  • Modified the Focal daily builds to no longer use -proposed and kicked off a daily build of the desktop image.
  • Released translation updates for the 20.04.5 point release.
  • Setup GPU passthrough for an NVIDIA graphics card such that I can do image testing with nvidia hardware in qemu.
  • ISO testing of Ubuntu 20.04.5 point release.
  • Reported bug 1988209 regarding nvidia-settings not displaying anything in 22.04.5.
  • Tested an install of the Kinetic daily image and encountered a crash which I reported as bug 1988228. Although I had to hack apport because there is a bug pattern for bug 1766945 which is not the bug I wanted to report.
  • Removed the bug pattern for bug 1766945.
  • Reported bug 1988332 regarding the desktop environment failing to start after installing a canidate image for 20.04.5 with the nouveau driver.
  • Created a ubiquity MP that allows the unhiding of the secure boot password.
  • Did a test round of Foundations team bug triage and took some notes on it.
2 Likes

Thursday 08 September 2022

autopkgtest

  • Investigation into issues with bos02 s390x test runners, pinged IS about networking issues (can’t ssh after launching an instance) there.
  • Submitted RT 153052 regarding the above networking issues with s390x in bos02. Those issues were resolved by restarting the nova / neutron services.
  • Merged an MP which adds multiple linux-hwe-* packages to long_tests for armhf.
  • Retried some zlib autopkgtest failures for Focal given the above change.
  • Modified permissions for autopkgtest-package-configs so that canonical-ubuntu-qa can push changes to it.
  • Pinged IS regarding making the changes in RT 152719, no SNAT-ing in bos01, then tested that the changes worked. Also notified the developers on +1 regarding the change.
  • Investigated the current status of [RT 151302] regarding failures to allocate networking in lgw01.
  • Investigation into systemd-udevd and a kernel oops w/ a hung test.

ISO Tracker

  • Replied to a Lubuntu team email about updating the test cases at the ISO tracker.

Ubuntu

  • Merged @waveform’s changes to add 20.04.5 information to the raspi imaging file and updated meta-release files for the 20.04.5 point release. Then updated the changelogs.ubuntu.com server with those changes.
  • Reported bug 1988465 regarding the inability to install linux-modules-nvidia-470-5.15.0-27-generic 5.15.0-27.28 on kinetic.
  • Sent an email to the server team about the ownership of the autofs package, it is currently subscribed to by foundations-bugs and server-bugs which isn’t ideal.
  • Reviewed @paride’s MP updating the packaging of auto-upgrade-testing.
  • Pinged an AA about fully phasing tzdata because it wouldn’t phase in time for the time change.
  • Fixed up 2022c tzdata updates for the ESM archive.
  • Updated meta-release-lts-proposed to point to -proposed for Jammy distribution upgrades.
  • Verified the SRU fix for bug 1985253 regarding the NVIDIA dist-upgrade quirk and lspci not being available.
  • Reviewed and approved @enron’s ubuntu-bugcontrol application.

Misc

  • Attended Ubuntu SRU team meeting.
  • Take home test review.
  • Conducted an interview.
  • Investigated wildcard.qa.ubuntu.com for IS (RT 102178)
1 Like

Thursday 08 September 2022

Ubuntu Server

Autopkgtest

Upgrade testing

  • Monitoring of the automated upgrade test jobs
    • Mostly passing, with occasional timeouts
  • General overhaul of the auto-upgrade-testing packaging
  • Added CI to the auto-upgrade-testing GitHub repository via GitHub Actions
    • Currently only running basic linting. More testing to be added, but CI itself is in place
  • Turned auto-upgrade-testing into a native package for easier maintenance
  • Uploaded auto-upgrade-testing to Kinetic after warning the archive admins about this.
    • Package currently waiting in NEW.

ISO testing

  • Usual maintenance and monitoring
  • Kinetic ISO testing builds are currently failing due to a systmed bug. Replied to bugs about this.

Misc

  • Reviewed some take home tests
  • Discussed OpenQA with certification team
2 Likes

Thursday 15 September 2022

autopkgtest

  • Committed a workaround for the issues with systemd-udevd by increasing the period which we’ll wait for the testbed to respond to ssh.
  • Running of cleanup-instances on the autopkgtest-cloud-worker units as the systemd-udevd kernel oops issue is leaving stray instances around.
  • Investigated what is causing the amd64 instances to boot slowly and causing the autopkgtest backlog. It seems to be an issue with the kernel as downgrading systemd did not resolve the issue. I reported bug 1989258 regarding the slow tasks timeout.
  • Modified the instance creation script so that the virtio_rng module is blacklisted allowing the instances to boot much more qucikly.
  • Tested an updated kernel for the kernel team which seems to resolve the issue.
  • Sent a PSA to ubuntu-devel regarding the issues with the autopkgtest environment.

Ubuntu

  • Added a force-badtest hint for nrepl-clojure as it dropped some arches that previously had autopkgtest results which had passed.
  • Declined a Launchpad invitation for ~ubuntu-core-dev to join ~ubuntucinnamon-dev and ~ubuntu-unity-devs (it should be ~motu per @vorlon). Accepted an invitation for ~motu to join ~farsi-fonts team.
  • Performed the SRU verification for an ubuntu-release-upgrader fix for bug 1987452 which is about a failure to upgrade from 20.04 or 20.10 if the latest version of apt is not installed.
  • Foundations team bug triage rotation x 2, once with @jawn-smith.
  • Let the Ubuntu Studio lead know about the possibility of adding a dist-upgrade quirk to ubuntu-release-upgrader to prevent upgrades from 20.04 to 22.04.
  • Submitted a pull request for ubuntu-server-triage changing who is considered a member of a team for TIL purposes.
  • Built a ubuntu-unity daily image per the request of the Ubuntu Unity lead.
  • Tested auto-upgrade-testing for @Paride and discovered a missing dependency so updated upstream with that missing dep.

Misc

  • Conducted an interview.
2 Likes

Thursday 22 September 2022

autopkgtest

  • Sorted out how to ssh to a test instance in lgw01 and then granted some kernel team members access to a system to experiment with the virtio_rng module.
  • Discussed with the CPC team the possibility of having minimal lxd images for armhf.
  • Merged my MP regarding blacklisting virtio_rng in Kinetic amd64 autopkgtest images.
  • Merged my MP regarding dropping the nohz change.
  • Worked with @Paride to diagnose and resolve an issue with all the armhf workers going AWOL. Additionally, documented the incident and bugs discovered for future reference.
  • Committed @Paride’s cowboy’ed changes switching container providers from images: to ubuntu: .
  • Updated 5 armhf remote workers so that they have the latest version of autopkgtest-build-lxd, new cronjobs for creating updated armhf containers, and manually created new kinetic armhf containers.
  • Reported bug 1990319 regarding the autopkgtest code not getting updated on armhf remote workers.
  • Submitted an MP which allows us to build trusty lxd armhf containers again.

Ubuntu

  • Helped @enr0n get started on his Foundations bug triage day.
  • Reviewed and replied to multiple Foundations team triage reports.
  • Created a foundations-todo report on cranberry.
  • Performed an install test of a daily build of Kinetic and report bug 1990535 regarding nvidia-settings not being available after rebooting.

Misc

  • Reviews of candidate essays.
  • Short week due to illness.
2 Likes

Friday 23 September 2022

autopkgtest

ISO testing / Upgrade testing

  • Fixed failing Ubuntu Desktop ISO testing jobs that prevented that Desktop daily ISOs from being proposed from -pending to -current
    • Underlying issue: test disk image not big enough, with consequent full partition and failures. Bumped the test disk image to 12GB (still less then the recommended minimum requisite)
  • General cleanup of venonat, with removal of lingering libvirt instances from failing ISO tests
  • Upgrade testing: uploaded auto-upgrade-testing 0.3.1 to Kinetic (bugfix upload)
  • Brought up (already known) issue with the build process of Desktop daily images (kinetic)

Server Team

  • Weekly triage rotation for the Server packageset
  • Cleanup of stale Azure cloud resources (subnets) that prevented UA and cloud-init tests from passing correnctly
    • Removal done manually, but leaky integration testing jobs have to be fixed
  • General maintenance of the Jenkins nodes
    • Fixed /usr/local/bin/sbuild-update-all exit with status 100 issue of the nodes. Reason: stale Impish schroot requiring manual removal.
  • Continued work on the “ssh login time” metric

Ubuntu Pro

  • Reviewed and sponsored uploads for ubuntu-advantage-tools 27.11 for the devel release and SRUs
  • Reviewed SRU uploads for update-notifier introducing the new Ubuntu Pro naming. Will upload on Monday.

Misc

1 Like

Friday 30 September 2022

autopkgtest

ISO testing and upgrade testing

  • ISO and upgrade tests for the Kinetic beta images
  • Published discourse post on how to use auto-upgrade-testing for upgrade
  • Marked Ubuntu Server beta images as ready after final round of tests

Ubuntu Server

Ubuntu Pro / ESM

  • Reviewed, merged and uploaded change to update-notifier for all the stable releases that updates ESM service name and description.
    • That is: uploaded update-notifier 3.168.16 (xenial), 3.192.1.13 (bionic), 3.192.30.12 (focal), 3.192.54.1 (jammy)
    • Issue: the changes in update-notifier 3.192.30.12 (focal) weren’t pushed to the packaging repository (this is a native package with per-release branches). Reconciled w.r.t the archive by cherry-picking from git-ubuntu import.
    • Issue: turns out there were more inconsistencies between the source package in the archive for Bionic and Focal and the upstream repository (= packaging repository, as the package is native). Thanks @lamoura and @ahasenack for sorting that out and reuploading.

Misc

  • Reviewed several candidate essays
3 Likes