Infrastructure outage

Announcements
1

We are experiencing an infrastructure outage in one of our data centers. The vast majority of Ubuntu users should not be impacted by the outage.

If you are an Ubuntu developer, you will notice some of the services you depend on are down. The debuginfod service is also currently unavailable. Our team is working to restore the impacted services.

We apologize for the inconvenience - updates will be posted here.

16 Likes
3

Brief Update: Canonical is working hard to restore services. We are aiming to have key services starting to come back online by 2026-07-17 afternoon UTC. We will update as services come back online.

6 Likes
4

Brief update: Services are gradually coming back online. Autopkgtest is nearly back. RISCV will be missing a while longer. We’re currently working on restoring connections between proposed-migration (britney) and autopkgtest to re-enable package migration.

Ingress is still being worked on, so public access to reports and sites will be spotty. We’re seeing static sites popping back up regularly as the IS team works through all the environments. pages like manpages.ubuntu.com and static-reports.ubuntu.com are coming back online but users may experience intermittent issues and slowness as we fully recover.

Other major services, such as debuginfod and the git-ubuntu importer are still being worked on.

7 Likes
5

autopkgtest recovered 2026-07-17 (not long after my last update) and has been processing tests over the weekend. The team will be combing through migration reports to see what needs to be re-queued.

we are still working on recovering git-ubuntu importer, debuginfod, langpacks, the transition tracker, and retracers.

5 Likes
6

update on recovering all Ubuntu services from the outage

  • autopkgtest is operational
    • riscv tests are currently disabled and set to non-blocking. riscv requires a custom built vm, and we are having issues recreating these vms
    • we have identified networking performance issues leading to very slow tests runs. This is why the queue is continuing to grow – it’s not that tests are not running, but they are running slower
  • debuginfod is not back online
    • we are working on restoring environments. unfortunately the local disk was lost, and restoring the multiple terabytes will take time. we’re still in the provisioning stage
  • git-ubuntu importer is back and operational.
  • transitions, merges, errors, and other services are back online. There were issues over the past 24 hours with certificates (so if you got an error message about HSTS and mismatched certs, sorry), but all these should be resolved.
8 Likes