APT server issues on the 22nd of every month?

kubacki · October 22, 2025, 10:55pm

I had this problem September 22nd, and now it is here October 22nd and I’m seeing the same problem again: not all mirrors have all packages? These errors are impactful because cloud-init is failing for me, and requiring intervention for every deployment on our platform to fix the server before it can be used.

For example, I get 404’s like this:

E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/bind9-host_9.18.39-0ubuntu0.24.04.2_amd64.deb 404 Not Found [IP: 91.189.91.83 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/bind9-dnsutils_9.18.39-0ubuntu0.24.04.2_amd64.deb 404 Not Found [IP: 91.189.91.83 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/bind9-libs_9.18.39-0ubuntu0.24.04.2_amd64.deb 404 Not Found [IP: 91.189.91.83 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/universe/b/bind9/dnsutils_9.18.39-0ubuntu0.24.04.2_all.deb 404 Not Found [IP: 91.189.91.83 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

I checked all the servers returned for this DNS record and found several are missing many:

$ nslookup security.ubuntu.com
Server:		192.168.86.1
Address:	192.168.86.1#53

Non-authoritative answer:
Name:	c
Address: 185.125.190.81
Name:	security.ubuntu.com
Address: 91.189.91.81
Name:	security.ubuntu.com
Address: 185.125.190.36
Name:	security.ubuntu.com
Address: 91.189.91.82
Name:	security.ubuntu.com
Address: 185.125.190.39
Name:	security.ubuntu.com
Address: 185.125.190.83
Name:	security.ubuntu.com
Address: 185.125.190.82
Name:	security.ubuntu.com
Address: 91.189.91.83

Three of eight are missing this package:

http://185.125.190.81/ubuntu/pool/main/b/bind9/bind9-host_9.18.39-0ubuntu0.24.04.2_amd64.deb → 200
http://91.189.91.81/ubuntu/pool/main/b/bind9/bind9-host_9.18.39-0ubuntu0.24.04.2_amd64.deb → 200
http://185.125.190.36/ubuntu/pool/main/b/bind9/bind9-host_9.18.39-0ubuntu0.24.04.2_amd64.deb → 404
http://91.189.91.82/ubuntu/pool/main/b/bind9/bind9-host_9.18.39-0ubuntu0.24.04.2_amd64.deb → 200
http://185.125.190.39/ubuntu/pool/main/b/bind9/bind9-host_9.18.39-0ubuntu0.24.04.2_amd64.deb → 404
http://185.125.190.83/ubuntu/pool/main/b/bind9/bind9-host_9.18.39-0ubuntu0.24.04.2_amd64.deb → 200
http://185.125.190.82/ubuntu/pool/main/b/bind9/bind9-host_9.18.39-0ubuntu0.24.04.2_amd64.deb → 200
http://91.189.91.83/ubuntu/pool/main/b/bind9/bind9-host_9.18.39-0ubuntu0.24.04.2_amd64.deb → 404

This feels like the "stop asking about it" sticky from Sep 9th, but i haven’t seen anything in the status page for security.ubuntu.com from Sept 22nd or now…so I’m asking what on earth is going on?

kubacki · October 22, 2025, 10:56pm

This has been impacting us for a few hours, now.

No proxy or anything, confirmed from several locations:

eeickmeyer · October 22, 2025, 11:00pm

This tells me your package list is out of date.

Run sudo apt update then proceed.

kubacki · October 22, 2025, 11:15pm

How are you reaching that conclusion? I would disagree, given that it’s trying to download the package bind9 with version 9.18.39-0ubuntu0.24.04.2, which was released in the last 24 hours?

https://launchpad.net/ubuntu/+source/bind9/1:9.18.39-0ubuntu0.24.04.2

My conclusion is that because it’s downloading a new version, my package list IS up to date?

Also do the logs not show it updating prior to download?

eeickmeyer · October 22, 2025, 11:17pm

Does it hurt to try? I’m just going based on experience and that I have had that happen before.

And for what it’s worth, I got it to work but only after I had run sudo apt update.

There’s no reason to be rude.

My rationale is that you have 404s, which means “file not found”. Last month the issue was 503s, which are a server error.

400 errors: error is on your end.
500 errors: error is on their end.

kubacki · October 22, 2025, 11:19pm

Additionally, it’s easy to verify that when accessing these servers directly by IP address, they are missing data? LIke the entire b folder, where is it?

eeickmeyer · October 22, 2025, 11:20pm

Why won’t you just run sudo apt update to be sure?

kubacki · October 22, 2025, 11:26pm

I did run it, and my cloud-init script runs it immediately before, eg apt-get update && apt-get upgrade -y. The problem is that 3 of the 8 servers that security.ubuntu.com resolve to don’t have the package, so about 40% of the time it fails.

eeickmeyer · October 22, 2025, 11:27pm

Well, regardless, you’re barking up the wrong tree here because this is community-level support and there is nothing anybody here can do about it.

Sounds like you need a Service Level Agreement from Canonical. To purchase that, visit Enterprise open source support | Ubuntu. You will not get the level of support here that you need.

kubacki · October 22, 2025, 11:30pm

Someone may know what’s going on, though? That’s why I am asking here, in case anyone know if there any visibility into the package release or server/mirror update process? It feels like too weird of a coincidence to happen on the same day of the month two months in a row for nobody in the community to have some insight.

eeickmeyer · October 22, 2025, 11:31pm

It takes time for all of the mirrors of security.ubuntu.com to get the update, so… be patient? It was only released today.

I had no issue getting the update for any of my machines, so it could even be geographic.

Regardless, your tone here is pretty offputting for anyone wanting to help you.

kubacki · October 22, 2025, 11:51pm

If i sound frustrated and abrasive, I apologize, but we’re talking past each other here. I get that a lot of people don’t know they need to update their package list, but I thought it was clear I was sharing evidence in the original post that i’ve done enough research to narrow this down to a server-side issue.

I had no issue getting the update for any of my machines, so it could even be geographic.

I see failures from us-west-1 and us-west-2. I’m personally in the mountain west and when i visit these URLs, only the .81 server works. Do both work for you?

http://185.125.190.81/ubuntu/pool/main/b/bind9
http://185.125.190.36/ubuntu/pool/main/b/bind9

I’m not sure how to tell if anycast is used or something, but a quick check of https://www.whatsmydns.net/#A/security.ubuntu.com shows that everyone should be advertised the same service IP regardless of geophraphical location.

eeickmeyer · October 22, 2025, 11:58pm

First one works for me, second one 404s.

$ dig security.ubuntu.com

; <<>> DiG 9.20.11-1ubuntu2.1-Ubuntu <<>> security.ubuntu.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10819
;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;security.ubuntu.com.           IN      A

;; ANSWER SECTION:
security.ubuntu.com.    60      IN      A       91.189.91.81
security.ubuntu.com.    60      IN      A       185.125.190.36
security.ubuntu.com.    60      IN      A       91.189.91.83
security.ubuntu.com.    60      IN      A       185.125.190.81
security.ubuntu.com.    60      IN      A       185.125.190.82
security.ubuntu.com.    60      IN      A       185.125.190.83
security.ubuntu.com.    60      IN      A       91.189.91.82
security.ubuntu.com.    60      IN      A       185.125.190.39

;; Query time: 91 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Wed Oct 22 16:53:26 PDT 2025
;; MSG SIZE  rcvd: 176

You can always try editing your /etc/apt/sources.list.d/ubuntu.sources to replace security.ubuntu.com with one of the working URLs if you simply cannot wait for the mirrors to propagate.

As far as I know, Canonical has been going through an infrastructure move, so things are propogating slower than you might be used to.

kubacki · October 23, 2025, 12:08am

Thanks for confirming, I did check ap-south-1 and eu-central-1 and saw the exact same results.

You can always try editing your /etc/apt/sources.list.d/ubuntu.sources to replace security.ubuntu.com with one of the working URLs if you simply cannot wait for the mirrors to propagate.

Yeah, that works, but is harder to get through change approval process, then build and test an updated AMI while i’m getting paged every 5-10 minutes when a new server boots and resolves one of the “bad” IPs.

I think i’ll probably set up a DNS record in our domain to point to these servers (and confirm we’re using plain http mode for apt). That way i only have to change the sources list once, and can just update the record to point at a working set of servers if we have issues again.

kubacki · October 23, 2025, 3:27am

For posterity, I updated the ec2 user data (cloud-init directives) we use to launch instances to specify an alternate security repository URI to be used. In my case I had to provide the whole section, even though I only want to update the security section:

## template: jinja
#cloud-config
system_info:
  package_mirrors:
    - arches: [i386, amd64]
      search:
        primary:
          - http://%(ec2_region)s.ec2.archive.ubuntu.com/ubuntu/
          - http://%(availability_zone)s.clouds.archive.ubuntu.com/ubuntu/
          - http://%(region)s.clouds.archive.ubuntu.com/ubuntu/
        security:
          - http://ubuntu-security-dns.myorg.com/ubuntu/
      failsafe:
        primary: http://archive.ubuntu.com/ubuntu
        security: http://security.ubuntu.com/ubuntu/

ubuntu-security-dns.myorg.com is the domain name we’re using to temporarily only return valid IP addresses.

I set up monitoring using the following script to check the path /ubuntu/dists/noble-security/, since that also seemed to be missing on the broken servers.

#!/bin/bash

set -euo pipefail

DOMAIN="${1:-security.ubuntu.com}"
PATH_TO_CHECK="${2:-/ubuntu/dists/noble-security/}"

echo "Checking servers for: $DOMAIN"
echo "Path: $PATH_TO_CHECK"
echo ""

# Resolve domain to IP addresses
IPS=$(dig +short "$DOMAIN" A | grep -E '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$')

if [ -z "$IPS" ]; then
  echo "Error: Could not resolve $DOMAIN to any IP addresses"
  exit 1
fi

# Check each IP address
results=()
failed=0

for ip in $IPS; do
  status_code=$(curl -s -o /dev/null -w "%{http_code}" \
    --connect-timeout 5 \
    --max-time 10 \
    "http://$ip$PATH_TO_CHECK" 2>/dev/null || echo "000")

  results+=("$ip: $status_code")

  if [[ ! "$status_code" =~ ^2 ]]; then
    failed=1
  fi
done

echo ""
echo "Summary:"
for result in "${results[@]}"; do
  echo "  $result"
done

exit $failed

The results right now:

[~]$ ./check-mirror.sh
Checking servers for: security.ubuntu.com
Path: /ubuntu/dists/noble-security/

Summary:
185.125.190.81: 200
91.189.91.83: 200
185.125.190.83: 200
91.189.91.81: 200
185.125.190.36: 404
185.125.190.39: 404
185.125.190.82: 200
91.189.91.82: 200
[~]$ echo $?
1

In summary there are two ways this problem manifests to the user:

First, and most obvious, you can run apt-get upgrade and get the following error indicating the server doesn’t have noble package list at all:

Err:9 Index of /ubuntu noble-security Release
404 Not Found [IP: 185.125.190.36 80]

The second way it can manifest is that you could get a working server and successfully update your package list. BUT, when you try to update packages, you could resolve a non-working server, which doesn’t have any of the packages you’re looking for:

E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/bind9-host_9.18.39-0ubuntu0.24.04.2_amd64.deb 404 Not Found [IP: 91.189.91.83 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/bind9-dnsutils_9.18.39-0ubuntu0.24.04.2_amd64.deb 404 Not Found [IP: 91.189.91.83 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/bind9-libs_9.18.39-0ubuntu0.24.04.2_amd64.deb 404 Not Found [IP: 91.189.91.83 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/universe/b/bind9/dnsutils_9.18.39-0ubuntu0.24.04.2_all.deb 404 Not Found [IP: 91.189.91.83 80]

This can even happen if you are running a script like apt-get update && apt-get upgrade because the DNS record has a 60 second TTL, and apt could use a different server for each request depending on when exactly your dns cache expires…

kubacki · October 23, 2025, 10:57pm

Found out from opening a ticket with mirrors@ubuntu.com that there were two things going on:

The IP addresses 185.125.190.36 and 185.125.190.39 were accidentally left in the security.ubuntu.com DNS record. They should be removed now and should alleviate part of the issues you are observing.

We’re actively troubleshooting an issue with 91.189.91.83 specifically where it misses a sync trigger to pull updates. We believe this is the only host exhibiting the behavior, and we’re working to mitigate it as best we can.

I’m not seeing any issues anymore, so I’m closing this topic.

sarnold · October 24, 2025, 1:31am

Thanks for the report; note that webservers determine what content to return to web clients based on the URL, and the examples of:

http://185.125.190.36/ubuntu/dists/
http://archive.ubuntu.com/ubuntu/dists/

are actually requesting two completely different resources from whatever host is serving these requests, even in the case that you make your request to the 185.125.190.36 machine.

We really should fix up the virtual hosting config on these machines, or remove then from this DNS record, or decommission them entirely, or something, but they have the correct resources when well-formed requests are made:

for addr in 91.189.91.81 91.189.91.82 91.189.91.83 185.125.190.36 185.125.190.39 185.125.190.81 185.125.190.82 185.125.190.83 ;
do echo ==== $addr ==== ;
curl --resolve archive.ubuntu.com:80:${addr} -sq http://archive.ubuntu.com/ubuntu/dists/ | wc -l ; echo ;
done 
==== 91.189.91.81 ====
65

==== 91.189.91.82 ====
65

==== 91.189.91.83 ====
65

==== 185.125.190.36 ====
65

==== 185.125.190.39 ====
65

==== 185.125.190.81 ====
65

==== 185.125.190.82 ====
65

==== 185.125.190.83 ====
65

system · October 27, 2025, 1:32am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.