Possible security.ubuntu.com infrastructure issue — TLS timeouts on specific backend IPs

Support and Help
1

Ubuntu Version

Ubuntu 24.04 LTS

Desktop Environment (if applicable)

GNOME

Problem Description

Possible security.ubuntu.com infrastructure issue — TLS timeouts on specific backend IPs
System

Ubuntu 24.04 LTS (Noble)
AMD64
Ubuntu Pro enabled (personal/free entitlement)
IPv6 is intentionally disabled at my router; IPv4 is the only active IP protocol
APT repositories are configured to use HTTPS

Problem

I am experiencing intermittent apt update failures when accessing security.ubuntu.com.

The failure is:

Err:8 Index of /ubuntu noble-security InRelease
Could not wait for server fd - select (11: Resource temporarily unavailable) [IP: 185.125.190.83 443]

W: Failed to fetch https://security.ubuntu.com/ubuntu/dists/noble-security/InRelease
Could not wait for server fd - select (11: Resource temporarily unavailable) [IP: 185.125.190.83 443]

A subsequent attempt may select 185.125.190.82 instead and produce the same error.

Other Ubuntu repositories, including archive.ubuntu.com and the ESM repositories, work normally.
Investigation

I checked DNS resolution:

$ getent ahosts security.ubuntu.com

185.125.190.81
185.125.190.82
185.125.190.83
91.189.91.81
91.189.91.82
91.189.91.83
91.189.92.22
91.189.92.23
91.189.92.24
…

I then tested the individual IPv4 addresses directly using curl --resolve, so that each backend could be tested independently.

The results were:

185.125.190.81 → HTTP 200
185.125.190.82 → SSL connection timeout
185.125.190.83 → SSL connection timeout

91.189.91.81 → HTTP 200
91.189.91.82 → HTTP 200
91.189.91.83 → HTTP 200
91.189.92.22 → HTTP 200
91.189.92.23 → HTTP 200
91.189.92.24 → HTTP 200

For example:

===== 185.125.190.81 =====
attempt 1: HTTP 200 | connect 0.044951s | total 1.141561s
attempt 2: HTTP 200 | connect 0.040735s | total 1.017670s
attempt 3: HTTP 200 | connect 0.042198s | total 1.267111s

===== 185.125.190.82 =====
curl: (28) SSL connection timeout
attempt 1: HTTP 000 | connect 0.043240s | total 5.002164s
curl: (28) SSL connection timeout
attempt 2: HTTP 000 | connect 0.043791s | total 5.002499s
curl: (28) SSL connection timeout
attempt 3: HTTP 000 | connect 0.043791s | total 5.002776s

===== 185.125.190.83 =====
curl: (28) SSL connection timeout
attempt 1: HTTP 000 | connect 0.041845s | total 5.002159s
curl: (28) SSL connection timeout
attempt 2: HTTP 000 | connect 0.040128s | total 5.002344s
curl: (28) SSL connection timeout
attempt 3: HTTP 000 | connect 0.040128s | total 5.002611s

The important characteristic is that the TCP connection itself succeeds very quickly (~40 ms), but TLS does not complete on .82 and .83.

For comparison, a direct HTTPS request to .23 completed normally:

  • Connected to security.ubuntu.com (91.189.92.23) port 443
  • SSL connection using TLSv1.3
    …
    < HTTP/1.1 200 OK

So this does not appear to be a general inability to reach security.ubuntu.com, a DNS problem, a CA/certificate problem, or a general HTTPS problem.
Possible correlation with a previous Livepatch issue

A few days ago I experienced a very similar problem with Ubuntu Livepatch.

At that time, Livepatch would not update and appeared to fail with TLS/HTTPS connectivity problems involving only certain Canonical/Ubuntu endpoint addresses. After extensive troubleshooting, the problem appeared to be upstream rather than a local Ubuntu configuration problem, and Livepatch subsequently began working without a significant local configuration change.

The current APT problem has a remarkably similar signature:

TCP connection succeeds
TLS handshake times out
only certain Ubuntu/Canonical IP addresses appear affected
other addresses for the same service work normally

I therefore wonder whether there is currently an infrastructure, routing, peering, or backend-server problem affecting some Canonical endpoints.
Current status

The local package system itself appears healthy:

$ dpkg --audit

returns no output.

There are no held packages:

$ apt-mark showhold

returns no output.

The only remaining upgrade is:

power-profiles-daemon 0.21-1ubuntu2 → 0.21-1ubuntu3

which is currently shown by APT as:

0.21-1ubuntu3 500 (phased 20%)

so I understand that this is unrelated and is being deferred normally due to phased updates.

The actual package upgrades performed before the reboot completed successfully.
Question

Could someone from the Ubuntu/Canonical community confirm whether 185.125.190.82 and 185.125.190.83 are known to have connectivity/TLS problems, or whether there is currently any known issue affecting these security.ubuntu.com backend addresses?

I am particularly interested in whether this could be related to the similar Livepatch connectivity problem I experienced a few days ago.

I have not changed my APT configuration or attempted to work around the problem yet, because I would prefer to preserve the reproducible behavior for diagnosis.

2
3

I’m still seeing the issue, and I think I may have isolated something useful.

My system is Ubuntu 24.04 LTS. security.ubuntu.com resolves to multiple IPv4 addresses. I tested the individual endpoints directly with curl --resolve.

The results are very consistent:

185.125.190.81  → HTTP 200, TLS completes normally
185.125.190.82  → TCP connects (~40 ms), then TLS times out at 5 seconds
185.125.190.83  → TCP connects (~40 ms), then TLS times out at 5 seconds

91.189.91.81    → HTTP 200
91.189.91.82    → HTTP 200
91.189.91.83    → HTTP 200
91.189.92.22    → HTTP 200
91.189.92.23    → HTTP 200
91.189.92.24    → HTTP 200

I repeated the tests three times against .81, .82 and .83. .81 succeeded every time, while .82 and .83 timed out every time.

APT has been failing specifically when it gets one of the problematic addresses, for example:

Could not wait for server fd - select (11: Resource temporarily unavailable)
[IP: 185.125.190.83 443]

and earlier:

[IP: 185.125.190.82 443]

Interestingly, I had an almost identical problem with Ubuntu Livepatch a few days ago, where only certain Canonical endpoints appeared to fail during TLS. Livepatch subsequently started working without any significant local configuration changes.

So although the status page now says the incident is resolved, I suspect there may still be an issue affecting particular backend endpoints or network paths.

Could you check whether you’re seeing the same specific IPs failing?

4

Interesting — I’m seeing what looks like a related issue from a completely different network.

I’m on Ubuntu 24.04 LTS and initially had apt failures against security.ubuntu.com. My failures are over HTTPS/TLS rather than HTTP. APT was specifically reporting failures such as:

Could not wait for server fd - select (11: Resource temporarily unavailable)
[IP: 185.125.190.83 443]  

I also had a very similar TLS timeout problem with Ubuntu Livepatch a few days ago, involving only certain Canonical endpoint addresses.

Since you're seeing failures from Blacksmith runners and I'm seeing them from my own connection, this makes me wonder if we're both hitting a remaining issue in Canonical's infrastructure after today's incident was marked resolved.

It would be interesting to know whether your failures are consistently associated with the same IPs when you retry.
5

On my local machine only 91.189.91.81 seems to time out and fail hard.

On the blacksmith runners, it cycles through all of them and fails every single time.


  Connection failed [IP: 91.189.92.22 80]


  Connection failed [IP: 91.189.92.24 80]


  Connection failed [IP: 91.189.91.83 80]


  Connection failed [IP: 91.189.91.82 80]


  Connection failed [IP: 185.125.190.81 80]
6

This seems to strengthen the possibility that this is path-dependent rather than simply a couple of bad Canonical endpoints.

My situation is different but has a similar pattern:

  • 185.125.190.81 works consistently for me

  • 185.125.190.82 consistently connects at TCP level but times out during TLS

  • 185.125.190.83 consistently connects at TCP level but times out during TLS

  • the 91.189.91.x and 91.189.92.x addresses I tested all completed HTTPS successfully

So we’re seeing different subsets of the Ubuntu infrastructure fail depending on where the connection originates.

The fact that the Blacksmith runners fail against essentially every address they try makes me wonder whether there is a routing/peering/upstream connectivity issue affecting certain source networks, rather than the individual Ubuntu servers themselves.

Given that there was a Canonical incident earlier today involving both archive.ubuntu.com and security.ubuntu.com, it would be interesting to know whether Canonical is aware of any remaining network-path issues after the incident was marked resolved.

I’ll keep monitoring my side as well.

7

Update from my side: apt update is now working again.

The latest run completed successfully against both security.ubuntu.com and archive.ubuntu.com:

Hit:5 https://security.ubuntu.com/ubuntu noble-security InRelease
Hit:6 https://archive.ubuntu.com/ubuntu noble InRelease
Get:7 https://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
...
Fetched 3,612 kB in 19s (192 kB/s)
Reading package lists... Done

There were no connection or TLS errors this time.

It is still noticeably slower than normal, however, so it seems like the infrastructure may be recovering but isn’t quite back to its usual performance yet.

I’ll continue monitoring it.

8

This is a known issue. See Redirecting apt sources to address increased load: A Guide to Using Community Mirrors for a reliable solution.

2 Likes