The Debian project is hosting their package mirror via the CDN service Fastly using the domain name, deb.debian.org. Since it is backed by a CDN service it is spread out across the globe and users of it should get good download speed regardless of their geographical location.
Since the recent swarms of AI crawlers and potentially intended DDoS attacks against Ubuntu package mirrors, such as the ongoing DDoS of the official Swedish mirror se.archive.ubuntu.com, would it not make sense to serve a similar CDN solution for Ubuntu similar to how Debian does it?
Perhaps Ubuntu should not rely on a third party for their official mirror but the main mirror, archive.ubuntu.com, is currently not beneficial for users outside of the USA since its DNS record seems to be setup using round-robin for two IP addresses located in USA. I’m not sure archive.ubuntu.com is the official mirror but it is the default mirror on a Ubuntu install.
I think Ubuntu would benefit of having one main mirror that is “GeoDNS aware”, meaning that users around the globe could use it, get good download speeds and Canoncial could focus on keeping it stable.
1 Like
Welcome to the Ubuntu Discourse @k-markus
I think the comparison with Debian is understandable, but there are a few important differences.
Debian’s deb.debian.org is indeed a very elegant solution because it abstracts the mirror selection process and benefits from CDN-backed distribution. For end users, that often means better consistency and less concern about choosing a suitable mirror.
However, Ubuntu’s infrastructure has historically been built around a large network of official and community-operated mirrors, including country mirrors. This approach distributes bandwidth costs and allows local organizations to contribute resources while keeping package distribution decentralized.
A CDN-backed endpoint could certainly improve resilience and provide a more uniform experience, especially during traffic spikes or large-scale automated crawling. On the other hand, operating such a service at Ubuntu’s scale is not only a technical decision but also an operational and financial one.
Rather than replacing the current mirror ecosystem, a hybrid model might be worth considering: keep the existing mirror network while offering a CDN-backed global endpoint as the recommended default for APT, similar to what Debian provides.
That would preserve the strengths of the current mirror infrastructure while improving user experience for people who are far away from major mirrors or affected by regional performance issues.
2 Likes
Welcome to the Ubuntu Discourse @k-markus
Thank you!
Rather than replacing the current mirror ecosystem, a hybrid model might be worth considering: keep the existing mirror network while offering a CDN-backed global endpoint as the recommended default for APT, similar to what Debian provides.
Yes, my intention was not that the CDN-backed mirror should replace the already established distributed mirror setup but as you mentioned the recommended default, indeed similar to what Debian provides.
Hm, I think might be missing that it’s powered by cloudflare:
$ nslookup archive.ubuntu.com
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
archive.ubuntu.com canonical name = archive.ubuntu.com.cdn.cloudflare.net.
Name: archive.ubuntu.com.cdn.cloudflare.net
Address: 172.66.152.176
Name: archive.ubuntu.com.cdn.cloudflare.net
Address: 104.20.28.246
Name: archive.ubuntu.com.cdn.cloudflare.net
Address: 2606:4700:10::6814:1cf6
Name: archive.ubuntu.com.cdn.cloudflare.net
Address: 2606:4700:10::ac42:98b0
This is what I get when trying to locate only the last IP; haven’t tried all:
IP Address: 2606:4700:10::ac42:98b0
Hostname:
Organization: Cloudflare
ASN: AS13335 Cloudflare, Inc.
Continent: Unknown
Country: Unknown
Latitude\Longitude: Unknown
Region: Unknown
City: Unknown
So I think the GeoDNS is included in that deal. How did you determine those IP’s to be in the USA?
Same here: `archive.ubuntu.com` currently CNAMEs to `archive.ubuntu.com.cdn.cloudflare.net` for me too, including IPv6. So I would first check with `dig +short archive.ubuntu.com CNAME` and also from the affected network/VPS. If you only look at one GeoIP database for the returned A records, it can easily look “US based” while Cloudflare is still serving it through a nearby edge.
1 Like
Exactly; looks like I just happened to pick the right IP, because choosing the other results in it being identified as located in the USA. Or just repeatedly running geoiplookup I get this:
$ geoiplookup --verbose archive.ubuntu.com
GeoIP Country Edition: US, United States
$ geoiplookup --verbose archive.ubuntu.com
GeoIP Country Edition: IP Address not found
It’s worth pointing out that deb.debian.org is provided by fastly as a donation to the Debian project. I think commercial rates for backing archive.ubuntu.com with fastly would be extremely expensive (it’s not really what they are set up to provide).
So I think the GeoDNS is included in that deal. How did you determine those IP’s to be in the USA?
From Sweden I get a consistent ~150ms latency towards archive.ubuntu.com and towards se.archive.ubuntu.com I get ~50ms. This is usually a good indicator that the service is hosted outside of Europe and most probably somewhere in the Americas.
Looking at the result from the web tool dnschecker.org that performs forward lookups of a domain from DNS servers spread out in the world it is pretty telling that archive.ubuntu.com is setup using round-robin between the 104.20.28.246 and 172.66.152.176 IP addresses.
It’s worth pointing out that deb.debian.org is provided by fastly as a donation to the Debian project. I think commercial rates for backing archive.ubuntu.com with fastly would be extremely expensive (it’s not really what they are set up to provide).
If archive.ubuntu.com is already backed by Cloudflare’s CDN solution, are Canonical not already investing in this? If its purpose is intended for the Americas I would say that it would be a bit unfair to other Ubuntu users outside of the Americas. Perhaps this is not true and in that case it would be good to document that.
I started this thread out of a bit of frustration since the regional mirror in Sweden, se.archive.ubuntu.com (hosted by a University) is currently having major issues with swarms of AI/bot crawlers effectively performs a DDoS attack to it. I though about the deb.debian.org mirror since it is backed by a CDN which has methods of dealing with these types of attacks and if Ubuntu has a similar solution that Ubuntu users can use. I have had release upgrades fail due to the se.archive.ubuntu.com mirror being unavailable at times.
From Germany I’m getting ~10ms, when I ping archive.ubuntu.com:
$ ping archive.ubuntu.com
PING archive.ubuntu.com (2606:4700:10::6814:1cf6) 56 data bytes
64 bytes from 2606:4700:10::6814:1cf6: icmp_seq=1 ttl=59 time=9.73 ms
64 bytes from 2606:4700:10::6814:1cf6: icmp_seq=2 ttl=59 time=9.86 ms
64 bytes from 2606:4700:10::6814:1cf6: icmp_seq=3 ttl=59 time=10.1 ms
^C
--- archive.ubuntu.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 9.730/9.898/10.109/0.157 ms
Got the same result on three consecutive runs. Maybe it’s an issue with IPv4; you only mentioned the IPv4 addresses, after all — those are a scarce resource. I am on a dual stack connection. I’m also a bit puzzled that you don’t seem to be using IPv6; from all I’ve heard, Sweden is running circles around Germany when it comes to broadband internet, so I kind of expected everything to be IPv6 (only, maybe; no dual stack).