Ubuntu 24.04 unattended-upgrades failed to restart nginx

Project Discussion Server
1

Today unattended-upgrades upgraded libc6 and restarted all services using it.

I don’t want to discuss whether unattended-upgrades should or should not restart services, I will go with whatever Ubuntu 24.04’s default behavior is.

I just want to initiate a discussion on how to prevent nginx from failing to restart again.

This is what happened:

# tail /var/log/apt/history.log

Start-Date: 2025-09-23  06:09:19
Commandline: /usr/bin/unattended-upgrade
Upgrade: libc6:amd64 (2.39-0ubuntu8.5, 2.39-0ubuntu8.6), locales:amd64 (2.39-0ubuntu8.5, 2.39-0ubuntu8.6), libc-dev-bin:amd64 (2.39-0ubuntu8.5, 2.39-0ubuntu8.6), libc-bin:amd64 (2.39-0ubuntu8.5, 2.39-0ubuntu8.6), libc-devtools:amd64 (2.39-0ubuntu8.5, 2.39-0ubuntu8.6), libc6-dev:amd64 (2.39-0ubuntu8.5, 2.39-0ubuntu8.6)
End-Date: 2025-09-23  06:09:31

Needrestart triggered service restarts

# less /var/log/unattended-upgrades/unattended-upgrades-dpkg.log

Restarting services...
 /etc/needrestart/restart.d/systemd-manager
 systemctl restart cron.service nginx.service ssh.service systemd-journald.service systemd-resolved.service
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.

nginx failed to restart with this error

# systemctl status nginx.service 
× nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2025-09-23 06:09:12 CEST; 2h 9min ago
   Duration: 6d 11h 15min 10.745s
       Docs: man:nginx(8)
        CPU: 53ms

Sep 23 06:09:12 server nginx[2416442]: 2025/09/23 06:09:12 [emerg] 2416442#2416442: host not found in upstream "ec2-3-120-6-158.eu-central-1.compute.amazonaws.com:8000" in /etc/nginx/sites->
Sep 23 06:09:12 server nginx[2416442]: nginx: configuration file /etc/nginx/nginx.conf test failed
Sep 23 06:09:12 server systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Sep 23 06:09:12 server systemd[1]: nginx.service: Failed with result 'exit-code'.
Sep 23 06:09:12 server systemd[1]: Failed to start nginx.service - A high performance web server and a reverse proxy server.

The upstream host DOES EXIST and IS RESOLVABLE.

The problem was, I assume, that systemd-resolved.service and nginx.service were restarted at the same time.
The fix, in this case, was easy, I just restarted nginx manually while systemd-resolved was running.

Unfortunately this caused production outage, not mentioning that all our maxscale, elasticsearch, mariadb, keepalived, etc… services were restarted as well. I am not going to comment on that, and I am glad we have not migrated our RabbitMQ cluster to Ubuntu 24.04 yet…

Since Ubuntu 24.04 defaulted to restarting all affected services after unattended-upgrade does a library upgrade, is it at least possible to restart them one by one? Or restart all systemd- services prior the non-systemd services?

Thanks,
Oskar

2

BTW, I really appreciate these two docs

They explain very well how unattended-upgrades and needrestart behave in Ubuntu 24.04. And thanks to them I know how to exclude nginx.service from being restarted (or how to go back to the old needrestart behavior altogether)

But what if I decide to go with the Ubuntu 24.04 server defaults? I might end up with not running nginx…

3

Sorry about this problem. An outage is definitely a bad thing.

Let’s unpack this into a few topics and see what options are available.

unattended-upgrades

It can be configured to not apply certain updates, and perhaps for your use case, glibc should be in that list. The documentation is at https://documentation.ubuntu.com/server/how-to/software/automatic-updates/

needrestart

I found this discourse post, linked from the README.Ubuntu file in the package: `needrestart` changes in Ubuntu 24.04: service restarts. This talks about the change in behavior introduced in 24.04 about automatic service restarts.

In particular, this talks about how to configure it to prevent restarting single services: `needrestart` changes in Ubuntu 24.04: service restarts

nginx

Perhaps it could treat an unresolvable name as a non-fatal error in its configuration check? But maybe in your case, if the name, at that time, is indeed unresolvable, then it would fail to start for real.

Bugs

In the end, regardless of the reason, what happened is that your service failed while an update was being applied. Maybe the less intrusive fix would be to not attempt restarts after a glibc security update, or block glibc from getting automatic security updates. There are pros and cons. An outage is definitely a con, but we also have to think about the generic case in Ubuntu.

Perhaps file a bug against glibc, since its update is what started this all, and the developers can go from there to see what’s the best way to handle this scenario in a generic way.

For your specific case, I suggest, for now, to investigate the alternatives I listed above, particularly in unattended-upgrades and needrestart configuration options.