Hello everyone,
I am working as a sysadmin in a smaller company and currently I am working on migrating and harmonization a handful of linux machines to ubuntu.
Most of them run software like gitea, bookstack, some docker containers, …
I was thinking of enabling unattended-upgrades but I am unsure if this can break things in future. I do have the following setup in unattended-upgrades:
// Note that in Ubuntu security updates may pull in new dependencies
// from non-security sources (e.g. chromium). By allowing the release
// pocket these get automatically pulled in.
Unattended-Upgrade::Allowed-Origins {
"${distro_id}:${distro_codename}";
"${distro_id}:${distro_codename}-security";
// Extended Security Maintenance; doesn't necessarily exist for
// every release and this system may not have it installed, but if
// available, the policy for updates is such that unattended-upgrades
// should also install from here by default.
"${distro_id}ESMApps:${distro_codename}-apps-security";
"${distro_id}ESM:${distro_codename}-infra-security";
// "${distro_id}:${distro_codename}-updates";
// "${distro_id}:${distro_codename}-proposed";
// "${distro_id}:${distro_codename}-backports";
};
So does e.g. a LAMP stack can get broken with this setup?
My suggestion is not to use unattended upgrade if you use ubuntu in an enterprise environment where stability is preferred. We installed ubuntu server version in our k8s cluster without knowing its automatic upgrade feature. After some days, it upgrades some packages by itself, including linux kernels. This behavior caused user jobs to fail. And after server reboot, the new kernel is having trouble with GPU usage inside containers.
We have a k8s GPU cluster running ubuntu24.04.2 LTS server version on all nodes. It was deployed in the last month. We are not aware of the automatic upgrade feature of ubuntu since we have been using CentOS/RockyLinux for many years. Two days ago, those servers upgraded to kernel 6.8.0-63 by themselves. We observed several issues:
kubelet was restarted on some nodes, which caused user job failure.
One server reported NCCL related errors in container. A manual server reboot fixed this.
One server’s raid configuration was corrupted and had to be rebuilt. Fortunately we don’t have valuable data on compute nodes.
We tried to downgrade the Linux kernel to 6.8.0-62. But it still has the GPU error issue. Probably we will have to reinstall those servers and disable automatic upgrade after the installation.
A kernel update requires a reboot, just updating the kernel is essentially a no-op on the system. I suspect the reboot could have had a role in these problems, specially because your downgrade didn’t solve it.
Unattended Upgrades, which has been in use on millions of machines for well over a decade is very unlikely to crash your LAMP stack. Ubuntu has supported (and tested) LAMP software for a very long time.
Of course, a clever sysadmin can find ways to crash any software…so we recommend sticking with the default settings when possible.
Since we don’t know anything about your specific circumstances, I suggests spinning up a test server and giving it a stress test. Everybody has opinions, but real experience to form your own opinions is even better.
Why take the risk? In an enterprise situation you want full control over everything that alters your installation. To have it any other way would be foolish. As a sysop you can monitor all candidate files that are used in unattended upgrades before you then manually deploy them if you decide they are risk-free.
This is what I always did with my servers before retirement, I never allowed unattended upgrades to take place.
Very useful @ahasenack . I have a question. I have unattended-upgrades for security upgrades in all servers that I administer, and in the communities that I participate, some of them with Debian servers, it’s also enabled for security upgrades. This sentence in your documentation is what motivates this:
Even with all the care in the world, applying updates to a running system comes with risk. Ubuntu believes that risk to be less than the risk of NOT applying a security update, which is why unattended-upgrades will apply security updates by default.
What I don’t have is automatic reboot. My question: If Livepatch is enabled, is automatic reboot needed when the security update is applied? I thought that one of the main features of Livepatch was to allow the security upgrade without rebooting, even when it is a kernel upgrade.
I think the integration between unattended-upgrades and livepatch is not yet well done. It might tell you that the system needs to be rebooted, when it fact it doesn’t have to, if a livepatch was applied.
Our use case is not only a LAMP stack. It’s a k8s based MLOps platform, which supports users to do AI related tasks. The containers may be stateful and may have GPU attached. In case the upgrade process triggers a kubelet restart or systemctl daemon-reload, user jobs may be affected. The number four issue I reported earlier is still bothering us right now. With the latest 6.8.0-63 kernel, any systemctl daemon-reload will cause containers to lose GPU connection.