About `apt upgrade` and phased updates

You may have noticed recently that updating your system with apt upgrade sometimes produces a weird message about packages being kept back…like this one:

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  (Names of <X> held back packages listed here)
0 upgraded, 0 newly installed, 0 to remove and <X> not upgraded.

If you’ve ever used combinations of packages from different releases or third party repos, you may be familiar with this message already. However, it has become a much more common occurrence due to something called “phased updates”.

What are phased updates?

Phased updates are software updates that are rolled out in stages, rather than being provided to everyone at the same time. Initially, the update is provided only to a small subset of Ubuntu machines. As the update proves to be stable, it is provided to an ever-increasing number of users until everyone has received it (i.e., when the update is “fully phased”).

The good news is, you don’t need to do anything about the “packages kept back” message – you can safely ignore it. Once the update has been deemed safe for release, you will receive the update automatically.

Why is Ubuntu doing this?

Although updates are thoroughly tested before they get released at all, sometimes bugs can be hidden well enough to escape our attention and make it into a release – especially in highly specific use cases that we didn’t know we needed to test. This can obviously cause problems for our users, and used to be the norm before we phased updates through apt.

Update phasing makes it much easier for us to detect serious breakages early on – before they have a chance to cause problems for the majority of our users. It gives us the opportunity to hold back the update until the bugs are fixed.

In other words, it directly benefits our users by increasing the safety, stability and reliability of Ubuntu.

The phasing system makes it so that different sets of users are chosen to be the first to get the updates, so that there isn’t one group of unlucky people who always get potentially broken updates soon after release.

Note:
It should be mentioned here that security updates are never phased.

Can I turn off phased updates?

That depends on how stable you need your system to be. If you just want to avoid any notices about packages being held back during apt updates, and you’re willing to be one of the first people to get updates whenever they’re released, you can turn off phased updates. Be warned, though – if an update is broken, you will almost always be in the first set of people to get it (i.e., you’re basically volunteering yourself as a guinea pig for the early update releases!). It will get rid of the “held back packages” in apt message, though.

If that doesn’t sound like something you want, leave phased updates on (this is the default). You will still temporarily get the “held back packages” message, but your machine will be more protected from updates that might otherwise break it – and once the packages are ready to be safely installed on your system, they will no longer be held back.

Can I apt upgrade the individual packages? (and should I?)

While you can technically get around phased updates by running apt install on individual held back packages, it’s not recommended. You’re unlikely to break your machine by doing this – as long as the package is being held back due to update phasing.

If you want to apt upgrade a package, you should first carefully examine the proposed changes that apt would make before you proceed. If the package update was kept back for a reason unrelated to phasing, apt may be forced to remove packages in order to complete your request, which could then cause problems elsewhere.

How do I turn off phased updates?

If you’re sure that you want to disable phased updates, reverting to the old behaviour, you can change apt’s configuration by creating a file in /etc/apt/apt.conf.d called 99-Phased-Updates (if /etc/apt/apt.conf.d/99-Phased-Updates doesn’t already exist). In the file, simply add the following lines:

Update-Manager::Always-Include-Phased-Updates true;
APT::Get::Always-Include-Phased-Updates true;

Again, please only do this if you really know what you’re doing and are absolutely sure you need to do it (for instance, if you are intentionally installing all the latest packages to help test them – and don’t mind if your system breaks). We definitely don’t recommend turning off phased updates if you’re a newer user.

Why is this suddenly happening now?

Phased updates have been part of the update-manager on Ubuntu Desktop for quite a while (since 13.04, in fact!), but were implemented in APT in 21.04. It now works on all versions of Ubuntu (including Ubuntu Server, Raspberry Pi, and containers). Since this includes the 22.04 LTS, it’s now getting a lot more attention as a result!

How does it actually work?

Phased updates depend on a value derived from your machine’s “Machine ID”, as well as the package name and package version. The neat thing about this is that phasing is determined completely at the client end; no identifiable information (or indeed any new information at all) is ever sent to the server to achieve update phasing.

When the software update is released, the initial subset of machines to receive the update first is chosen at random. Only if there are no problems detected by the first set of users will the update be made available to everyone.

For more detailed information, including about how changes to phasing are timed, you can check the Ubuntu wiki page on phased updates.

How can I find out more information about currently phased packages?

You can find out the phasing details of a package by using the apt policy command:

apt policy <package>

For example, at the time of writing, the package libglapi-mesa has a phased update. Running apt policy libglapi-mesa then produces an output like this:

libglapi-mesa:
  Installed: 22.0.5-0ubuntu0.3
  Candidate: 22.2.5-0ubuntu0.1~22.04.1
  Version table:
 	22.2.5-0ubuntu0.1~22.04.1 500 (phased 20%)
    	500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
 *** 22.0.5-0ubuntu0.3 100
    	100 /var/lib/dpkg/status
 	22.0.1-1ubuntu2 500
    	500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

In this output you can see that this package is 20% phased.

You can see the status of all packages currently being phased in Ubuntu at Released Ubuntu SRUs

Note:
There is a bug report currently active about the fact that the “kept back” message is not as informative as it could be, and the issue is on our radar. If this issue also affects you, you have an Ubuntu Single Sign-On (SSO) account and can log into Launchpad, you can click on the link near the top of the page that says “This bug affects 85 people. Does this bug affect you?”. If you then click on “Yes, it affects me”, it will increase the bug heat rating, making it more significant.

Further reading

5 Likes

The first time I got it, I thought I had broken something in my system and I was trying to fix it until I realize it was on purpose. I think the message in the terminal could be more explicit. For example:
“The package is in a phased update state and will be available to you in the coming days. If you wish to disable phased updates, please use etc…”

5 Likes

Thanks for commenting! I think a lot of people have unfortunately had the same experience (at least, judging by the bug that was reported about it), so fixing that message to make it more helpful is definitely something that’s being looked into.

Hi, nice write up. There’s a few things that I’m not clear on and haven’t seen addressed on AskUbuntu or any of the usual info sources.

Currently, I’m running 22.04.1 desktop and I’ve noticed that there are few packages that have been at set at 0% progress for an extended period of time. Looking at https://people.canonical.com/~ubuntu-archive/phased-updates.html you can see that gnome-remote-desktop has been at 0% for almost a month now after being reset from 70%. So obviously errors have been found in the update and further deployment is halted, but no information is available to general users about the nature of the bug(s) as error reports are only able to be viewed by specific members of a group granted access to the error reports.

Is there anyway to tell if the bugs found in the update packages are being addressed? What’s the policy for how long updates should be held at 0% before its removed and another update is offered to revert the changes in the buggy one? Have these updates been abandoned by the maintainers? Focal has gnome-software that originally made it to 90% and then was set to 0% where its remained for 99 days! The implication is that a number of Focal systems have a gnome-software buggy update installed on their systems that apparently maintainers have let fall by the wayside.

Generally, I think phased updates are a good idea, but it would be beneficial to allow general users the ability see what errors are associated with the updates to determine if the error is applicable to their system so they can decide whether to proceed with the upgrade or not. For example, if the error(s) with gnome-remote-desktop only affects ARM systems and not amd64, then users with amd64 systems could go ahead and install the update. There’s no way of knowing as it stands now.

p.s.
$ apt show now will display Phased-Update-Percentage field for packages in a phased update.
$ apt policy ~U will show policy information on all packages available for upgrades, include those with phased update status.
Also the utility grep-aptavail from the dctrl-tools package will show packages with Phased-Update-Percentage.
$ grep-aptavail Phased -sPackage,Phased-Update-Percentage

1 Like

Those are some great questions, Steven!

There’s an old bit of software tester wisdom that says if you move a piece of furniture and find one bug, then assume there’s more. So with software, if you find a bug it’s best to assume more are hiding in the corners and cracks. I think with phasing this is some of the unspoken philosophy - once an update is found to have one defect, there’s a good chance that there’s other bugs that just haven’t been found yet. Some may be hiding behind the first bug, others might be triggered by its fix, or whatever oversight led to the first bug could have missed memory leaks or other issues that’d only reveal themselves over longer time or heavier usage. So, the phasing in a way treats a bad bug as a signal that the update itself needs a full workdown of retesting and analysis.

2 Likes

Ah, that’s why I was seeing many more ‘kept back’ package upgrades since moving to 22.04. Having this IN BIG LETTERS in the release notes would have been nice: it looked like a problem.

Because of that, I’d added aptitude safe-upgrade to the update script - it turns out that ignores the phasing.

Now I know what’s going on, I might remove it, but I would be slightly more convinced that security updates were never phased if that didn’t happen to snap packages, where updates with security patches can be delayed for a day or two. Oh, that’s rationing them (because of under resourcing the snap store?) not phasing them, isn’t it?

This isn’t quite true - if multiple packages are phasing all at once and only some of them finish phasing for a particular user, they will stop being displayed, but the old ones will still be there. Also the message will eventually come back. As a suggestion, maybe rather than “the message will disappear when the phasing has reached 100%”, perhaps it could read something similar to “the packages will not be held back once it’s ready to be installed on your system” would work better? That way users won’t be confused when the message comes back or only part of it disappears.

This is awesome, by the way, thanks!

1 Like

Phased Updates have served me with the best security and user
updates for a while with no complaints. It is seamless on time with each update before each CVE came out after Bionic Ubuntu. Thus I applause our Ubuntu Security Team for this. Along with the entire Ubuntu Team for this effort, Foundations & Desktop.

1 Like

Thanks for commenting, that’s a great point and an excellent suggestion – I’ve edited the post now.

And thanks for the original post! :slight_smile:

1 Like

You mention that security updates are never phased, yet in my experience that doesn’t seem to be the case. For example, right now I’m seeing shim-signed and grub-efi-amd64 both phasing on my system with both of them having security-related updates. I’m also noticing cryptic messages for certain packages on the “Released Ubuntu SRUs” status page with notices like “The phasing of this update was manually stopped by an AA” for grub2-signed (and it has been like that for nearly a week). I applaud the effort of your team to create a system where packages are tested in a controlled manner to increase reliability but there still are a number of areas where there is a lot of confusion. Resolving the bug report about clarifying the “held back” notices would be a great first step to removing some of this confusion.

Hello @number-one, I can confirm security updates are not phased. The grub2-signed which you report being phased is likely version 1.187.3~22.04.1, which is available via jammy-updates, and not jammy-security. In other words the package has not been released as a security update. You can check the publication history of the package from this page: https://launchpad.net/ubuntu/+source/grub2-signed/+publishinghistory. In the Pocket column you can check if a given version has been released to -updates or -security.

4 Likes

@paride Thanks for the clarification. It’s a bit confusing because there can apparently be some security-related updates coming from the “standard” update channel (the notes reference CVE IDs and security issues). So I guess the phasing is really just channel-based, though I’m not clear on what criteria is used to determine the channel/pocket through which an update is released.

EDIT I should also note that some of the confusion may be coming from the cockpit console (for which Ubuntu is not responsible). It appears that cockpit may label package updates as security-related because previous versions had security-related changes even if the current update in question does NOT contain any security-related patches.