Phased updates in APT in 21.04

Foundations
#1

Starting with this development release, with APT 2.1.16 to be precise, APT now implements phased updates. Previously, only update-manager implemented phased updates, and this was only used on desktops - the implementation in APT means this works on servers, raspberry pis, and containers, too.

This means that some updates will be hold back on some machines while they are being phased. This is being decided by a value derived from the machine-id, the package name, and package version.

To control phased updates in APT:

  • To have all your machines phase the same, set the APT::Machine-IDoption to a UUID like /etc/machine-id (the format is not being checked right now, but that might change); or point Dir::Etc::machine-id to another machine-id file.
  • To always include phased updates, set APT::Get::Always-Include-Phased-Updates to true. This is the old behavior.
  • To never include phased updates, set APT::Get::Never-Include-Phased-Updates to false.

Note that this does not apply to fresh package installs. APT also respects the options from update-manager about phased updates which are similar to the ones above.

This also applies to any tool using the APT library, such as aptitude and packagekit.

Special notes:

  • To avoid breaking existing build chroots, chroots are excluded from this change for now (as are systems without an /etc/machine-id). This may change in 21.10.
  • The /etc/machine-id file is looked up relatively to Dir::Etc setting, which is /etc/apt. So if you point Dir or Dir::Etc elsewhere, you might have to copy machine-id to the right place or set that option to /etc/machine-id, otherwise phasing will be disabled, and phased updates always included.
9 Likes
Hirsute Hippo Release Notes
#2

Hey Julian,

Great, I always felt a bit unhappy that phasing only applied to some classes of system. I guess if we can start getting crash reports / metrics (e.g. if a systemd unit always fails after an update) out of servers, we could feed that back to the error tracker and then the phaser to make this even more useful.

Anyway, a question: for people who build packages, do we need to make any changes to always build with all updates or is apt doing something to detect these and opt out there? I’m thinking build chroots such as developers have, and also the proper Launchpad/Debian buildds.

#3

Oh right, I forgot to mention that systems without a machine-id and chroots (as detected by ischroot) are still using the classic behavior equivalent to always-include-phased-updates such that they don’t break build chroots.

I hope we can remove that ischroot code path in 21.10, that should give people time to change their build chroots.

(fixed in post)

3 Likes
#4
#5

Given that there are no packages in -updates for Hirsute is there anyway to dog food this feature before the final release of Hirsute?

#6

So should we modify LP buildd chroots to set APT::Get::Always-Include-Phased-Updates=true? (The best place to do this would probably be in livecd-rootfs.)

#7

Is somebody working on making mk-sbuild do this, and maybe sbuild-createchroot? It might be worth SRUing such changes too.

#8

I’m inclined to just make debootstrap --variant=buildd configure apt accordingly, that deals with most things I suppose, certainly those two.

Yes, livecd-rootfs needs work for that I suppose.

It’s just a confusing experience if you chroot into a system and it behaves different than when it’s your /, hence I want to avoid this mid-term.

#9

Agreed, doing it in debootstrap makes sense. livecd-rootfs doesn’t use --variant=buildd for buildd images; I’m sure there was some reason for that since I certainly knew about that variant when creating the buildd images, though I’ve forgotten what it was.

Chroots already have several other significant differences, of course, such as it being cumbersome to run services in them. So I don’t know if that should necessarily be a blocker, and I’m certainly comfortable with build chroots being different.

#10

Feedback: Over in AskUbuntu we get a lot of folks with multiple systems (or whole labs), and a lot of them are going to trip over the discrepancies caused by phased updates.

Suggestion: I think that it’s worthwhile to expose these settings, with a bit of explanatory commenting, in a config file. This will make support much easier. Something like…

/etc/apt/apt.conf.d/20phased-updates

// To have all your machines phase the same, set the same string in this field
// If commented out, apt will use /etc/machine-id to seed the random number generator
# APT::Machine-ID "aaaabbbbccccddddeeeeffff";

// Always include phased updates (Default = 1)
APT::Get::Always-Include-Phased-Updates "1"; 

// Never include phased updates
# APT::Get::Never-Include-Phased-Updates "0";

Question: Why both Always- and Never- ? Is there a use case for both being false (or true) simultaneously?

Question: Is there a way to output/dump/see the Phased Update Percentage? (“Run X on both of your systems – see how they are not the same? Your discrepancy is caused by phased updates.”)

Question: Is there a use case for manually setting the Phased Update Percentage? (not asking how)

#11

What I find awkward is that if I break my system and repair it, and I chroot into it from a live disc, and install stuff, I’ll get different updates than I get when I normally boot it, because it will now upgrade to everything that’s still phasing.

Or that a lxd container or a systemd-nspawn container or a docker container get different updates than chroots just because they’re containers and not chroots.

#12

We want to avoid shipping config files, as that means that if we wanted to change them to like fix a typo or add more details, we break automatic security updates.

Setting both to true is undefined. It’s a tri-state really, but making it true/auto/false is somewhat harder to code, I suppose. I’ve just copied the names from update-manager, they’ve been around for like half a decade.

apt policy shows you the percentages, and the pin will change to -1 if this update is not yet phased for your system, for example, in the apt test suite we have:

phased1:
  Installed: 1
  Candidate: 100
  Version table:
     100 500
        500 file:${TMPWORKINGDIRECTORY}/aptarchive unstable/main all Packages
     50 500 (phased 50%)
        500 file:${TMPWORKINGDIRECTORY}/aptarchive unstable/main all Packages
     10 1 (phased 10%)
        500 file:${TMPWORKINGDIRECTORY}/aptarchive unstable/main all Packages
 *** 1 100
        100 ${TMPWORKINGDIRECTORY}/rootdir/var/lib/dpkg/status

Version 50 is rolled out for your system, but still at 50% phasing, and version 1 at 10% phasing is disabled (pinned down to 1) so won’t install automatically.

For an organization? I mean maybe, they could do test roll outs to like 10% of their machines of SRUs, but that would require rewriting repos, which is probably not so nice.

#13

Hello,
I’m a Ubuntu end-user, and sysadmin for a few hundred servers and desktops. The concept of phased upgrades was not on my radar until reading the 21.04 beta release notes.
It would be nice if there were more practical explanations of the impact of this change, and how to tune it in APT configs. Particularly in the wiki page https://wiki.ubuntu.com/PhasedUpdates, as that seems to be where the APT 2.2 release notes blog post (https://blog.jak-linux.org/2021/02/18/apt-2.2/), points to.

I’d be keen to better understand exactly how this could impact servers, so that I can consider what, if any changes need to be proposed to the predominant APT management role in Ansible: weareinteractive.apt.

Thank you.

2 Likes
#14

Please read the top post, it explains the change in detail, and how you can configure it. I don’t see what to possibly add to that. Feel free to update the wiki page.

#15

I had a less productive response, but I’ve removed that, and I’ll just say this:
As a paying customer coming from the RHEL world - I’m asserting that the current coverage of this feature/change feels incomplete and unclear.

Please try to step outside the internal view you have from being intimately aware of the mechanics and function of APT phasing, and document it as though you are trying to explain the change to an ops guy.

If that doesn’t make sense, run it by Canonical’s documentation team, assuming they have one. There is a general lack of completeness and clarity that pervades Canonical documentation and the difference is really stark when coming to Ubuntu from Enterprise Linux.

$0.02 from a paying UA customer.

1 Like
#16

Your original comment said you wanted explanation of how to tune the behavior in apt configs, which is definitely covered in the top post. Do you have specific questions about what’s written there?

I agree that this post doesn’t currently explain what phased updates ARE, which was also one of your concerns.

You describe the wiki page as “stale”, but it hasn’t changed because neither have the mechanics or definition of phased updates. It does contain a lot of information that is irrelevant and distracting to end users, because it is a developer spec, not end-user documentation. I agree that we should have some more user-facing documentation of the nature of this feature.

2 Likes
#17

I’m sorry I don’t understand what you want. The post describes all the details relevant to end users / system administratrors.

It started with “what changed”, then “what are phased updates even”, then “how can I control that”, then “what if i configured update-manager already”, and then “advanced caveats for people working with chroots”.

What it specifically does not include is details as to how phased updates work, because that’s not relevant for people not writing package managers. It’s documented in the historical spec on the wiki, and of course in the DebianRepository/Format specification in the Debian wiki. But this seems completely out of scope for user-facing communication.

If you have a specific question, please ask it. As far as I can tell, everything you talked about is addressed in the initial post in as much detail as is relevant for end users.

1 Like
#18

We became aware that some packages currently or in the past days failed to install due to this feature. For example, if you had ubuntu-release-upgrader-core installed, and wanted to install ubuntu-release-upgrader-gtk, the phasing was only applied to ubuntu-release-upgrader core, and hence it tried to install ubuntu-release-upgrader-gtk 21.04.11, but ubuntu-release-upgrader-core 21.04.10.

There will be a stable release update shortly to fix those issues by applying phasing to not-yet-installed packages as well.

2 Likes