[Spec] APT deb822 sources by default

Index FO066
Title APT deb822 sources by default
Status Draft
Authors @juliank
Type Standard
Created 2021-05-13

Abstract

APT’s deb822 sources format is an alternative, often shorter, representation of sources.list that also interacts better with new features in apt. We need to port tools to understand it and then switch the default.

Rationale

APT introduced deb822 .sources file as an alternative to sources.list files a couple of years ago. The new format is more succinct and is substantially easier to read, especially when options come into play. To cite the sources.list manual page:

The format for two one-line-style entries using the deb and deb-src types is:

deb [ option1=value1 option2=value2 ] uri suite [component1] [component2] [...]

deb-src [ option1=value1 option2=value2 ] uri suite [component1] [component2] [...]

Alternatively the equivalent entry in deb822 style looks like this:

Types: deb deb-src
URIs: uri
Suites: suite
Components: [component1] [component2] [...]
option1: value1
option2: value2

Most important in terms of options, the Signed-By option allows the specification of a keyring that can sign the repository. We would like to migrate to requiring Signed-By for all sources, and in the old syntax this becomes unwieldy.

Recently APT gained support for specifying signing keys directly in the Signed-By option, allowing third-party repositories to be delivered as a single .sources file that just needs to be dropped into sources.list.d as opposed to a sources.list entry and a separate keyring file that needs to be managed.

This is a two cycle process: In the first cycle we need to prepare the foundations for supporting it, and in the second cycle we need to switch the default.

Specification

1. Support for deb822 format in remaining software in 22.10

Support for deb822 format needs to be implemented in python-apt’s aptsources module, which requires breaking the API, and then porting the existing code over. To avoid a flag day, it is proposed to adopt a new apt.sources module that extends the support from sources.list files to .sources files, then port software in the archive, such as

  • software-properties (critical)
  • Ubuntu-release-upgrader (critical)
  • command-not-found (critical)
  • aptdaemon (or kill the feature, do we even use it?)
  • ansible (not critical)
  • apt-clone (not critical)
  • apturl (not critical)
  • Curtin (can be done in 23.04 likely)
  • Cloud-init (can be done in 23.04 likely)
  • … missing …

and finally mark aptsources as deprecated.

The porting essentially revolves around having multiple URIs and Suites instead of singular.

Further software may need to gain support for the format that is yet unknown and might only be discovered once the default has been switched. People working on this need to run with the new default sources list documented in 2.

2. Replacing the default /etc/apt/sources.list in 23.04 (partially 22.10)

The default default sources list should be /etc/apt/sources.list.d/ubuntu.sources instead of /etc/apt/sources.list. A default ubuntu.sources matching the current sources.list is shown in appendix A. Note that it already includes Signed-By. add-apt-repository / software-properties should start writing deb822 at the same time.

The caveat with moving to a sources file like that is that it appears in a different order relative to other files in sources.list.d. This will affect the order in which apt update downloads files, and third-party repositories with an earlier name in the alphabet will take precedence if they ship the same version of a package (identical debs), but that is a minor cosmetic concern. There is no precedence for enumerating files in that directory, so going with 00ubuntu.sources or similar would be a larger diversion from the common naming schemes.

The rollout should happen in a staged fashion, with a canary MVP in 22.10 on a product that impacts fewer people. The idea is to switch Docker images for 22.10, and then switch the rest in 23.04. To ensure broader testing of the change, we should also put out a call for testing in 22.10. We will work with Debian to implement the same approach there, starting with the debian:sid docker image once the software is ported to gain more community testing.

3. Upgrades to 23.04

On upgrades to 23.04 and later versions, ubuntu-release-upgrader should remove Ubuntu entries from sources.list and write them /etc/apt/sources.list.d/ubuntu.sources, adding Signed-By fields to them in the process.

The ubuntu-release-upgrader tarball will have to vendorize the apt.sources module for this to work when upgrading from 22.04 to 24.04, or alternatively, the new module needs to be ported back to 22.04.

Further Information

Outlook: Mandatory Signed-By

In an interim release (24.10) following the first LTS with deb822 sources by default (24.04), APT will start issuing warnings about sources that do not have Signed-By set. After an LTS with warnings for missing Signed-By fields, APT will start enforcing Signed-By for all sources, and trusted.gpg.d becomes obsolete.

Due to ubuntu-release-upgrader adding the Signed-By field when migrating the main list to deb822, and it disabling third-party sources, users should not face any errors, but it might be worthwhile figuring out which key signed which repository, and then rewriting the sources.list files despite commenting them out, but this only concerns 24.10 or so.

Add-apt-repository needs to be changed so it writes the key into the Signed-By field of the sources file directly instead of creating a key in trusted.gpg.d.

Appendix A: Default sources.list

a+   buffers
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.

## Ubuntu distribution repository
##
## The following settings can be tweaked to configure which packages to use from Ubuntu.
## Mirror your choices (except for URIs and Suites) in the security section below to
## ensure timely security updates.
## 
## Types: Append deb-src to enable the fetching of source package.
## URIs: A URL to the repository (you may add multiple URLs)
## Suites: The following additional suites can be configured
##   <name>-updates   - Major bug fix updates produced after the final release of the
##                      distribution.
##   <name>-backports - software from this repository may not have been tested as
##                      extensively as that contained in the main release, although it includes
##                      newer versions of some applications which may provide useful features.
##                      Also, please note that software in backports WILL NOT receive any review
##                      or updates from the Ubuntu security team.
## Components: Aside from main, the following components can be added to the list
##   restricted  - Software that may not be under a free license, or protected by patents.
##   universe    - Community maintained packages. Software from this repository is 
##                 ENTIRELY UNSUPPORTED by the Ubuntu team. Also, please note
##                 that software in universe WILL NOT receive any
##                 review or updates from the Ubuntu security team.
##   multiverse  - Community maintained of restricted. Software from this repository is
##                 ENTIRELY UNSUPPORTED by the Ubuntu team, and may not be under a free 
##                 licence. Please satisfy yourself as to your rights to use the software.
##                 Also, please note that software in multiverse WILL NOT receive any 
##                 review or updates from the Ubuntu security team.
##
## See the sources.list(5) manual page for further settings.
Types: deb
URIs: http://archive.ubuntu.com/ubuntu
Suites: kinetic kinetic-updates
Components: main universe
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

## Ubuntu security updates. Aside from URIs and Suites,
## this should mirror your choices in the previous section.
Types: deb
URIs: http://security.ubuntu.com/ubuntu
Suites: kinetic-security
Components: main universe
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

One idea that came up is to also represent deb822 sources to legacy aptsources users by encoding the plural cases into a singular string. e.g. where it says Types: deb deb-src you could encode that as type="deb, deb-src" perhaps, and most stuff would be happy?

But I am not sure if this would actually makes things better or worse.