[Spec] Include Performance tooling in Ubuntu

Index FO147
Title Include Performance tooling in Ubuntu
Status Pending Review
Author Matthieu Clemenceau, Christian Ehrhardt, Jon Seager, Robie Basak
Type Standard
Created 2023-12-13


As we raise the bar at Canonical on performance management and optimization we will be enabling frame pointers for Ubuntu 24.04 by default. To further enhance application developer experience while trying to find performance gain, we will include performance and profiling tooling by default on Ubuntu 24.04 (classic, not core) as well


Mark Shuttleworth and Jon Seager had a productive chat with Brendan Gregg talking about what we could do in Ubuntu 24.04 to make Ubuntu absolutely great for performance engineers. A key suggestion was to include this tooling by default.


Install Performance Engineering tools by default

A set of packages shall be installed and available by default on all relevant systems. These are as follows. Some of these packages are already seeded and for these packages implementation requires no changes, but they remain mentioned here so that this specification serves to provide an additional purpose for which they should remain seeded. For the purpose of this specification, all defaults in seeds shall be provided by Recommends, even if they are “upgraded” to Depends for other purposes in practice.

Package Responsible Team Seed
procps Foundations required (already seeded)
util-linux Foundations required (Essential: yes)
sysstat Server standard (already in Supported-Sysadmin-Server)
iproute2 Kernel minimal (already seeded)
numactl Server standard (already in Supported-Sysadmin-Server)
bpfcc-tools Foundations kernel-accessories (see US094 - Kernel Accessories Seed and Metapackage)
bpftrace Foundations kernel-accessories (see US094 - Kernel Accessories Seed and Metapackage)
trace-cmd Foundations standard
ethtool Server standard (already in server and Supported-Hardware-Common)

Make the linux-tools-$version package available by default

This package shall be added to the Recommends of the relevant kernel package using the linux-meta source package.

Provide a performance-tools metapackage

We shall add a new metapackage to universe called performance-tools.

Description: utilities for debugging performance and reliability issues
  This metapackage installs utilities for inspecting and debugging systems that are 
  experiencing performance and/or reliability issues.
  Included are more traditional performance analysis and inspection tools, along with
  dynamic tracing and instrumentation tools which make use of the eBPF subsystem.

  On Ubuntu, some of these packages are installed by default already, but are
  also included here to assist with deployments that do not use the defaults.

This metapackage shall depend on the following packages:

  • procps
  • sysstat
  • iproute2
  • numactl
  • bpfcc-tools
  • bpftrace
  • perf-tools-unstable
  • trace-cmd
  • nicstat
  • ethtool
  • tiptop
  • sysprof

Further Information

Related specifications that implementation of this specification is dependent upon:

Design Decisions and Rationale

Considering the purpose of the performance-tools metapackage:

  1. Since we already include performance tools that we can support in main, and arrange for them to be installed by default, this metapackage should be unnecessary in “usual” use cases.

  2. However, some packages do not meet our standards for main, or otherwise have not been included into main yet, yet are available in universe. Performance engineers might be expected to reference the book “Systems Performance” by Brendan Gregg and it is convenient for such engineers for there to be a metapackage that pulls in everything that we have available in the archive referred to from the book. We wouldn’t expect to do this for all books, but this is the relevant book in this area of expertise.

  3. Users who are using minimal images that do not install these packages by default (eg. cloud-minimal for OCI generation purposes) can easily install our recommended set.

The performance-tools metapackage is therefore provided in addition to, and independent of, the defaults provided through the seeds. It uses Depends rather than Recommends since the entire purpose of the metapackage is to install its dependencies, one would not install the metapackage without wanting its dependencies, and it’d be correct to say that the metapackage is not installed if the packages it “ships” are not installed. Users who would like a subset of the packages the metapackages provides always have the option to individually install those manually. The metapackage can be in universe because there is no requirement for it to be in main. This makes adding to it easier, since there is no MIR requirement.

The metapackage may duplicate packages that are already brought in by other seeds, whether directly or indirectly. But since one use case above means that a user may be preparing an image not driven by those seeds, we include them anyway. We make one exception: since “Essential: yes” packages are not permitted to be depended upon by policy, we exclude these to avoid breaking any existing assumptions in tooling.

linux-tools-$version, having the corresponding required kernel version as part of its package name, cannot be provided by default through seeds in the usual fashion. Therefore we use the linux-meta source package to do this instead.

Expected Negative Consequences

We expect the following negative consequences as a result of this change, but in examining the trade-off, consider that the benefits outweigh these.

Some packages will be installed by default that were not previously. The following are estimates for the upper bound of increase in image and installation size for all server and desktop images (excluding cloud-minimal, which will not be affected since they are out of scope of the change):

Package Image Size Increase Install Size Increase
sysstat 525 kB 1,640 kB
numactl 37.3 kB 136 kB
bpfcc-tools (images with kernels only) 43 MB 206 MB
bpftrace (images with kernels only) (depends on bpfcc-tools and the figures also include those of that dependency) 80.8 MB 367 MB
trace-cmd 469 kB 2,002 kB
ethtool 229 kB 752 kB

LXD images will now be differentiated from other image types. Whereas before they were identical except for the absence of kernel packages, now additionally the kernel-accessories metapackage will be installed by default everywhere except LXD images. This is intentional, to avoid increasing LXD images with unnecessary BPF-related packages. However it is additionally noted that for product purposes, this means that LXD images will be further differentiated than previously.


Increasing the install size my ~0.5GiB for all users for something only a very small handful of users use seems like a poor idea, considering that handful can very easily install those tools when needed. IMHO, bpfcc-tools and bpftrace should NOT be included in the default image, given their extremely large size and extremely specific use-cases.

However, including the other packages by default would be totally fine, considering the total size increase would only be around 4MiB.


I’ve seen various people misunderstanding the image size increase analysis. Sorry I wasn’t clear on that before - I think I misunderstood the audience that would be reading what I wrote.

  1. Please note that the 367 MB for bpftrace includes the 206 MB for bpfcc-tools since one is a dependency of the other. It is incorrect to add them together again.

  2. This is a worst case scenario analysis. In reality, different images types will increase in size by different amounts. The actual image size increase will be less if the image already includes the required dependencies. For example, the desktop install is likely to already include the most dependencies, so the increase in size is expected to be smaller there. I’ll see if I can come up with a more accurate figure specifically for the desktop install.


I have bpftools installed from Ubuntu packages and cloned from git repo twice on this very machine! But even I don’t want to see it in any base image. Brendan is not a typical user :wink:

There are things Ubuntu can do today. Can you make symbolization work please? Install debuginfod and make sure it works. Why do does perf tell me “WARNING: debuginfod support requested, but perf is not built with it”? And even that happens only for those savvy enough to add explicit “–debuginfod” argument. I can “apt install perf” (I mean “linux-tools-common”?) install in no time. Rebuilding it is hard and frustrating.

Debuginfod is great beyond profiling - debugging, core dumps… It is the kind of package I would like to see in base image. For me that would be worth the few hundred kB. Fedora has it out of the box even in live cd.

Thanks for the feedback, @jendap.

First of all, it’d be great to have debuginfod present in the image! I am “a bit” biased when it comes to it, of course. But just having GDB installed will automatically pull the necessary bits to make debuginfod work. The issue here is that one could argue that GDB isn’t really a tool aimed at performance engineering, and one wouldn’t be entirely wrong.

As for the integration between debuginfod and perf, it has been disabled because of licensing concerns. The Linux kernel is distributed under GPLv2 and debuginfod is licensed under GPLv3 (although I now see that it is dual-licensed under GPLv2 as well), so Debian chose to disable this integration on purpose, and Ubuntu follows suit. We’re not the only ones doing that, though: Fedora also disables the integration.

Sorry, I spoke too soon: it’s debuginfod-client that’s released under the dual-license scheme. libdebuginfod, which is the library perf needs to link against, is released under GPLv3+ only and therefore perf cannot use it.

Thank you for looking at profiling! It is great to see frame pointers enabled! That’s where distro can help a lot. It is also great to have the tools packaged even if they are not in base image. It would be nice if profiling tools (perf) would just work (like gdb does).

Would you mind asking debuginfod (elfutils?) to consider dual licensing? I can’t even create an issue (or I’m in a wrong bugzilla altogether).

BTW: Even Brendan says it is not the last word in stack unwinding / symbolization. Googlers could also chime in. Would it make sense to propose a new project? It would high impact project at the border of languages, compilers, data structures, server protocol, etc. It needs to be fast - Rust would be a good fit. Rustaceans are reinventing the infrastructure in all ecosystems - kernel, python, javascript, etc. It may be a nice project to help all ecosystems at once :slight_smile: