About debuginfod

Note:
This documentation has moved to a new home! Please update your bookmarks to the new URL for the up-to-date version of this page.

debuginfod is a service for software developers. It helps with diagnosing issues in software and centralises the storage of debug symbols, source code, etc.

One of the main advantages to debuginfod is that debugging information can be retrieved on-demand for packages shipped with Ubuntu without the need to manually install the debug symbol packages.

Ubuntu maintains its own debuginfod service, which regularly indexes the debug symbols present in ddebs and other packages and serves this information over HTTPS.

Currently, the service only provides DWARF information. There are plans for it to also index and serve source-code in the future.

Using the service

debuginfod is indexing ddebs packages from all supported Ubuntu releases. Once a release goes unsupported, we stop indexing ddebs from it and eventually stop serving debug symbols for its packages.

From Kinetic onwards, when you install GNU Debugger (GDB) your system will be automatically configured to use Ubuntu’s debuginfod service. For previous Ubuntu releases, you can manually enable the service by setting the DEBUGINFOD_URLS environment variable in your shell. If you use Bash, you can do that by adding the following snippet to your ~/.bashrc:

export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com"

When you run GDB, and if you have the DEBUGINFOD_URLS variable in your environment, you will be asked whether you would like to use the service. If you want to make sure that GDB always uses debuginfod, you can put the following snippet inside your ~/.gdbinit file:

set debuginfod enabled on

The debug symbol files will be downloaded on-the-fly during your debugging session, and will be saved locally inside the $XDG_CACHE_HOME/.debuginfod_client/ directory. If $XDG_CACHE_HOME is empty, then ~/.cache/debuginfod_client is used instead.

You can safely remove this directory or any files inside it; they will only be downloaded again if and when they are needed.

Example session with GDB

If you have enabled use of debuginfod on your system, here is what happens when you invoke GDB to debug a binary from an Ubuntu package:

$ gdb -q program
Reading symbols from program...

This GDB supports auto-downloading debuginfo from the following URLs:
https://debuginfod.ubuntu.com 
Enable debuginfod for this session? (y or [n]) 

When you answer y to the question above, GDB will download the debug symbols for program:

Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Downloading 0.20 MB separate debug info for /home/ubuntu/program
Reading symbols from /home/ubuntu/.cache/debuginfod_client/c0fbda15a807f880e9d0b2dcc635eeeb1f0f728e/debuginfo...                                                                           
(gdb) 

Opting out of the service

If, for some reason, you prefer not to use the service, you can opt-out of it by un-setting the DEBUGINFOD_URLS environment variable. This can be done by putting the following snippet inside your shell’s configuration file:

unset DEBUGINFOD_URLS

You can also disable GDB’s willingness to use debuginfod by putting the following snippet inside your ~/.gdbinit:

set debuginfod enabled off

How does debuginfod find the debug symbols for the binary I am debugging?

debuginfod relies on a unique hash that identifies binaries and shared libraries called Build-ID. This 160-bit SHA-1 hash is generated by the compiler, and can be consulted using tools like readelf:

$ readelf -n /usr/bin/bash

Displaying notes found in: .note.gnu.property
  Owner                Data size        Description
  GNU                  0x00000020       NT_GNU_PROPERTY_TYPE_0
      Properties: x86 feature: IBT, SHSTK
        x86 ISA needed: x86-64-baseline

Displaying notes found in: .note.gnu.build-id
  Owner                Data size        Description
  GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 3e770d2cd0302c6ff2a184e8d2bf4ec98cfcded4

Displaying notes found in: .note.ABI-tag
  Owner                Data size        Description
  GNU                  0x00000010       NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 3.2.0

When you are debugging a program, GDB will send the program’s Build-ID to the debuginfod server, which will check if it has the corresponding debug information for that binary/library. If it does, then it will send the debug symbols via HTTPS back to GDB.

Can ddebs packages co-exist with debuginfod?

Yes. GDB will try to use local debug information if available. That means that if you have a ddeb package installed that provides the necessary debug symbols for the program being debugged (or if you have already downloaded that information from the debuginfod service earlier), then GDB will use it in favour of performing the download.

Can I use debuginfod with my own binary that links against system libraries?

Yes! debuginfod will not be able to provide any debug symbols for your own program, but it will happily serve debug information for any system libraries that your program links against.

3 Likes

Maybe link “manually install” to Debug Symbol Packages - Ubuntu Wiki, if that page is still up-to-date.

Can you show an example of debuginfod in action, on the client side?

Also, does each invocation of gdb trigger a download, or are those symbols installed in some cache directory? Or are the actual ddebs installed, as if I followed https://wiki.ubuntu.com/Debug%20Symbol%20Packages and installed them manually?

Good idea, thanks. Link done.

I have expanded the article to better explain these points. Please take a look and let me know what you think. Thanks.

1 Like

I don’t think you need to repeat that.

I didn’t like repeating it, but I also thought it’d be worth mentioning again. Anyway, I’ve removed the paragraph.

Nice!
Other questions that come to mind now, which you can decide if are worth expanding in the docs or not (after all, this is not a tutorial):

  • if the binary is updated (assuming it came from a deb), gdb will me smart enough to download a new set of symbols?
  • will that local cache just keep growing? Any tips on pruning it? Can it be pruned per-package? (i.e., "please remove debuginfod info for package libreoffice only). I’m making a comparison with ddebs, which I can remove individually.
  • can ddebs coexist with the data downloaded by debuginfod? Or should I remove the ddeb lines from sources.list? Is there a fallback ordering of some sort?
  • how tightly is this coupled with deb packaging? Can I use gdb against something I coded and built locally (i.e., hello_world.c), which uses some system libraries, and have gdb download debuginfod data for those system libraries?
1 Like

Thanks for the careful review!

I would like to keep this page simple because it will be the landing page for https://debuginfod.ubuntu.com, but I also understand that there might be other details about the service that would be good to cover.

I think it may be worth having a separate FAQ page that’s not directly part of the Server Guide but is linked from this page, WDYT? Then I can not only go over more details about the service, but also allow the users to ask their questions in a single place.

A FAQ is fine, elsewhere here on Discourse, or even in the Server Guide as another page/section, I wouldn’t mind.

1 Like
  • if the binary is updated, its buildid will change, thus a new debuginfo file would be sought
  • the local cache is aged periodically, so that old unaccessed files are deleted, see [man debuginfod-client-config(7)]
  • by the way, for running locally built binaries that have stripped debuginfo, you could run a personal debuginfod instance whereever the complete unstripped binaries may still be found. Clients can be configured to ask multiple servers. (Servers can also federate to each other, as each server can also be a client.)

There is additional information that can be supplemented here: “debuginfod is indexing ddebs packages from all supported Ubuntu releases. Once a release goes unsupported, we stop indexing ddebs from it and eventually stop serving debug symbols for its packages.”

For example, the situation of ddeb support for different releases and respective suites

  • Suites from regular source: noble, noble-updates, noble-backports, noble-proposed
  • Suites from security source: noble-security
  • Suites from esm-app source: noble-apps-security, noble-apps-updates
  • Suites from esm-infra source: noble-infra-security, noble-infra-updates

I have noticed that the packages from noble-security cannot fetch dbg-sym information.
For instance, in Ubuntu 24.04, openssh-server:

buildid of sshd
c6b029db53c7dd5f3afc7909d5fea694851055bf

gdb sshd output as below:

Reading symbols from sshd...
Reading symhols from /home/user/.cache/debuginfod_client/5b1aebbe7e....
(gdb)

Packages in other suites have corresponding debug info.