Building custom LXD binaries for side loading into an existing snap installation

This can be useful if you want to test a feature in a specific environment you already have set up, like a cluster or one with multiple projects, profiles, …, etc. You can place your newly compiled binaries in that environment as .debug files, restart the LXD daemon for the system to pick up these special binaries, test and then simply erase the .debug files and restart the daemon to revert back to normal.

Note that if you test a code change that modifies your database, you will have to revert your database manually when you go back to the previous state or wait for that feature to make it to the official store before you can remove your .debug files.

1. Use a builder container with the same Ubuntu version we are using to create official releases.

This will ensure you have the same dependencies as those used in the published snaps. The table below lists the base containers we use in all channels.

Snap Channel Base Container
latest/edge 22.04
latest/candidate 22.04
latest/stable 22.04
5.0/stable 20.04

Here my container is called “builder”:

lxc launch ubuntu:22.04 builder
lxc shell builder

2. Inside the container, from instructions provided here

apt update
apt -y upgrade

Then install the required packages except golang which we will install as a snap later.

apt -y install acl attr autoconf automake dnsmasq-base git libacl1-dev libcap-dev liblxc1 liblxc-dev libsqlite3-dev libtool libudev-dev liblz4-dev libuv1-dev make pkg-config rsync squashfs-tools tar tcl xz-utils ebtables
apt -y install lvm2 thin-provisioning-tools
apt -y install btrfs-progs
apt -y install curl gettext jq sqlite3 socat bind9-dnsutils
apt -y install shellcheck

3. Install a recent Go environment

apt -y install snapd
snap install go --classic

4. Get the source code

git clone https://github.com/canonical/lxd

5. Build

cd lxd

Build raft and dqlite

make deps

As per make’s instructions

export CGO_CFLAGS="-I/root/go/deps/raft/include/ -I/root/go/deps/dqlite/include/"
export CGO_LDFLAGS="-L/root/go/deps/raft/.libs -L/root/go/deps/dqlite/.libs/"
export LD_LIBRARY_PATH="/root/go/deps/raft/.libs/:/root/go/deps/dqlite/.libs/"
export CGO_LDFLAGS_ALLOW="(-Wl,-wrap,pthread_create)|(-Wl,-z,now)"

Build lxd, lxc, … etc

make
exit

6. Install

Get binaries from your builder and add .debug extension

lxc file pull builder/root/go/bin/lxd lxd.debug
lxc file pull builder/root/go/bin/lxc lxc.debug

Push binaries in your current working environment

sudo mv lx[cd].debug /var/snap/lxd/common

Restart LXD daemon

sudo systemctl reload snap.lxd.daemon

Check using top that lxd.debug is running

7. Revert

sudo rm /var/snap/lxd/common/lx[cd].debug
sudo systemctl reload snap.lxd.daemon

Check using top that the regular lxd is running

5 Likes