Possible memory leak in LXD 5.21

LXD
1

It looks like the LXD 5.21 has a memory leak as a result of executing LXC commands.
I’ve made the following simple test:

  • LXD with just one running container
  • I run a script that executes ‘lxc exec – ls >/dev/null’ each 2 sec
  • another script that monitors the RSS values in the /proc/<LXD_PID>/status file.
    After one day of running:
*****Tue Oct  8 03:18:44 PM UTC 2024
VmRSS:    173892 kB
RssAnon:          125636 kB
RssFile:           48256 kB
RssShmem:              0 kB
.........
*****Wed Oct  9 01:29:09 PM UTC 2024
VmRSS:    206708 kB
RssAnon:          157940 kB
RssFile:           48768 kB
RssShmem:              0 kB

We are using LXD as a part of our product and we have a service that periodically scans the containers and pulls the required data from them by running the LXC commands. The typical server has up to 150 containers. So, the memory leak becomes to be a “time bomb”.
The same problem I saw in LXD 3.0.3 in Ubuntu 18.04. As a workaround, I have to periodically restart the LXD service to reclaim the memory back.
Now with 5.21, the LXD service restarting causes the containers to restart and we can not afford it.
Does somebody know if the problem exists also in 6.x version?
Is it possible to install the old LXD 3.0.3 or 3.0.4 on Ubuntu 24.04?
Is there an alternative way to execute a command inside a container and get the result without involving LXD?
Thank you.

2

This is incorrect, which command are you using to restart LXD?

See https://documentation.ubuntu.com/lxd/en/latest/howto/snap/#start-and-stop-the-daemon

If you want to keep the instances running, reload the daemon instead:

sudo systemctl reload snap.lxd.daemon

3

Please can you try this:

lxc config set core.debug_address 127.0.0.1:8444
curl http://127.0.0.1:8444/debug/pprof/goroutine?debug=2

And capture the go routines just after reload and then again after the memory has increased to see if we are leaking go routines somehow.

4

Thank you very much!
You are right, ‘reload’ doesn’t restart the containers. I didn’t know it.

1 Like
5

I will send the results tomorrow.

1 Like
6

Hi @tomp
I see the same goroutine number before and after the test:
root@mt24-1:~# grep goroutine ./gorutines-before |wc -l
55
root@mt24-1:~# grep goroutine ./gorutines-after |wc -l
55
Therefore, there is no go routine leak.

1 Like