`Too many open files` when running `find` on a disk mounted from host

[danielarndt]

Perhaps this is more of a LXC issue than an LXD one. Let me know if I should ask over there instead.

I’m not quite sure which of the many limits I might be brushing up against here, or why the open files are not being released.

All I’ve done is mount a disk from my host into a VM and mapped my host user ID to the guest user ID:

$ lxc version   
Client version: 5.21.2 LTS
Server version: 5.21.2 LTS
$ lxc config device add dev canonical disk source=/home/darndt/git/canonical/ path=/home/dev/canonical
$ lxc config set dev raw.idmap "both 1000 1002"

Then I’m trying to run this command (in the VM):

find "${HOME}/canonical" -name .git -prune

This works fine for a while, but eventually it fails with Too many open files.

After this, I can’t do anything on the mounted filesystem.

$ cd canonical
$ ls                             
ls: reading directory '.': Too many open files

I can still read/write outside of the mounted disk.

$ cd ..
$ ls
canonical  Downloads  snap  vm-share

What I’ve tried

(In the VM):

lsof didn’t indicate anything weird (to my eyes). I didn’t see large amounts of files remaining opened by the find command.

ulimit -n 2048

sudo sysctl -w fs.inotify.max_user_instances=2048

and adding

dev hard nofile 65536

to the /etc/security/limits.conf file without any luck.

Maybe the limit is coming from the host, but in any case I can’t figure out:

1. How to increase this limit
2. How to ensure these open files are eventually released so that this doesn’t become some sort of “maximum amount of files ever opened” limit.

I’m not very familiar with LXD/LXC, but even nudging me in the right direction would help a lot.

[pedro-rib]

Hi! Sorry for the late response, but perhaps we can still shed some light on this. I tried to reproduce this a couple of times on my end without success, could you try to supply more information on your setup?

Particularly, I think it would be useful to know what users are you using for the operations and some idea of what is inside the shared directory.

[danielarndt]

Hi Pedro,

I am using a non-root user with sudo privileges if that’s what you mean. In the past when this worked I was using root, but I’ve just tried again as root and the same issue persists. I’m not sure what else might have changed.

The directory itself just contains all the git repos I’ve checked out – nothing crazy. However, it is somewhat large (1,909,849 items, totalling 26.4 GB).

[danielarndt]

It is perhaps also worth mentioning that the drive containing the VM and the drive that is being mounted are different physical disks.

[pedro-rib]

Cool, thanks for the additional infomation. I will make some attempts here to reproduce it.

In the meantime, this one liner helps to see the processes with the most files open: sudo lsof | head -1 | awk '{ print "COUNT " $2 " " $1; }' && sudo lsof +c 0 | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20. You can try that on the host as well

[tomp]

Is this instance a VM or a container btw (I note you’ve said VM a few times, but sometimes folks use VM when they mean container :))

[danielarndt]

Fair question, as I was tricked into thinking a container was a VM just last week. This is a VM.

However, I’ve just tested this with the latest version of LXD 5.21.3 (although, I’m not sure that was the root cause) and this issue no longer persists.