Easy multi-user LXD setup

Project Discussion Desktop
1

One nifty new feature of the newly released LXD 4.22 is the ability for regular users to safely interact with LXD.

Until now, LXD has suffered from much the same issue as the likes of Docker and Libvirt in indirectly granting full root access to anyone who’s allowed to interact with it. This was possible through a number of different options (device passthrough, privileged container, custom ID maps, …). While not a problem on a developer’s laptop, it’s a no-go for most shared environments, especially in a corporate setting.

Over the years, LXD has grown support for proper remote authentication, fancy access control (RBAC) and projects as ways to restrict specific users to a subset of LXD’s features so multiple people can safely share a LXD server or cluster.

With LXD 4.22, this is now all doable locally with normal local users making it a great fit for desktop systems, especially multi-users desktop in a corporate environment.

Here is an overview of the feature from our release live stream:
https://www.youtube.com/watch?v=Blx7cdygiS8&t=848s

For those who aren’t into video watching, the instructions basically are:

snap install lxd
snap set lxd daemon.user.group=users

With that done, any user in the users group will be allowed to interact with LXD despite not being in the all-powerful lxd group.

The first time one such user interacts with LXD, they will automatically get their own restricted project which will look like this:

foo@v1:~$ lxc project list
+---------------------+--------+----------+-----------------+----------+------------------------------------------+---------+
|        NAME         | IMAGES | PROFILES | STORAGE VOLUMES | NETWORKS |               DESCRIPTION                | USED BY |
+---------------------+--------+----------+-----------------+----------+------------------------------------------+---------+
| user-1001 (current) | YES    | YES      | YES             | NO       | User restricted project for "foo" (1001) | 3       |
+---------------------+--------+----------+-----------------+----------+------------------------------------------+---------+

foo@v1:~$ lxc project show user-1001
config:
  features.images: "true"
  features.networks: "false"
  features.profiles: "true"
  features.storage.volumes: "true"
  restricted: "true"
  restricted.containers.nesting: allow
  restricted.devices.disk: allow
  restricted.devices.disk.paths: /home/foo
  restricted.devices.gpu: allow
  restricted.idmap.gid: "1003"
  restricted.idmap.uid: "1001"
description: User restricted project for "foo" (1001)
name: user-1001
used_by:
- /1.0/instances/welcome-earwig?project=user-1001
- /1.0/images/ced57a80f2b761c3cdab867c2296b801c6adfe521f811bacdd61410da4bc2734?project=user-1001
- /1.0/profiles/default?project=user-1001

Which in practice allows the user to just do:

foo@v1:~$ lxc launch ubuntu:20.04
Creating the instance
Instance name is: welcome-earwig        
Starting welcome-earwig

foo@v1:~$ lxc list
+----------------+---------+---------------------+-----------------------------------------------+-----------+-----------+
|      NAME      |  STATE  |        IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+----------------+---------+---------------------+-----------------------------------------------+-----------+-----------+
| welcome-earwig | RUNNING | 10.31.36.109 (eth0) | fd42:fa4a:d38d:1c7b:216:3eff:fed1:63aa (eth0) | CONTAINER | 0         |
+----------------+---------+---------------------+-----------------------------------------------+-----------+-----------+

And get a container, or virtual machine, running immediately, with no configuration and without ever having needed any kind of elevated privileges.

Should that user try to create a privileged container, pass in paths outside of their home directory or do any device passthrough other than GPUs, it will be rejected by LXD.

Login as another user on the system and you’ll get the exact same behavior, every user gets their own personal project and can’t see the others. Well, unless they are part of the lxd group, then they can see everything going on on the system.

We’re hoping that this feature can be used to provide experiences similar to that of WSL on Windows or Crostini on ChromeOS where getting containers going is just a few clicks away and the main system remains nice and safe.

Enjoy!

6 Likes
2

This is awesome :slight_smile: Is there an “easy” way to migrate an existing LXD install to this new less-privileged scenario?

Currently my user is in the lxd group and I have a bunch of (unprivileged) containers and VMs running under the default project - can I easily export my containers / VMs, remove myself from the lxd group and then set those back up under a restricted personal project for my user?

3

This is possible if slightly more involved :slight_smile:

I’ve tried it here, starting with:

ubuntu@u2004-desktop:~$ id
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),44(video),100(users),998(lxd)
ubuntu@u2004-desktop:~$ lxc project list
+-------------------+--------+----------+-----------------+----------+---------------------+---------+
|       NAME        | IMAGES | PROFILES | STORAGE VOLUMES | NETWORKS |     DESCRIPTION     | USED BY |
+-------------------+--------+----------+-----------------+----------+---------------------+---------+
| default (current) | YES    | YES      | YES             | YES      | Default LXD project | 5       |
+-------------------+--------+----------+-----------------+----------+---------------------+---------+
ubuntu@u2004-desktop:~$ lxc list
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| NAME |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| a1   | RUNNING | 10.225.65.247 (eth0) | fd42:2c06:5b38:f668:216:3eff:fe38:f20 (eth0)  | CONTAINER | 0         |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| a2   | RUNNING | 10.225.65.206 (eth0) | fd42:2c06:5b38:f668:216:3eff:fefa:2294 (eth0) | CONTAINER | 0         |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
ubuntu@u2004-desktop:~$

I’ve then removed my user from the lxd group and logged in again:

ubuntu@u2004-desktop:~$ id
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),44(video),100(users)
ubuntu@u2004-desktop:~$ lxc list
Error: not authorized
ubuntu@u2004-desktop:~$ lxc project list
+-----------+--------+----------+-----------------+----------+---------------------------------------------+---------+
|   NAME    | IMAGES | PROFILES | STORAGE VOLUMES | NETWORKS |                 DESCRIPTION                 | USED BY |
+-----------+--------+----------+-----------------+----------+---------------------------------------------+---------+
| user-1000 | YES    | YES      | YES             | NO       | User restricted project for "ubuntu" (1000) | 1       |
+-----------+--------+----------+-----------------+----------+---------------------------------------------+---------+
ubuntu@u2004-desktop:~$ lxc project switch user-1000
ubuntu@u2004-desktop:~$ lxc list
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+
ubuntu@u2004-desktop:~$

So the user is now restricted to an empty project. Next we need to move the instances from a privileged user:

root@u2004-desktop:~# lxc list
To start your first container, try: lxc launch ubuntu:20.04
Or for a virtual machine: lxc launch ubuntu:20.04 --vm

+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| NAME |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| a1   | RUNNING | 10.225.65.247 (eth0) | fd42:2c06:5b38:f668:216:3eff:fe38:f20 (eth0)  | CONTAINER | 0         |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| a2   | RUNNING | 10.225.65.206 (eth0) | fd42:2c06:5b38:f668:216:3eff:fefa:2294 (eth0) | CONTAINER | 0         |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
root@u2004-desktop:~# lxc stop --all
root@u2004-desktop:~# lxc move a1 --target-project user-1000
root@u2004-desktop:~# lxc move a2 --target-project user-1000
root@u2004-desktop:~# lxc list
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+

And after that, starting everything back as the user:

ubuntu@u2004-desktop:~$ lxc list
+------+---------+------+------+-----------+-----------+
| NAME |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+------+---------+------+------+-----------+-----------+
| a1   | STOPPED |      |      | CONTAINER | 0         |
+------+---------+------+------+-----------+-----------+
| a2   | STOPPED |      |      | CONTAINER | 0         |
+------+---------+------+------+-----------+-----------+
ubuntu@u2004-desktop:~$ lxc start --all
ubuntu@u2004-desktop:~$ lxc list
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| NAME |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| a1   | RUNNING | 10.225.65.247 (eth0) | fd42:2c06:5b38:f668:216:3eff:fe38:f20 (eth0)  | CONTAINER | 0         |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| a2   | RUNNING | 10.225.65.206 (eth0) | fd42:2c06:5b38:f668:216:3eff:fefa:2294 (eth0) | CONTAINER | 0         |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
ubuntu@u2004-desktop:~$

Note that the lxc move step requires that any project or custom volume you need exist in the target project, so if you had anything fancy on that front, you’ll want to re-create your profiles and move any custom storage volumes ahead of time.

If your instances use features which aren’t allowed under a restricted project, the lxc move will fail with Forbidden, normally telling you what’s being rejected. You can either modify the instance to comply with the restrictions or use lxc project edit user-1000 to alter the restrictions to allow what you need.

1 Like
4

Thanks @stgraber - that worked like a charm :slight_smile:

For posterity, it was as simple as:

# remove my user from the lxd group
sudo gpasswd -d $USER lxd

# reboot to cleanly logout and back in again without being in
# the lxd group anymore

# set the lxd user group to just my user's primary group since 
# there is no other  users on my local machine
sudo snap set lxd daemon.user.group=$USER

# lxc will have created a new project called user-$UID for your user
lxc project list
# set this as the default project
lxc project switch user-$UID

# then migrate instances to this user's project by stopping them all
# and then moving them across
sudo lxc stop --all
# moving may take a while depending on how many containers/vms you have
for instance in $(sudo lxc list --format json | jq ".[].name" -r); do
  sudo lxc move $instance --target-project user-$UID
done

# then I can use those instances from my local user as before
3 Likes
5

I have tried it with local users and it works fine

But if I try with AD users (joined via SSSD), I get the following error

Error: Get "http://unix.socket/1.0": read unix @->/var/snap/lxd/common/lxd-user/unix.socket: read: connection reset by peer

The domain user is member of the local group “users”

6

We merged a fix for this. It will be in LXD 5.3 for sure (and in 5.0.1 LTS) but may also make it as a hotfix in a future update of LXD 5.2 (we’re collecting some other fixes for it now).

1 Like
7

Thanks for this great feature!

I’ve been playing around with it a bit and it seems like daemon.user.group has to be a local group. I’ve tried to use a LDAP group but then get this error:

Error: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied

Is this some sort of limitation related to name spaces?

After changing this parameter to a local group, it works as intended with a new user. But trying to use the user account I used first (when daemon.user.group was set to a LDAP group) I then get this error instead:

Error: not authorized

Any idea how to recover from that for this user account? I tried deleting the project, hoping that it would get recreated when running lxc list, but that didn’t happen.

8

So indeed the snap environment doesn’t have access to the host NSS stack, meaning it can’t resolve group names that come from a remote system like LDAP.

I would have expected group numbers to work though as all we really do is feed that to a chgrp call.
But there’s been some other users suggesting this may not work either? Would be great to confirm though.

Now for the second issue. What happened is that the LXD client wasn’t able to talk to lxd-user and so initialized a local configuration of its own. This most likely means it’s trying to access the default project rather than the user-specific project.

To fix that, you can do lxc project list to see the list of allowed projects and then lxc project switch to have the client use the correct one.

9

Hi, thanks for this feature!

What is the equivalent of snap set lxd daemon.user.group=users for non-snap lxd installations?

10

Assuming your distro wrote a systemd unit for the lxd-user daemon along with a match systemd unit for socket activation, then the equivalent would be the SocketGroup field in that unit which controls who owns the lxd-user socket.

11

Thanks for the reply. Is there a config file somewhere that the daemon ingests in case I’m not using systemd?

Here’s a separate multiuser issue: on an ubuntu system that is using snap (and systemd), user directories are mounted at /home/$user via NFS with default root squashing. This seems to cause permission problems, as any sudo lxd command results in things like “cannot open path of the current working directory: Permission denied”. Any advice?

12

No, the daemon on its own doesn’t have any configuration.
You can do something like:

mkdir -p /tmp/lxd-user/
cd /tmp/lxd-user/
lxd-user

And the daemon will run on /tmp/lxd-user/unix.socket connecting to /var/lib/lxd/unix.socket by default.

13

Not that I can think of. The upside is that most commands don’t need to be run as root at all, even lxd init doesn’t need root privileges so long as you’re in the lxd group.

If you must run something as root, I’d suggest trying with sudo -i to get a shell as a full root user with the home directory set to /root, then run from that which should avoid the NFS permission issue.

14

Thanks, makes sense.

I get the permission issues even when running commands as a normal user. lxd init, lxc info etc.

Something must be misconfigured on this machine. With a root shell, lxc info and lxd version both return “internal error, please report: running “lxd” failed: timeout waiting for snap system profiles to get updated”

It looks like there are quite a number of issues with snap and having (root squashed) NFS-mounted homedirs. E.g. just sudo snap find lxd returns an error “error: cannot list snaps: cannot add authorization: open /home/myuser/.snap/auth.json: permission denied” and snap install asks me to use sudo…

15

Could be that snap-confine, the privileged wrapper which puts all the confinement of the snap in place may be hitting an issue with that home directory.

I don’t know that snap-confine itself needs to directly access anything inside of the user’s home directory, so it not being able to read or even traverse the home directory shouldn’t have to be a blocker, but it could be that its current design is running into problems here.

@mvo I know that home directories in unusual locations are causing problems, but in this case, we’re dealing with home directories at a usual location, just not root accessible. I believe the same can also happen with the likes of ecryptfs.

16

I understood that lxc-user is a separate “proxy” daemon with a socket that has different access permissions than the main lxd daemon. This is straight forward also for the non-snap user case.

But I didn’t understand how lxc (the client) does the lookup of which socket to connect to. Somehow the source code doesn’t have any reference to how it differentiates between regular /var/snap/lxd/common/lxd/unix.socket and the /var/snap/lxd/common/lxd-user/unix.socket.

Where would I need to place the lxd-user socket in the non-snap case so that lxc will find it? And how is this related to the $LXD_SOCKET environment variable? There is super minimal documentation around this topic.

17

Hey there,

The way this is done in the snap is that lxc in the snap is a wrapper which then basically checks if the user is allowed access to /var/snap/lxd/common/lxd/unix.socket, if they are, then LXD_DIR is set to /var/snap/lxd/common/lxd, if not, it checks if /var/snap/lxd/common/lxd-user/unix.socket exists, if it does, it sets LXD_DIR to /var/snap/lxd/common/lxd-user/.

The client always consults LXD_DIR in its environment to know where to find the default LXD unix socket, so that’s how we make it use lxd-user when available and when the user isn’t directly allowed to talk to LXD.

18

Someone used microsoft ldap authentication for lxd? Any suggestion about it?