Ubuntu's virtualization HWE stack and LXD's snap

Project Discussion LXD
1

Canonical just announced the release of a virtualization HWE stack: Ubuntu’s virtualization hardware enablement (HWE) stack: a new model for confidential computing enablement | Ubuntu

That is great news! I personally run HWE kernels on all my LXD nodes for the latest support and features, and this new stack should further improve that!

But how will Canonical’s in-house virtualization solution LXD benefit from this new stack?

With LXD being confined to a snap, as far as I am aware the system’s packages are not used at all. Only the kernel is. qemu and edk2 binaries are included with the snap.

LXD started out as a management daemon for LXC containers. LXD stood for LXC on steroids, was the joke we used in our team. A breeze compared to setting LXCs up yourself on a plain debian system!

A few years ago, LXD introduced support for virtual machines. Starting with LTS 5.0 those became a first class citizen.

Today, with all the features that 6.9 offers us, LXD is becoming a full blown virtualization solution. A lightweight openstack is how I would describe it!

For a LXC management daemon, snap was fine. But when you start to look at LXD as a hypervisor, there are some limitations of lxd being confined to a snap that are in the way.

Compared to alternatives like Proxmox, OpenNebula, XCP-ng or Harvester there are some features not possible due to the limitations of a snap.

Basic features like:

  • Cockpit-like host management:
    • being able to power off or reboot a node via API/UI.
    • Host OS updates visibility
    • Host logs
  • Host metrics visibility (SMART / sensors / thermal)
  • Network configuration beyond LXD-managed bridges

Powerful and basic features that LXD’s competitors offer.

Almost all of our/my deployments rely on host OS managed bridges to directly connect to the physical VLANs, all the way back to 2017. Network management is consistently the thing people struggle with most when I suggest LXD to someone. Most people just want to bridge a VM directly onto the physical network, and not everyone has a spare NIC to dedicate to a managed LXD bridge. “Give my VM an IP on vlan 20” is what is expected by most.

I am not suggesting LXD immediately abandons the snap model, i get why it exists. There are also benefits to shipping binaries/libraries with your software (Hello docker!).

But I feel that LXD (and microovn, microceph) are fighting the limitations of snap. They’re not simple desktop applications like Spotify, Firefox, Discord, or server applications that work wonderfully in a docker container like nginx or mysql.

LXD (And the MicroCloud snaps) are becoming richer and richer in features. And to grow even further beyond I think the snap confinement needs to be discussed.

Curious to hear your thoughts!

1 Like
2

I have to agree. since the demise of vmware workstation. and the overall suckiness of anything oracle (virtualbox comes to mind). i have been dying to see lxd move into the vmware workstations’ void. speaking of lxd further…..when are we going to get sound in vm’s.

3

Isn’t VirtualBox quite a different type of beast to LXC? From what I understand, LXC is a container, whereas VirtualBox provides a completely self-contained virtual machine including virtual hardware and even a virtual TPM2 chip.

Each product has its strong points, and I use both VirtualBox and LXD/LXC for that reason, but they’re not interchangeable.

Personally, I don’t find VirtualBox “sucky”. I use it a tremendous amount, and it’s been brilliant for my purposes. It does things that LXC cannot do (and vice-versa).

4

it is true that virtual box has absorbed the least amount of oracles sucky than any of the other products they have subsumed. at least for now they are allowing virtual box to live. when they purchased vmware they immediately stopped all support for vmware workstation (the best hypervisor ever). before they decide to kill virtual box as well. I would like to see lxd often referred to as (lxc on steroids) get the vitamins and proteins. to be able to do everything both virtual box and vmware workstation, does or did. then canonical will not only have the best operating system, it will also have the best hypervisor for both containers and virtual machines.

5

It’s possible, but unlikely. Oracle releases updates frequently, and upgrades regularly. It is fully committed to VirtualBox.

Edit: Plus, Oracle makes money from VirtualBox.

I can’t talk about VMware because I’ve never used it, but it’s not easy for LXC to do what VirtualBox does, because they are different. LXC is a container. VirtualBox is a fully virtual machine with fully independent complete hardware emulation.

I believe that LXC already can do most of what VirtualBox can do (I could be wrong), but not as easily. For example, on a standard ext4 system, taking and restoring an LXC snapshot is slow and memory-intensive. Tailoring virtual hardware requires knowing command line options and how to use them.

On VirtualBox, taking and restoring a snapshot take less than a second. Tailoring virtual hardware is ridiculously easy. It’s also cross-platform, unlike LXC. VirtualBox isn’t “sucky” in the slightest.

As I mentioned, I use both LXC and VirtualBox, because they provide very different benefits and functions. Indeed, even if LXC could do everything that VirtualBox does, I still wouldn’t use it for that purpose, because it is so easy to design each VM in VirtualBox, and to chop and change at a moment’s notice.

They are both outstanding products, with different intentions, and highly useful. I love using both of them.

6

I think you are having LXC and LXD confused. LXC is a type of system container, a full blown OS in a namespaced environment, sharing the kernel with the host. LXD is a tool by Canonical that helps you manage LXCs and VMs.

LXD manages both LXC and VMs, not just LXC. Support for VMs was added in the 3.x releases 5+ years ago, we are now at 6.9. LXCs and VMs have feature parity where possible (they are different after all)

LXD can be used in several ways, for example:

  • Running LXCs and VMs locally, similar to VirtualBox/VMWare Workstation. Great for development.
  • Small private clouds on edge devices, even running on raspberry pi kind of devices
  • Private clouds at the datacenter level, running hundreds or thousands of LXCs and VMs.

I used LXD locally extensively at the beginning of this year creating an ISO installer for airgapped edge devices on my laptop. But most of my experience with LXD is running it at the datacenter level. Started out with LXD in 2017 and have run clusters with hundreds of VMs/LXCs at the ISP I worked at.

The power of LXD lies in how flexible it is. You can run a microcloud on a raspberry pi, or create a private cloud to host hundreds/thousands of VMs/LXCs in a datacenter. With software defined networking, or passed through onto the physical network. It’s your call.

The reason for me to create this topic was for the limitations I was seeing because of snap. If you’re only using it for the local development example it’s not really an issue. snap install lxd is a great fit for that use-case.

But when you want to use LXD/MicroCloud for your private cloud needs, be it the edge, a large datacenter scale or an airgapped super secure environment, host management/visibility comes into play. And snap is currently a limiting factor there.

1 Like
7

https://snapcraft.io/docs/reference/interfaces/shutdown-interface/

Technically there is nothing the snap could not do that a native app couldn’t do either (host OS update visibility you list above would need some system-files interface definitions and a generous reviewer team though), these are not snap limitations but simply non-existent features yet that would need implementation, the snap packaging is surely not the limiting factor here …

3 Likes
8

Hi @vosdev

Thanks for your post.

But how will Canonical’s in-house virtualization solution LXD benefit from this new stack?

LXD’s snap already bundles customised and minimal builds (using the same Ubuntu sources, but with unnecessary features and dependencies removed to reduce size and security issue exposure) of QEMU, EDK2 and Seabios. This provides predictability of behaviour rather than using the host’s virtualisation stack.

So the new HWE stack doesn’t directly affect LXD right now. However there may be scope to use the associated HWE sources in the future for non-LTS series of the LXD snap. This is to be discussed internally to understand how we can utilise this.

For a LXC management daemon, snap was fine. But when you start to look at LXD as a hypervisor, there are some limitations of lxd being confined to a snap that are in the way.
Cockpit-like host management:
being able to power off or reboot a node via API/UI.
Host OS updates visibility
Host logs
Host metrics visibility (SMART / sensors / thermal)
Network configuration beyond LXD-managed bridges

LXD is able to access and manage the system as it needs to, so there is no inherent reason why it could not do any of these things. Instead is is more of a product scope question.

Almost all of our/my deployments rely on host OS managed bridges to directly connect to the physical VLANs, all the way back to 2017. Network management is consistently the thing people struggle with most when I suggest LXD to someone. Most people just want to bridge a VM directly onto the physical network, and not everyone has a spare NIC to dedicate to a managed LXD bridge. “Give my VM an IP on vlan 20” is what is expected by most.

This is already supported by LXD by using a macvlan NIC type and specifying the vlan option:

This will create a VLAN interface if needed ontop of the parent, and then configure a macvlan interface ontop of that and pass it into the instance.

You can also use a single existing bridge with multiple VLANs connected, and then use vlan and vlan.tagged options of the bridged NIC type:

For cluster wide overlay networking the product focus is on MicroOVN integration.

1 Like
9

Thank you, you are right — I didn’t know about that.

1 Like
10

Ah that makes sense! And that is an added bonus of a snap, bundling dependencies with the program.

It’s great to know that these features are all possible! I thought snap would be a limitation but I was mistaken. Thank you @ogra for showing me! Then I guess this turns into a product scope question.

Unfortunately macvlan has a huge downside and that is that the guest and host are not able to speak to each other. Monitoring, backups, management from host-guest or guest-host will not be possible. This disqualifies it as a production candidate for me and probably most users.

The bridged + vlan.tagged route needs a VLAN aware bridge with all the right VLANs already trunked onto it. This needs to be configured by hand in netplan, outside of LXD, for each VLAN. This adds a lot of manual work for each node in your cluster, or each vlan that you wish to add. With debian’s ENI networking we could simply add the bridge-vlan-aware tag but netplan does not support this. It has been a feature request for almost 8 years now: Bug #1793128 “support vlan-aware bridges (vlan-filtering)” : Bugs : Netplan. It even mentions LXD in the comments. (Maybe you can use your influence to increase the priority :slight_smile: )

I’ve tried this route and it has been a pain adding every vlan to each node every time we add a vlan or someone wishes to deploy a vlan that has not been configured onto the bridges yet.

LXD managed bridge

LXD already creates and configures host network interfaces (lxdbr0) and manages host firewall rules.

With netplan configuration possible (https://snapcraft.io/docs/reference/interfaces/network-setup-control-interface/ ), could this feature be extended to bridges attached to a physical uplink?

With persistent, netplan declared bridges rather than on-the-fly created bridges like lxdbr0. This allows nodes with single NICs still use the same nic for host management’s ip address, which has to exist whether or not LXD is running.

Being able to directly control the bridges on the host is a feature that I have been missing since I started using LXD back in 2017. And now that I am mostly managing LXD via the UI, even more so. This type of setup is the default deployment for alternative kvm-based hypervisor solutions, and for a good reason. Granting instances access to your physical network just makes sense.

OVN is also very powerful. I have it running in two environments. A downside is that it adds a lot of complexity. A lot of configuration is required to route this traffic onto your physical network through BGP. (Hence my request for Routed OVN set up guide and microovn guide · Issue #15844 · canonical/lxd · GitHub) And BGP is still kind of a black box with LXD (related request: Proper API support for BGP state · Issue #16691 · canonical/lxd · GitHub)

The use-case for LXD is different for everyone. I have had environments with limited IPs where the default NAT bridge and port-forwarding abilities of LXD were a lovely built-in feature.

Previously LXD was CLI only (And API of course but I think the vast majority of users used the CLI) but now with the UI reaching feature-parity and clustering becoming richer and richer in features, these kind of features for network and host management start to make more sense.

Are these features related to host management in scope for LXD as a virtualization/private cloud solution at all, and if so, shall I make feature requests for them?