OVN Uplink Configuration - VFIO Error with Physical Network Type

LXD MicroCloud
,
1

Hello,

I’m setting up a 3-node MicroCloud cluster and having issues with OVN uplink network configuration for VMs.

My Setup:

  • 3 nodes: microcloud-01, microcloud-02, microcloud-03
  • Uplink network: 192.168.90.0/24 connected via VLAN 90
  • Physical interface: enp10s0 (no IP assigned directly)
  • VLAN interface: enp10s0.90 (has IP 192.168.90.11/24)
  • Server management IP is on the VLAN interface, not the physical interface

Network Configuration:

yaml

ethernets:
  enp10s0:
    dhcp4: false
    dhcp6: false
    # No IP - used as trunk

vlans:
  enp10s0.90:
    id: 90
    link: enp10s0
    addresses: [192.168.90.11/24]
    routes:
      - to: default
        via: 192.168.90.1

Current MicroCloud Configuration: During microcloud init, I specified:

  • OVN uplink interface: enp10s0 (physical interface without IP)
  • IPv4 gateway: 192.168.90.1/24

This created an UPLINK network with type: physical.

The Problem: When I try to create a VM and assign it a NIC from the UPLINK network:

bash

lxc config device add myvm eth0 nic network=UPLINK name=eth0

I get this error:

Failed setting up device via monitor: Failed setting up device "eth0": 
Failed adding NIC device: vfio 0000:0a:00.0: group 15 is not viable

My Questions:

  1. Should I create an OVS bridge (like uplink-br) with enp10s0 as a port and use that as the OVN uplink interface instead of the physical interface directly?
  2. What should be the correct network type for UPLINK - physical, bridge, or OVN?
  3. How should I configure the uplink interface when using VLANs? Should I:
  • Use the physical interface (enp10s0) without IP?
  • Create an OVS bridge first?
  1. Is there a proper way to configure OVN uplink for both containers and VMs to work with the same network definition?
2

When adding a NIC to an instance that references a managed network definition (in this case UPLINK) the NIC type is inferred from the network type (in this case physical).

When a physical NIC is added to an instance the parent interface is removed from the host and passed through to the instance. If there is any usage of the physical interface on the host (such as in this case a child VLAN interface) it will fail or worse break the usage.

So this isn’t what you want to do.

https://documentation.ubuntu.com/lxd/latest/reference/devices_nic/#nic-physical

A physical NIC provides straight physical device pass-through from the host. The targeted device will vanish from the host and appear in the instance (which means that you can have only one physical NIC for each targeted device).

If you want to share your physical interface with directly connected instances and with ovn networks as an uplink then I think you’re are on the right track with creating an externally managed OVS bridge and then using that as the parent for the physical UPLINK network.

You can then use a bridged NIC type with the parent set to the OVS bridge for the directly connected instance too.

https://documentation.ubuntu.com/lxd/latest/reference/devices_nic/#device-nic-bridged-device-conf:parent

There are some future ideas to make this transparent so that if you reference a managed physical network that is an external bridge for an instance NIC it would convert that into a bridged NIC automatically.