Bridge uplink network for OVN

LXD MicroCloud
1

I apologise if I overlooked something in the documentation. I am not an expert in networking.

I am trying to set up OVN networking for a new cluster consisting of four physical nodes using MicroCloud.

It is not very clear to me which networks can be set as “uplink” of the OVN network using the configuration option network.

According to the description here:

For outside connectivity, the OVN network requires an uplink network (a Bridge network or a Physical network)

And here it is written:

A high availability OVN cluster requires a shared layer 2 network, so that the active OVN chassis can move between cluster members (which effectively allows the OVN router’s external IP to be reachable from a different host).

Therefore, you must specify either an unmanaged bridge interface or an unused physical interface as the parent for the physical network that is used for OVN uplink.

Does it mean that the network value for the OVN network must be a network of type physical, whose parent can be either an unused interface or a bridge?

In this old tutorial a bridge created by LXD is used directly as a network of the OVN network. Is this configuration officially supported? If so, is it recommended for production?

Thanks!

2

During the MicroCloud setup process the questions will guide you through creating an “UPLINK” network which will create a network of type physical with a parent set to an unused physical interface or an existing bridge network (not managed by MicroCloud/LXD). This UPLINK network can then be used for providing external connectivity to OVN networks you create.

You can create multiple OVN networks that all use the same UPLINK network for external connectivity.

If you want to create your own additional uplink network for use with specific OVN networks then you can define additional physical networks and these networks can also reference an unused physical interface or an existing external bridge interface.

LXD will detect the interface type and act accordingly to connect it to OVN networks.

In this old tutorial a bridge created by LXD is used directly as a network of the OVN network. Is this configuration officially supported? If so, is it recommended for production?

You can indeed connect an OVN network to a LXD managed bridge network.
However this is not particularly useful in a cluster scenario because each cluster member will have the bridge created locally with the same gateway IP and private DHCP range.

This means that OVN’s ability to failover the active chassis automatically to a different cluster member will not work properly, as the uplink network is not a shared layer 2 network (which is required for IP portability).

3

Many thanks for the detailed explanations!

Unfortunately, our servers do not have a spare physical interface with external connectivity, hence I am looking for workarounds.

Each of our cluster nodes has two NICs:

  • One NIC, say eth0 is assigned with a static (public) IPv4 address, providing (external) internet connectivity.
  • Another NIC, say eth1 is connected to a (100GBit) switch, not connected to the outside network (only connecting the cluster nodes).

Currently I have assigned the second NIC for the microcloud internal traffic (lxd + ceph + ovn) by configuring static IPs on one of the VLANs:

network:
  version: 2
  ethernets:
    eth1:
      dhcp4: false
      dhcp6: false
  vlans:
    lxd-vlan10:
      id: 10
      link: eth1
      mtu: 9600
      addresses:
      - 10.0.1.x/24

Since I cannot use the physical NIC eth0 for the OVN uplink network, I would either need to use the NIC eth1 (with a different VLAN) or a bridge.

I tried creating another VLAN interface for eth1 and using it for the OVN uplink, which was accepted in the setup process, but my containers would not be able to reach the external network (ping 8.8.8.8). I guess this is because my second network is isolated. The containers could ping each other without problems.

If I create a bridge, then (I guess) I need to use the same gateway as for the external interface eth0, but I do not have any spare (public) IP addresses to be assigned for ipv4.ovn.ranges (which, as I understand, need to be on the same subnet as the gateway).

Hence I was interested in the above-mentioned tutorial that used an LXD-managed bridge as the OVN uplink interface, without using any physical interface. This is the only solution that worked for me so far: the containers could ping each other as well as the external network.

I have now tested the failover for virtual switches, and so far I have not noticed any problems. Here is what I tried:

My configuration:

# lxc network show lxdbr0 --target node1
name: lxdbr0
description: ""
type: bridge
managed: true
status: Created
config:
  ipv4.address: 10.0.2.1/24
  ipv4.dhcp: "false"
  ipv4.nat: "true"
  ipv4.ovn.ranges: 10.0.2.101-10.0.2.254
  ipv6.address: fd42:69a:afe:aafa::1/64
  ipv6.nat: "true"
used_by:
- /1.0/networks/default
- /1.0/networks/ovn1
locations:
- node1
- node2
- node3
- node4

Note that dhcp is disabled. It does not look like it is being used. I guess, OVN itself assigns the IP addresses to virtual switches. Just like in the original tutorial, the bridge also does not have any parent (physical) interface.

Here is my OVN network, which is used in the default profile:

# lxc network show default
name: default
description: ""
type: ovn
managed: true
status: Created
config:
  bridge.mtu: "1500"
  ipv4.address: 10.128.134.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:5964:943:bad9::1/64
  ipv6.nat: "true"
  network: lxdbr0
  volatile.network.ipv4.address: 10.0.2.101
  volatile.network.ipv6.address: fd42:69a:afe:aafa:216:3eff:fe20:e45a
used_by:
- /1.0/instances/c1
- /1.0/instances/c2
- /1.0/instances/c3
- /1.0/instances/c4
- /1.0/profiles/default
locations:
- node1
- node2
- node3
- node4

I can only ping 10.0.2.101 from node3, so I guess this means that the virtual switch for the OVN network default is currently running on this node.

root@node3:~# ping -c 3 10.0.2.101
PING 10.0.2.101 (10.0.2.101) 56(84) bytes of data.
64 bytes from 10.0.2.101: icmp_seq=1 ttl=254 time=0.922 ms
64 bytes from 10.0.2.101: icmp_seq=2 ttl=254 time=0.759 ms
64 bytes from 10.0.2.101: icmp_seq=3 ttl=254 time=0.677 ms

--- 10.0.2.101 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2019ms
rtt min/avg/max/mdev = 0.677/0.786/0.922/0.101 ms

Now, while on node3 I turn off the microovn switch service:

root@node3:~# microovn disable switch
Service switch disabled

After that the virtual switch becomes unreachable:

root@node3:~# ping -c 3 10.0.2.101
PING 10.0.2.101 (10.0.2.101) 56(84) bytes of data.
From 10.0.2.1 icmp_seq=1 Destination Host Unreachable
From 10.0.2.1 icmp_seq=2 Destination Host Unreachable
From 10.0.2.1 icmp_seq=3 Destination Host Unreachable

--- 10.0.2.101 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2024ms
pipe 3

But now I can ping it on node1:

root@node1:~# ping -c 3 10.0.2.101
PING 10.0.2.101 (10.0.2.101) 56(84) bytes of data.
64 bytes from 10.0.2.101: icmp_seq=1 ttl=254 time=2.07 ms
64 bytes from 10.0.2.101: icmp_seq=2 ttl=254 time=0.691 ms
64 bytes from 10.0.2.101: icmp_seq=3 ttl=254 time=0.750 ms

--- 10.0.2.101 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.691/1.169/2.067/0.635 ms

So I guess, that means that the failover is working?

All containers on node1, node2 and node4 can ping each other and the external network (8.8.8.8).
Only the containers on node3 lost the network connectivity. I do not know if it is to be expected.

Sorry for a long reply.

4

After I have re-enabled the OVN service on node3:

microovn enable switch

I have now simulated an “unorderly” failure of node3 by bringing down the interface eth1 (used for the internal traffic) on node3. The network seemed to have survive as well:

root@node1:~# lxc list
+------+---------+---------------------+----------------------------------------------+-----------+-----------+----------+
| NAME |  STATE  |        IPV4         |                     IPV6                     |   TYPE    | SNAPSHOTS | LOCATION |
+------+---------+---------------------+----------------------------------------------+-----------+-----------+----------+
| c1   | RUNNING | 10.128.134.2 (eth0) | fd42:5964:943:bad9:216:3eff:fe7e:90 (eth0)   | CONTAINER | 0         | node1    |
+------+---------+---------------------+----------------------------------------------+-----------+-----------+----------+
| c2   | RUNNING | 10.128.134.3 (eth0) | fd42:5964:943:bad9:216:3eff:fe45:ce4c (eth0) | CONTAINER | 0         | node2    |
+------+---------+---------------------+----------------------------------------------+-----------+-----------+----------+
| c3   | ERROR   |                     |                                              | CONTAINER | 0         | node3    |
+------+---------+---------------------+----------------------------------------------+-----------+-----------+----------+
| c4   | RUNNING | 10.128.134.5 (eth0) | fd42:5964:943:bad9:216:3eff:fee2:baac (eth0) | CONTAINER | 0         | node4    |
+------+---------+---------------------+----------------------------------------------+-----------+-----------+----------+
root@node1:~# lxc exec c1 -- ping -4 -c 3 c2
PING c2 (10.128.134.3) 56(84) bytes of data.
64 bytes from c2.lxd (10.128.134.3): icmp_seq=1 ttl=64 time=2.80 ms
64 bytes from c2.lxd (10.128.134.3): icmp_seq=2 ttl=64 time=0.565 ms
64 bytes from c2.lxd (10.128.134.3): icmp_seq=3 ttl=64 time=0.423 ms

--- c2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2052ms
rtt min/avg/max/mdev = 0.423/1.262/2.799/1.088 ms
root@node1:~# lxc exec c1 -- ping -4 -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=115 time=10.2 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=115 time=6.63 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=115 time=5.48 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 5.483/7.442/10.210/2.012 ms
5

Yes best bet here is to attach your eth0 interface to a manually created br0 and move the IP from eth0 to br0, then you can use br0 as your OVN uplink interface.

I need to use the same gateway as for the external interface eth0

Correct

but I do not have any spare (public) IP addresses to be assigned for ipv4.ovn.ranges (which, as I understand, need to be on the same subnet as the gateway).

Yes this is a problem as each OVN network gets its own IP on the uplink network (it provides a virtual router separate from the host itself).

Hence I was interested in the above-mentioned tutorial that used an LXD-managed bridge as the OVN uplink interface, without using any physical interface. This is the only solution that worked for me so far: the containers could ping each other as well as the external network.

If this works for you then thats all good.

However it is problematic because:

  1. The lxdbr0 network is duplicated on every cluster member with the same IPs - this will be an issue if you ever want to expose services from your OVN networks to the external network.
  2. There’s no true shared L2 between the cluster members for the OVN UPLINK which means egress traffic is going to get double NATted to the IP of each cluster member on the external network. This means that failover during a cluster member failover won’t be as clean as suddenly the external source address of ongoing connections will change.
6

But shouldn’t the bridge be “unmanaged” to be used as an uplink? Or does it mean that it should not be managed by LXD?

I am exposing services using a separate Macvlan network. The containers are assigned with static (public) IP addresses on this network. So indeed, I have a few spare IP addresses, but I prefer to use them in a controlled way.

I do not have a deep understanding of implications for “duplicated” networks, but I thought that since lxdbr0 bridges are not connected to any common (physical) interface, these networks are completely separated and should not cause any conflicts. Their only purpose for OVN is NAT translation.

Isn’t the problem of double NAT traversal, generally, unavoidable if the egress traffic needs to pass through three different networks (OVN underlay network > OVN overlay network > public network)?

In principle, I can try connecting these bridges using a spare VLAN on eth1 by setting bridge.external_interfaces, but this did not seem to work in my tests. I guess this causes some conflicts due to four identical networks.

An AI assistant proposed a solution to create a separate network on eth1 (on a spare VLAN), and use NAT rules to connect the gateway of this network to the external network (eth0) using a Keepalived service, which implements a “Virtual Router Redundancy Protocol” (VRRP). In my understanding, this should be similar to OVN handover for virtual routers. However, it is unclear to me if this method has any advantages over using the bridge lxdbr0 as described in the tutorial (which is, clearly, much simpler).

In regards to MicroCloud setup, I am wondering if in situations like mine (where there are no spare interfaces or IP addresses on the external network), which, presumably, should be a very common situation, it would be better to default to using an LXD bridge as OVN uplink (as described in the linked tutorial) over using a fan network (as it is currently implemented).

7

Oh, I see, that could be a problem. Does it mean that, say, an SSH connection from a container to an external host could be interrupted during the failover?

I guess a keepalive service will then also have the same problem.

I think this setup is still better than using a fan network, in which case the failover can result in a VM suddenly changing its IP address (if running VMs can be migrated over a fan network at all).

8

You can use any bridge, but if that bridge is connected to a physical port with a shared L2, but rather is isolated from the wider network then it is sub-optimal for the reasons I mentioned.

FWIW its possible to use an external IP for an OVN virtual router, and then also use that same IP for a network forward into services inside that OVN network, so one can use the same IP for multiple things for a single OVN network.

but I thought that since lxdbr0 bridges are not connected to any common (physical) interface, these networks are completely separated and should not cause any conflicts. Their only purpose for OVN is NAT translation.

In that case if you’re only using it for that, and your happy with the per-cluster member NAT which will occur then it should be fine.

Isn’t the problem of double NAT traversal, generally, unavoidable if the egress traffic needs to pass through three different networks (OVN underlay network > OVN overlay network > public network)?

No in recommended setups, only OVN virtual router does NAT directly onto the uplink network.

In principle, I can try connecting these bridges using a spare VLAN on eth1 by setting bridge.external_interfaces , but this did not seem to work in my tests. I guess this causes some conflicts due to four identical networks.

This can work too, but you’d need to also remove the ipv{4,6}.address from the bridge and disabled DHCP otherwise you’ll get IP conflicts on the VLAN.

Easier in my opinion to use an unmanaged bridge created by netplan to achieve this rather than trying to convert a LXD managed bridge into effectively an unmanaged one.

In regards to MicroCloud setup, I am wondering if in situations like mine (where there are no spare interfaces or IP addresses on the external network), which, presumably, should be a very common situation, it would be better to default to using an LXD bridge as OVN uplink (as described in the linked tutorial) over using a fan network (as it is currently implemented).

This is not the recommended deployment strategy for MicroCloud.

The uplink IPs don’t have to be globally routable (there can be another NAT router converting the traffic to globally routable IPs) but the recommendation is to have a shared L2 for OVN chassis failover.

See https://documentation.ubuntu.com/microcloud/v2-edge/microcloud/how-to/install/#pre-deployment-requirements

9

OK, that is useful to know. The problem is that I do not know how many OVN routers need to be created. Is it the case that each project requires a separate router, even if I configure just one OVN network?

I think I tried that as well. The main problem is to configure NAT on these bridges. I will probably need to manually modify the iptables and I am not very well familiar with them. LXD conveniently does that for you. I guess I will give another try to setup a keepalive service to implement NAT to a spare fixed public IP address.

Could it be the reason for the following segfault when trying to add a new cluster member (node3 which I previously removed from lxd, microceph, microovn and microcloud) after I configured OVN network as I described?

# microcloud add
Waiting for services to start ...
Use the following command on systems that you want to join the cluster:
                                                                       
 microcloud join                                     
                                     
When requested, enter the passphrase:
                                     
 molecule utterance february peony                       
                       
Verify the fingerprint d7bc0bc96467 is displayed on joining systems.

 Selected node3 at 10.0.1.13

Gathering system information...
Would you like to set up local storage? (yes/no) [default=yes]: no
Would you like to set up distributed storage? (yes/no) [default=yes]: no
Would you like to set up CephFS remote storage? (yes/no) [default=yes]: no
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0xb31c70]

goroutine 1 [running]:
github.com/canonical/microcloud/microcloud/service.(*SystemInformation).SupportsOVNNetwork(...)
        github.com/canonical/microcloud/microcloud/service/system_information.go:313
main.(*initConfig).askOVNNetwork(0xc00039a460, 0xc0001c7880)
        github.com/canonical/microcloud/microcloud/cmd/microcloud/ask.go:1000 +0x210
main.(*initConfig).askNetwork(0xc00039a460, 0xc0001c7880)
        github.com/canonical/microcloud/microcloud/cmd/microcloud/ask.go:1335 +0x32
main.(*cmdAdd).Run(0xc00012d010, 0xc000173ad8?, {0xc00012fbc0?, 0x0?, 0x0?})
        github.com/canonical/microcloud/microcloud/cmd/microcloud/add.go:232 +0x167e
github.com/spf13/cobra.(*Command).execute(0xc000288608, {0xc00012fba0, 0x2, 0x2})
        github.com/spf13/cobra@v1.9.1/command.go:1015 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc000288008)
        github.com/spf13/cobra@v1.9.1/command.go:1148 +0x46f
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.9.1/command.go:1071
main.main()
        github.com/canonical/microcloud/microcloud/cmd/microcloud/main.go:105 +0xb0d
10

Hmm… I get exactly the same error even after changing the network according to the recommendation (and removing all previous networks and containers).

# lxc network show ovn0 --target node1
name: ovn0
description: ""
type: ovn
managed: true
status: Created
config:
  bridge.mtu: "1500"
  ipv4.address: 10.214.26.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:107:dc1e:1a7f::1/64
  ipv6.nat: "true"
  network: UPLINK
  volatile.network.ipv4.address: xxx.xx.xx.223
used_by:
- /1.0/profiles/default
locations:
- node1
- node2
- node4

# lxc network show UPLINK --target ember
name: UPLINK
description: ""
type: physical
managed: true
status: Created
config:
  ipv4.gateway: xxx.xx.xx.1/24
  ipv4.ovn.ranges: xxx.xx.xx.223-xxx.xx.xx.229
  parent: br0
  volatile.last_state.created: "false"
used_by:
- /1.0/networks/ovn0
locations:
- node1
- node2
- node4

My system:

# microcloud --version
2.1.1 LTS
# lxd --version
5.21.3 LTS
# microceph --version
ceph-version: 19.2.0-0ubuntu0.24.04.2; microceph-git: ab139d4a1f
# microovn --version
microovn: a2c59c105b
ovn: 24.03.2-0ubuntu0.24.04.1
openvswitch: 3.3.0-1ubuntu3
# uname -a
Linux ember 6.8.0-78-generic #78-Ubuntu SMP PREEMPT_DYNAMIC Tue Aug 12 11:34:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
# lsb_release -d
No LSB modules are available.
Description:    Ubuntu 24.04.3 LTS

I will try installing MicroCloud from scratch.

11

Its 1 uplink IP per OVN network, irrespective of which project they are in.

12

If you do that then you’re effectively back to the same downsides as using a LXD managed bridge - no shared L2 on the uplink between cluster members.

13

I’ve asked one of the MicroCloud developers to look into that stack trace. Please can you provide snap list output.

14

Then I do not understand your suggestion. How do I ensure external connectivity without using public addresses for OVN ranges and without NAT?

15

Unfortunately I already wiped the systems, but I will try to reproduce the problem with a fresh install.

16

Thanks for the report @ykazakov. I was able to reproduce it. You mentioned you have removed cluster members which caused also some networks (e.g. UPLINK or default) to still remain on the removed system. This caused the panic. You can reproduce it by:

root@micro01:~# brctl addbr UPLINK
root@micro01:~# microcloud init
Waiting for services to start ...
Do you want to set up more than one cluster member? (yes/no) [default=yes]: no

 Using address 10.87.144.124 for MicroCloud

! Warning: Discovered non-LTS version "6.5" of LXD
Gathering system information ...
Would you like to set up local storage? (yes/no) [default=yes]: no
Would you like to set up distributed storage? (yes/no) [default=yes]: no
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0xb31c18]

goroutine 1 [running]:
github.com/canonical/microcloud/microcloud/service.(*SystemInformation).SupportsOVNNetwork(...)
    github.com/canonical/microcloud/microcloud/service/system_information.go:313
main.(*initConfig).askOVNNetwork(0xc000157180, 0xc0001cc770)
    github.com/canonical/microcloud/microcloud/cmd/microcloud/ask.go:1000 +0x1b8
main.(*initConfig).askNetwork(0xc000157180, 0xc0001cc770)
    github.com/canonical/microcloud/microcloud/cmd/microcloud/ask.go:1335 +0x32
main.(*initConfig).RunInteractive(0xc000157180, 0x0?, {0x0?, 0x0?, 0x0?})
    github.com/canonical/microcloud/microcloud/cmd/microcloud/main_init.go:301 +0xe7e
main.(*cmdInit).Run(0xc00004aa00, 0xc000210308, {0xc0000578c0, 0x0, 0x2})
    github.com/canonical/microcloud/microcloud/cmd/microcloud/main_init.go:158 +0x185
github.com/spf13/cobra.(*Command).execute(0xc000210308, {0xc0000578a0, 0x2, 0x2})
    github.com/spf13/cobra@v1.9.1/command.go:1015 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc000210008)
    github.com/spf13/cobra@v1.9.1/command.go:1148 +0x46f
github.com/spf13/cobra.(*Command).Execute(...)
    github.com/spf13/cobra@v1.9.1/command.go:1071
main.main()
    github.com/canonical/microcloud/microcloud/cmd/microcloud/main.go:105 +0xb0d

I’ll have a fix ready shortly.

1 Like
17

It sounds like in your setup you cannot, you have to choose either per-cluster member double NAT or consume 1 of your external IPs per OVN network.

18

Fix in Service: Account for leftover network resources to prevent panic by roosterfish · Pull Request #937 · canonical/microcloud · GitHub

1 Like