Allowing a MicroCloud user to specify an underlay network for OVN

Project MicroCloud
Status Drafting
Author(s) @gabrielmougard
Approver(s) @tomp @fnordahl
Internal ID LX060


Introduce an OVN underlay network for handling east-west traffic.


As of the current state, MicroCloud relies on its management network for handling “east-west” traffic, which imposes certain limitations. One of the constraints is the potential for network congestion and reduced performance due to the management network being overburdened with both control and data plane traffic. This setup can lead to inefficiencies, particularly in environments with high network traffic or large-scale deployments. This current setup, while functional for smaller or less demanding environments, might not scale efficiently or meet the performance and flexibility needs of more complex, dynamic, or larger-scale cloud deployments (e.g: the nodes have a 100GB NIC that they would use for the data plane and a 1GB for the management plane).


To address the current limitations and enhance network efficiency in MicroCloud, we propose to give the option to the user to define a dedicated OVN underlay network for east-west traffic. By doing so, we aim to alleviate network congestion, improve overall performance, and enable more efficient scaling:

  • The proposal includes introducing configurable service handlers and dynamic configuration capabilities within the MicroCloud and MicroOVN codebases.
  • This will allow for greater flexibility and control over network settings, including the option to specify custom IP addresses for OVN Geneve tunnel encapsulation.
  • Such configurability ensures that the network can be optimized according to the specific needs of different deployments.

Manual interactive setup (microcloud init)

We would start by asking the following question:

  • Configure dedicated underlay networking? (yes/no) [default=no]
    This question is asked after the OVN management network questions. If no is answered, we skip the underlay configuration and we configure MicroCloud as usual. If yes is answered, we’ll give to the user a list of all the network interfaces with an IP address among the detected mDNS nodes. The user will need to choose one per node. If done properly, each InitSystem struct should have a non-nil OVNGeneveAddr field containing the chosen IP address of this node’s interface.

  • Once the nodes are all characterized, we arrive at the cluster setup phase. We’ll now want to make use of the new feature introduced in MicroCluster which is that we can now pass an initConfig map (map[string]string) to the MicroCluster hooks. When creating the MicroOVN cluster (OnBootstrap hook), we propose to give the following initConfig map:


  “<node0_hostname>.ovn-encap-ip”: “<chosen_iface_ip_for_node0>”,
  “<node1_hostname>.ovn-encap-ip”: “<chosen_iface_ip_for_node1>”,
  • Now, on the MicroOVN Bootstrap(initConfig map[string]string) function, if these fields are present, we can set the OVS configuration with the proper interface ip for the Geneve tunnel:
// Configure OVS to either use a custom encapsulation IP for the geneve tunel
// or the hostname of the node.
var ovnEncapIp string
if initConfig != nil {
	for k, v := range initConfig {
		if k == fmt.Sprintf("%s.ovn-encap-ip", s.Name()) {
			ovnEncapIp = v

if ovnEncapIp == "" {
	ovnEncapIp = s.Address().Hostname()

_, err = VSCtl(
	"set", "open_vswitch", ".",
	fmt.Sprintf("external_ids:system-id=%s", s.Name()),
	fmt.Sprintf("external_ids:ovn-remote=%s", sbConnect),
	fmt.Sprintf("external_ids:ovn-encap-ip=%s", ovnEncapIp),

Preseed setup (microcloud init --preseed <PRESEED_FILE.yaml>)

Just like we have the ovn_uplink_interface , we will introduce a new underlay_ip field that is also an IP address in CIDR notation. After we validate that each node has a valid underlay_ip and that the state of the LXD networks correspond to the values entered by the user, we set the OVNGeneveAddr to this IP address (no CIDR notation needed). The cluster setup phase is the same as described in the above section.

Auto setup (microcloud init --auto)

The auto setup just skips the option of having a dedicated OVN underlay network.

Communication with MicroOVN

  • The obvious issue here is what should we do in the case where the installed version of MicroOVN does not support consuming the initConfig map in the Bootstrap function? In such a situation, the configuration given by the MicroCloud user will simply not be applied and will cause no setup error but this will definitely feel wrong. To counter that, and this is an open interrogation, would it be a good idea to add an API extension system (that is queryable through MicroOVN’s REST API) like how it is done in LXD for example? We could then return early in the MicroCloud setup phase with a warning saying that the installed version of MicroOVN does not support that kind of setting.

  • If the API extension solution is out of scope for now, what could be a workable solution for this?

API Changes


CLI Changes

  • Adding a new question after the OVN uplink configuration to ask if the user desires to configure a custom underlay for OVN. If applicable, a table with a list of network interface with allocated IPs will be shown for the user to choose one per node to describe the underlay.

Database changes


Thank you for putting this spec together. It looks good to me overall. I can’t say much about the MicroCloud side of things, but i do have couple notes on the MicroOVN side.

  1. Similar changes, as those proposed to the Bootstrap.go, will be required for Join.go

  2. Would you consider extending MicroOVN’s init dialog with a question that would enable setting up this feature in standalone MicroOVN deployment?

  3. It would be great if we could get this new functionality covered in tests as well. (example of existing test). MicroOVN uses BATS for functional testing. The structure and organisation of tests may seem bit daunting. Feel free to reach out to me or @fnordahl with any questions.
    The way it works is that tests root directory contains .bats files that represent individual test suits executed independently. These .bats files are usually just a symlink to actual test implementation in the test_helper/bats/ directory. The test implementations load different setup/teardown scripts based on the test suite name that invoked them. This allows us to reuse same test suite for different cluster setups.
    For example, both init_cluster.bats and init_cluster_ipv6.bats are just a links to test_helper/bats/cluster.bats. That means that during testing we setup one IPv4 and one IPv6 cluster and run exactly same set of tests on each.

3.1 I’m thinking that it’d be best to have two new suites based on cluster.bats in a similar fashion as currently existing init_cluster.bats and init_cluster_ipv6.bats, that would test init_cluster_custom_encap.bats and init_cluster_ipv6_custom_encap.bats. What do you think @fnordahl ? (Although if we are going to be adding more init options in the future, we’ll probably eventually have to figure out automatic test suite generation for each possible init combination. But I don’t think we’re there yet.)