Hi @dnebing
I wanted to take some time to explain how MicroCloud networking operates and hopefully correlate that to your setup.
The microcloud init has two different network related phases.
The first stage asks you to “Select an address for MicroCloud’s internal traffic”.
This will list all network interfaces that are up and configured with an IP address.
This is used for intra-cluster traffic, including MicroCloud’s own mutlicast discovery process, as well as OVN and Ceph traffic between cluster members.
At this time it is not possible to specify different interfaces for OVN and Ceph intra-cluster (east-west) traffic, but this is on our roadmap.
This first interface must be multicast compatible (i.e a real layer 2 network) and be connected to the other MicroCloud cluster members.
The second networking stage asks you to "Configure distributed networking? (yes/no) [default=yes]: " and then asks you to pick “Select exactly one network interface from each cluster member:”.
This second stage is asking you to configure which network interface should be used for the so-called Uplink interface, which is used to connect the OVN network(s) created to the external network (north-south traffic), and potentially the internet.
These interfaces must be either:
- Unused (no IPs configured on them).
- A bridge interface.
The reason for this is that in order to connect a physical interface to the OVN virtual router that is created for each OVN network that interface needs to be connected to an associated OVS bridge on the host. LXD will do this for you, but the act of doing this will render any IPs configured on the interface inactive, so in order to avoid accidentally disconnecting yourself we don’t allow interfaces with IPs configured to be used.
The exception here is if the interface is a bridge already, because in that case we can connect that bridge to the OVS bridge using a virtual veth connection.
This uplink interface doesn’t have to be the same interface on every cluster member, which is why you can pick a different one for each member, but they must be connected to the same layer 2 network (which doesn’t necessarily have to be the same as as used for the internal traffic).
The reason for this is that OVN will pick a single cluster member to use as the active chassis when routing traffic to/from the uplink network. Each OVN network will get a virtual router and its virtual external interface will be assigned an IP address on the uplink network from the range provided during the initialisation.
If one of the cluster members goes down, then OVN will move that virtual router’s IP to a different cluster member by way of ARP/NDP adverts. So the uplink interfaces must be on the same layer 2.
So hopefully that helps you understand what is going on under the hood here.
@maria-seralessandri @masnax would it be possible to improve the microcloud init process to make it clearer what the selected interface in the OVN setup phase is going to be used for? As currently it doesn’t say why the user is being asked to pick an interface, nor is it clear that this interface must be unused or a bridge.