Trying to add a node which does not have a NIC on the uplink network to our MicroCloud cluster.
My assumption was that I could simply say microcloud join and answer no when prompted about configuring distributed networking.
However, when I do so, the join procedure ends with a rather obscure error:
Selected yoke at 10.2.0.242
Gathering system information...
Would you like to set up local storage? (yes/no) [default=yes]: no
Would you like to set up distributed storage? (yes/no) [default=yes]: yes
Using 1 disk(s) already setup on flap for remote storage pool
Using 1 disk(s) already setup on roll for remote storage pool
Using 1 disk(s) on yoke for remote storage pool
Do you want to encrypt the selected disks? (yes/no) [default=no]: no
Configure distributed networking? (yes/no) [default=yes]: no
Replace existing default profile configuration? (yes/no) [default=no]: no
Initializing new services ...
Awaiting cluster formation ...
Peer yoke has joined the cluster
Configuring cluster-wide devices ...
⨯ Error: Failed adding link:
Failed to run: ip link add name lxdfan0-fan type vxlan id 15728640 dev eno8303 local 192.168.0.240 dstport 0 fan-map 240.0.0.0/8:192.168.0.0/24:
exit status 2 (Error: binary attribute size out of range.)
(Added linebreaks to the last line for readability.)
It sounds like MicroCloud tries to configure distributed networking on the node even though we said no.
Also, the IP 192.168.0.240 that MicroCloud uses for the ip link add command is a network completely unrelated to MicroCloud. We specified 10.2.0.0/24 for internal MicroCloud traffic.
After this, lxc network list reports lxdfan0 in ERRORED state, which sounds like I broke existing distributed networking configuration.
Any hints on how to add a node without participating as a gateway chassis?
This error is not a MicroCloud issue, but rather that Ubuntu Fan has regressed support for certain iproute command versions when using the 6.14 kernel.
There are two types of network configurations supported by MicroCloud:
True distributed networking (using MicroOVN)
And the fallback FAN networking
The latter gets configured as a fallback when you answer with “no”. The same happens if you don’t have MicroOVN installed when forming the MicroCloud.
If you already have distributed networking configured, there should not be an option to opt out as it is required to also be present on any new member joining the cluster as already mentioned by @tomp.
I have just reproduced this using version 2.1.1 and it seems like if you select “no”, MicroCloud configures the new member to also use MicroOVN but also configures cluster wide FAN networking, which is not right. This is a bug. Thanks for reporting @christianrishoj.
If you already have distributed networking configured, there should not be an option to opt out as it is required to also be present on any new member joining the cluster.
Just to ensure my understanding is correct:
What we would like is for some cluster nodes to have uplink connections, while other nodes don’t. In other words, we would like to use some nodes as gateway chassis, while using other nodes mainly for storage.
Is this use case supported? Or, would it be better for us to add the “storage only” node using microceph commands?
MicroCloud will try to configure LXD on this new member, which requires the same set of networks as on the other cluster members.
If you just want to add another storage node (without LXD and MicroOVN), this is currently not supported by the microcloud interface as we expect all members to participate in everything.
A chassis is where traffic physically ingresses into or egresses out of the virtual network. In LXD, there will usually be one chassis per cluster member. If LXD is configured to use OVN networking, then all members can be used as OVN chassis.
If any cluster members have the role ovn-chassis, only those members are represented as chassis in the chassis group table (see below). If no members have the role, all cluster members are added to the chassis group.
In addition please make sure that your UPLINK network on the new member has a parent network set as this probably also got lost because by skipping the distributed networking configuration you haven’t selected one so it wasn’t configured in LXD.
Please verify the config setting isn’t present by running lxc network get UPLINK parent --target <new cluster member>. The response should be empty.
The best way to clean this up is to remove the member and add it back into MicroCloud to get the members LXD UPLINK network fixed.
A last resort option would be to run some SQL statements to address this but it modifies the live database so we should rather not go this route.
Please verify the config setting isn’t present by running lxc network get UPLINK parent --target <new cluster member> . The response should be empty.
Can confirm. No parent network for UPLINK for the newly added node.
Also, comparing to other members, lxdovn1 is missing from lxc network list.