OVN not presenting Routes to Bridge Networks/Host

LXD Support
1

Hey All,

I’m trying to understand if my OVN networks can have their routes made dynamically available.

I have an LXD cluster with OVN up and working for communication between hosts.

My network setup is Grafana-ovn (OVN Network) > UPLINK (Physical) > ovnbr0 (OVN Bridge linked to ethernet)

Right now, my OVN network’s route is not dynamically presented to the host, or ovnbr0 (I added a container there and it did not see the route).

If I create a static route to the volatile IP of the OVN on the UPLINK network, I am able to connect as expected.

Is there a way for me to do this dynamically?

Here is my network config

A host’s route:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.200.1   0.0.0.0         UG    100    0        0 eth0
default         192.168.200.1   0.0.0.0         UG    600    0        0 wlan0
10.32.41.0      0.0.0.0         255.255.255.0   U     0      0        0 ovnbr0
10.38.194.0     0.0.0.0         255.255.255.0   U     0      0        0 lxdbr0
192.168.200.0   0.0.0.0         255.255.255.0   U     100    0        0 eth0
192.168.200.0   0.0.0.0         255.255.255.0   U     600    0        0 wlan0

lxc network show Grafana-ovn 
config:
  bridge.mtu: "1442"
  ipv4.address: 10.172.74.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:2c84:afc5:19ce::1/64
  ipv6.nat: "true"
  network: UPLINK
  volatile.network.ipv4.address: 10.32.41.10
description: ""
name: Grafana-ovn
type: ovn
used_by:
- /1.0/instances/Grafana
managed: true
status: Created
locations:
- host-01
- host-02

lxc network show UPLINK 
config:
  bgp.peers.grafana.address: 10.32.41.1
  bgp.peers.grafana.asn: "10001"
  dns.nameservers: 10.32.41.1
  ipv4.gateway: 10.32.41.1/24
  ipv4.ovn.ranges: 10.32.41.10-10.32.41.19
  ipv4.routes: 10.172.74.0/24
  ovn.ingress_mode: routed
  volatile.last_state.created: "false"
description: ""
name: UPLINK
type: physical
used_by:
- /1.0/networks/Grafana-ovn
managed: true
status: Created
locations:
- host-01
- host-02

lxc network show ovnbr0 
config:
  ipv4.address: 10.32.41.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:b306:8819:e44a::1/64
  ipv6.nat: "true"
description: ""
name: ovnbr0
type: bridge
used_by:
- /1.0/instances/Grafana2
- /1.0/networks/UPLINK
managed: true
status: Created
locations:
- host-01
- host-02
2

I see you are using BGP integration with LXD.

Right now LXD will only export routes for instances for OVN networks if the OVN network has ipv4.nat=false. The thinking being that SNATted networks shouldn’t be announced externally.

You can use lxc query /internal/testing/bgp to see the routes that LXD is currently exporting.

3

Hi Tomp,

To clarify what I’m trying to do: Present OVN networks inside ovnbr0/UPLINK to lxdbr0.

I don’t really need the routes presented outside the host (I dont care if they are for this purpose). I mainly want them presented to lxdbr0 to simulate a migration between a normal network to OVN.

Obligatory not an expert in BGP. I swapped a few things around in my network during troubleshooting including the ipv4.nat to false for a couple ovn networks (and changing the ASN to a private range and the same across this environment, I believe that is normal).

When you say announced externally, what is the “external” in this context, outside the ovnbr0 bridge, or the server itself/something else?

The query command shows the following.

{
        "peers": [
                {
                        "address": "10.0.100.1",
                        "asn": 64512,
                        "count": 1,
                        "holdtime": 0,
                        "password": ""
                },
                {
                        "address": "10.32.41.1",
                        "asn": 64512,
                        "count": 1,
                        "holdtime": 0,
                        "password": ""
                }
        ],
        "prefixes": [
                {
                        "nexthop": "10.32.41.10",
                        "owner": "network_65",
                        "prefix": "10.172.74.0/24"
                },
                {
                        "nexthop": "10.32.41.11",
                        "owner": "network_68",
                        "prefix": "10.0.100.0/24"
                }
        ],
        "server": {
                "address": "192.168.200.12:179",
                "asn": 64512,
                "router_id": "192.168.200.12",
                "running": true
        }
}

Note, I have two nodes in my cluster. The config on my second LXD server has the IP of that server as the server/routerID (I was unclear in the documentation if they were supposed to be a shared IP for clusters or not)

From the hosts perspective, routes are as follows

default via 192.168.200.1 dev wlan0 proto static metric 600 
10.32.41.0/24 dev ovnbr0 proto kernel scope link src 10.32.41.1 
10.38.194.0/24 dev lxdbr0 proto kernel scope link src 10.38.194.1 linkdown 
192.168.200.0/24 dev wlan0 proto kernel scope link src 192.168.200.12 metric 600

From a container inside one of the OVN networks

default via 10.172.74.1 dev eth0 proto dhcp src 10.172.74.10 metric 1024 
10.32.41.1 via 10.172.74.1 dev eth0 proto dhcp src 10.172.74.10 metric 1024 
10.172.74.0/24 dev eth0 proto kernel scope link src 10.172.74.10 metric 1024 
10.172.74.1 dev eth0 proto dhcp scope link src 10.172.74.10 metric 1024

On a container in the lxdbr0:

default via 10.38.194.1 dev eth0 proto dhcp src 10.38.194.111 metric 1024 
10.38.194.0/24 dev eth0 proto kernel scope link src 10.38.194.111 metric 1024 
10.38.194.1 dev eth0 proto dhcp scope link src 10.38.194.111 metric 1024

Thanks for any input!

4

I had a chance to revisit this. I rebuilt everything and am using ovnbr0 as an unmanaged bridge.

That seemed to make me have to set up some nft rules to get outbound access working, so I’m currently thinking this is actually a firewall issue RE: the BGP routes.

I see the BGP server trying to reach the OVN gateways but failing in tshark.

I’m going to do a deep dive on nftables tomorrow and hopefully get things sorted, or at least rule it out.

Long story short, I am going to try some things. No need to respond, this is just an update on my end as I’m making progress

pi@lxd-host-01:~ $ sudo tshark -f "port 179" -i any
Running as user "root" and group "root". This could be dangerous.
Capturing on 'any'
 ** (tshark:98817) 15:44:32.303002 [Main MESSAGE] -- Capture started.
 ** (tshark:98817) 15:44:32.303084 [Main MESSAGE] -- File: "/tmp/wireshark_anyBX3LM2.pcapng"
    1 0.000000000 192.168.200.6 → 10.0.101.1   TCP 76 46837 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=3606230691 TSecr=0 WS=128
    2 16.128014693 192.168.200.6 → 10.0.101.1   TCP 76 [TCP Retransmission] 46837 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=3606246819 TSecr=0 WS=128
    3 49.920022649 192.168.200.6 → 10.0.101.1   TCP 76 [TCP Retransmission] 46837 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=3606280611 TSecr=0 WS=128
    4 95.016865480 192.168.200.6 → 10.0.100.1   TCP 76 45471 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183065107 TSecr=0 WS=128
    5 96.032007782 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 45471 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183066123 TSecr=0 WS=128
    6 97.056001960 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 45471 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183067147 TSecr=0 WS=128
    7 98.079998304 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 45471 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183068171 TSecr=0 WS=128
    8 99.103985759 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 45471 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183069195 TSecr=0 WS=128
    9 100.127985586 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 45471 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183070219 TSecr=0 WS=128
   10 102.143989724 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 45471 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183072235 TSecr=0 WS=128
   11 106.243993153 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 45471 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183076335 TSecr=0 WS=128
   12 308.449917075 192.168.200.6 → 10.0.101.1   TCP 76 35605 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=3606539140 TSecr=0 WS=128
   13 321.252258381 192.168.200.6 → 10.0.100.1   TCP 76 55107 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183291343 TSecr=0 WS=128
   14 322.272022752 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 55107 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183292363 TSecr=0 WS=128
   15 323.295993608 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 55107 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183293387 TSecr=0 WS=128
   16 324.319992530 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 55107 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183294411 TSecr=0 WS=128
   17 325.343982741 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 55107 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183295435 TSecr=0 WS=128
   18 326.368002462 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 55107 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183296459 TSecr=0 WS=128
   19 328.383998611 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 55107 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183298475 TSecr=0 WS=128
   20 332.544003559 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 55107 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183302635 TSecr=0 WS=128
   21 340.736002576 192.168.200.6 → 10.0.100.1   TCP 76 [TCP Retransmission] 55107 → 179 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=4183310827 TSecr=0 WS=128
5

The routes are announced to what ever routers your have connected to LXD’s BGP server for that uplink network.

So in theory you might be able to setup a BGP server like bird that peers with LXD and then sets up the routes on the local host.

6

But equally you could just add static routes on the host that route towards the OVN network’s external volatile IP on the uplink network lxc network show <ovn network>.

7

But the OVN network must have ipv4.nat disabled to include them in the BGP routes announced by BGP.

8

Thanks tomp, if I’m understanding correctly, that clarifies what I found earlier today.

I was hoping the LXD BGP server would actually create the routes on the host systems themselves. allowing communication between different interfaces on the same host (or cluster) without needing to present it to the greater network.

Based on your comments and the 2:45-3:00 mark on this official LXD video, LXD BGP just does announcements, no actual configuration on the hosts.

https://www.youtube.com/watch?v=C9zU-FEqtTw

For my immediate purposes I’ll just stick with static routes configured with Ansible for now, as that seems to work as expected.

I’ll revisit configuring my routers with BGP at a later time for a fully automated approach. Maybe I’ll be ambitious enough to peek into the source code and try building something myself to handle the host-only approach :stuck_out_tongue:

Thanks again.

1 Like