Ubuntu Version:
Desktop Environment (if applicable):
Problem Description:
I am building an Ubuntu router with 1 WAN interface, and 3 LAN interfaces. I would like all interfaces to be able to communicate with each other for the time being, so I have created custom routes using the table directive.
- I have have bonded (802.3ad) and then bridged all the LAN ports. Each LAN interface acts as its own gateway.
- Each LAN will be connected to its own switch with IP ending in .254 for each corresponding LAN subnet (i.e LAN1 has gateway 192.168.0.1 connected to switch with IP 192.168.0.254).
- All LANs are routed through each other and to the internet.
Relevant System Information:
Machine: Dell R730xd, 256 Gb DDR4 LRDIMM, Dual E5-2690 V4, 250 GB storage + 64 GB boot
eno1: Single Broadcom 10GBase-T (WAN)
eno2: Single Broadcom 10GBase-T (Management) ----> operating on 192.168.1.1/24
br0: Dual 100 GBe ConnectX-4 Mellanox (LAN 1) —> subnet 192.168.0.0/16
br1: Dual 100 GBe ConnectX-5 Mellanox (LAN 2) ----> subnet 10.0.0.0/8 & subnet 203.0.113.0/24
br2: Dual Broadcom 10Gb sfp+ (LAN 3) -----> subnet 176.12.0.0/16
The router IP is 192.168.0.1
Configuration .yaml file:
network:
version: 2
renderer: networkd
ethernets:
eno1: #WAN
dhcp4: false
dhcp6: false
optional: false
addresses:
- XXX.XXX.XXX.XXX/24 # Static IP from ISP
routes:
- to: default # Only the wan has default route as the route of last resort
via: XXX.XXX.XXX.XXX # ISP Gateway
nameservers:
search: [gupta.dev]
addresses: [XXX.XXX.XXX.XXX, 8.8.8.8]
eno2: # Management Port directly connected to switch on 192.168.1.0 subnet
dhcp4: false
dhcp6: false
optional: false
addresses: [192.168.0.194/24]
nameservers:
addresses: [192.168.0.1, XXX.XXX.XXX.XXX]
routes:
- to: 192.168.0.0/24
via: 192.168.0.254 # This interface will go to the switch to provide connectivity and will not serve as a gateway
table: 200 #Assigns this route to table 200
metric: 500
eno3:
dhcp4: false
dhcp6: false
optional: true
eno4:
dhcp4: false
dhcp6: false
optional: true
enp4s0f0np0:
dhcp4: false
dhcp6: false
optional: true
enp4s0f0np1:
dhcp4: false
dhcp6: false
optional: true
enp13s0f0np0:
dhcp4: false
dhcp6: false
optional: true
enp13s0f0np1:
dhcp4: false
dhcp6: false
optional: true
bonds:
bond0:
interfaces: [eno3, eno4]
parameters:
lacp-rate: fast
mode: 802.3ad
transmit-hash-policy: layer2+3
mii-monitor-interval: 1
bond1:
interfaces: [enp4s0f0np0, enp4s0f0np1]
parameters:
lacp-rate: fast
mode: 802.3ad
transmit-hash-policy: layer2+3
mii-monitor-interval: 1
bond2:
interfaces: [enp13s0f0np0, enp13s0f0np1]
parameters:
lacp-rate: fast
mode: 802.3ad
transmit-hash-policy: layer2+3
mii-monitor-interval: 1
bridges:
br0:
dhcp4: no
dhcp6: no
interfaces: [bond0]
addresses: [192.168.0.1/16]
nameservers:
search: [gupta.dev]
addresses: [192.168.0.1, 69.59.32.9]
parameters:
stp: true
forward-delay: 0
routes:
- to: 192.168.0.0/16
via: 192.168.0.1
metric: 100
table:100
scope: link
on-link: true #no need to specify the gateway because the gateway is directly reachable on the br0 interface
br1: # will have two subnets on one physical interface for Openstack
dhcp4: no
dhcp6: no
interfaces: [bond1]
addresses:
- 10.0.0.1/8
label: "openstack-management"
- 203.0.113.1/24
label: "openstack-provider"
nameservers:
search: [openstack.gupta.dev]
addresses: [192.168.0.1, XXX.XXX.XXX.XXX]
parameters:
stp: true
forward-delay: 0
routes:
- to: 10.0.0.0/8
via: 10.0.0.1 # Gateway for 10.0.0.0 subnet
table: 101
metric: 100 # This subnet takes precedence due to lower metric
scope: link
on-link: true
- to: 203.0.113.0/24
via: 203.0.113.1 # Gateway for 203.0.113.0 subnet
table: 102
metric: 200
br2:
dhcp4: no
dhcp6: no
interfaces: [bond2]
addresses: [172.16.0.1/16]
nameservers:
search: [gupta.host]
addresses: [192.168.0.1, XXX.XXX.XXX.XXX]
parameters:
stp: true
forward-delay: 0
routes:
- to: 172.16.0.0/16
via: 172.16.0.1
table: 103
metric: 100
scope: link
on-link: true
routing-policy: # This is placed at the end to ensure ALL interfaces are interconnected
- from: 192.168.0.0/16
table: 101 # Allow br0 to route through br1 (10 subnet)
- from: 192.168.0.0/16
table: 102 # Allow br0 to route through br1 (203 subnet)
- from: 192.168.0.0/16
table: 103 # Allow br0 to route through br2
- from: 10.0.0.0/8
table: 100 # Allow br1 (10 subnet) to route through br0
- from: 10.0.0.0/8
table: 102 # Allow br1 (10 subnet) to route through br1 (203 subnet)
- from: 10.0.0.0/8
table: 103 # Allow br1 (10 subnet) to route through br2
- from: 203.0.113.0/24
table: 100 # Allow br1 (203 subnet) to route through br0
- from: 203.0.113.0/24
table: 100 # Allow br1 (203 subnet) to route through br1 (10 subnet)
- from: 203.0.113.0/24
table: 100 # Allow br1 (203 subnet) to route through br2
- from: 172.16.0.0/16
table: 100 # Allow br2 to route through br0
- from: 172.16.0.0/16
table: 101 # Allow br2 to route through br1 (10 subnet)
- from: 172.16.0.0/16
table: 102 # Allow br2 to route through br0 (203 subnet)
What I’ve Tried:
I have not yet applied the configuration as I want to verify the my logic and understanding is correct. The areas highlighted in bold are the ones where I am still struggling from a knowledge perspective, including the routing table. Any help or input is appreciated.
My Question:
My question is this: Is my reasoning correct with respect to this configuration? I have placed comments as much as possible to explain my reasoning, but is my understanding correct? Have I suceeded in getting all LANs to communicate with each other creating one big network?
- Specifically I am unsure about the default route usage. It is my understanding that one configuration can only have ONE default route as it is the route of last resort. Am I correct to place this on the WAN interface? Can there be more than one default route (i.e can each interface have a default route?)
- Am I using the metric variable correctly? can I assign a metric of 100 starting at each LAN interface?
- Is the stp:true parameter necessary here?
- Are scope: link and on-link: true statements necessary?
- I eventually hope to segment the traffic on each LAN. Is it my understanding that deleting the routing policy would accomplish this?