Ubuntu LTS server 24.04 KVM and internal routing

Support and Help
, ,
1

Environment : High end PC with a Single NIC

OS Ubuntu: LTS 24.04 Server

KVM host

I think I know the issue but I can’ t resolves it.

I have a pc with ubuntu server LTS 24.04 running KVM and multiple virtual guests. The PC has a single NIC which is running in bridge mode for KVM

One of the guests is another ubuntu LTS 24.04 server running as a router configured with 10 virtual nics all using the br0 bridge. The nics are configured 92.168.0.0/24 though to 192.168.10.0/24, each subnet on a single virtual nic.

Iptables is controlling forwarding between the interfaces, and works well with one exception

The vm which is running as a router has SSHD locked down to a single interface. And the lock down works perfectly

The issues is if I have a physical windows host (or virtual or any other type of host for that matter ) connected to let’s say the 192.168.4.0/24 network, for which this router vm is connected and routing, using ip 192.168.4.2, but virtualised to 192.168.4.1 using keepalived.

Despite ssh being blocked on 192.168.4.2. The windows host can reach the open ssh on 192.168.0.2, WITHOUT ROUTING. As proved by a tracert.

A tcp dump proves the traffic is routing via the host.

Ip tables f/w rules on the host or the vm router (to either stop forwarding or input) or inside kvm with nwfilter, do not stop the connection.

It looks like this connection is happening at layer 2 not layer 3 and in the kvm internal switch.

Apart from adding additional physical interfaces to the host and then having new bridges how can I stop this internal level 2 routing ??

2

When every tap-device for your VMs is attached to the same Linux bridge, br0, they all share one Layer-2 segment. A packet from a host on 192.168.4.0/24 that targets 192.168.0.2 never leaves the bridge: it is delivered directly to the tap of the router-VM. Because delivery happens inside the bridge at L2:

the host OS never sees the packet, so host-level iptables cannot filter it
the router-VM’s own kernel never sees the packet on its IP stack, so its iptables cannot filter it
libvirt’s default nwfilter does not help, because bridge net-filtering is off by default

Hence traceroute shows only one hop.

Ways to prevent that L2 “shortcut” while keeping a single physical NIC

  1. Put every subnet on its own bridge.
    Create br0, br1, br2, … on the host. In each VM definition, point the interface for 192.168.0.0/24 to br0, the interface for 192.168.1.0/24 to br1, and so on. Packets can no longer “jump” between bridges.

  2. Use VLANs.
    Create VLAN sub-interfaces on the host NIC (e.g., enp5s0.10, enp5s0.20). Create a bridge per VLAN and attach the VM taps to the correct bridge. The physical switch port must be configured as a VLAN trunk.

  3. Enable bridge netfilter so iptables can see the traffic.

    sudo sysctl -w net.bridge.bridge-nf-call-iptables=1
sudo sysctl -w net.bridge.bridge-nf-call-ip6tables=1
sudo sysctl -w net.bridge.bridge-nf-call-arptables=1

    After that, packets that pass through the bridge are processed by the host’s iptables and you can drop unwanted inter-tap traffic. There is a small performance hit.

  4. Use ebtables to filter at Layer-2.
    Example:

    sudo ebtables -A FORWARD -i vnet4 -o vnet0 -j DROP

    This blocks frames from the tap for 192.168.4.0/24 to the tap for 192.168.0.0/24. Repeat for every pair you want to isolate.

  5. Switch to Open vSwitch or enable libvirt port isolation.
    Both provide a “private port” feature so that taps cannot talk directly to each other on the same virtual switch.

If you need a quick fix for a small number of subnets, separate bridges are simplest. For many subnets on one cable, VLANs scale better. If you do not want to change the topology, enable bridge-netfilter (or use ebtables) and let the host firewall drop cross-subnet traffic. Once isolation is in place, the router-VM’s SSH daemon on 192.168.0.2 will be reachable only from its intended subnet.

3

Firstly many thanks,

so i tried additional bridges first not too much work, and i found netplan wont allow you to do multiple bridges per nic

Then i tried the vlan, and the network wes on link but i could not ping any other host on the network, including in the same subnet .

So i tried a number of tests.
I bought on line the spare nic (slower) enp5s0, where as my main nic is enx5c857e38e9b7

I used the following

network:
version: 2
renderer: networkd
ethernets:
enx5c857e38e9b7:
dhcp4: false
enp5s0:
dhcp4: false
bridges:
br0:
interfaces: [enx5c857e38e9b7]
dhcp4: no
addresses: [192.168.0.6/24]
routes:
- to: default
via: 192.168.0.1
metric: 100

  nameservers:
    search: [mydomain]
    addresses: [192.168.0.1,192.168.0.2,192.168.0.3]

vlans:
enp5s0.10: # e.g., eth0.10 for VLAN 10
id: 10 # e.g., 10
link: enp5s0
dhcp4: false
addresses: [192.168.4.26/24]

i had a win11 pc on the 192.168.4.0/24 subnet with an iip address 192.168.4.170 and i could not ping the vlan

WARNING I DO NOT HAVE MANAGED SWUITCHES , but as far as i understand they just pass but ignore the tagging ?

If i did a tcpsump on the server i could see repeated

17:10:22.957499 ARP, Request who-has 192.168.4.170 tell 192.168.4.26, length 28

In order to test the nic was actually working and i had connectivity i tried the following config

network:
version: 2
renderer: networkd
ethernets:
enx5c857e38e9b7:
dhcp4: false
enp5s0:
dhcp4: false
addresses: [192.168.4.26/24]

bridges:
br0:
interfaces: [enx5c857e38e9b7]
dhcp4: no
addresses: [192.168.0.6/24]
routes:
- to: default
via: 192.168.0.1
metric: 100

  nameservers:
    search: [mydomainname]
    addresses: [192.168.0.1,192.168.0.2,192.168.0.3]

and i could ping, so all the connectivity works.

I also tried physical nic , bridge then vlan inclase it was my switched upsetting the vlam.

network:
version: 2
renderer: networkd
ethernets:
enx5c857e38e9b7:
dhcp4: false
enp5s0:
dhcp4: false
bridges:
br0:
interfaces: [enx5c857e38e9b7]
dhcp4: no
addresses: [192.168.0.6/24]
routes:
- to: default
via: 192.168.0.1
metric: 100

  nameservers:
    search: [mydomain]
    addresses: [192.168.0.1,192.168.0.2,192.168.0.3]
br1:
  dhcp4: no

vlans:
br1.10: # e.g., eth0.10 for VLAN 10
id: 10 # e.g., 10
link: br1
dhcp4: false
addresses: [192.168.4.26/24]

exactly the same result as with nic then vlan…

Any ideas what i have done wrong ?

4

Nothing is wrong with Netplan at all – the packets just never reach the
other host because they are now tagged with an 802.1Q header and your LAN
doesn’t understand those tags.

Why your pings stopped working

The stanza

vlans:
  enp5s0.10:
    id: 10
    link: enp5s0

makes the kernel add VLAN-10 tags to every frame that leaves
enp5s0.10.

Your Windows box is still sending and listening for untagged
frames on 192.168.4.0/24.

The little unmanaged (“dumb”) switch in between strips nothing, adds
nothing, simply drops the tagged frames (most cheap switches treat
unknown ethertypes as errors).

So the ARP you saw in tcpdump (who-has 192.168.4.170) never
returns – Windows never saw it.

Three ways forward

Keep the unmanaged switch – don’t use VLAN tagging

Put each subnet on its own Linux bridge instead of VLANs.
An interface may live in only one bridge, so you create one bridge per
VLAN and enslave a tagged sub-interface to each bridge:

network:
  version: 2
  renderer: networkd

  ethernets:
    enp5s0: {}

  vlans:
    enp5s0.4:  { id: 4,  link: enp5s0 }
    enp5s0.5:  { id: 5,  link: enp5s0 }
    enp5s0.10: { id: 10, link: enp5s0 }

  bridges:
    br4:
      interfaces: [enp5s0.4]
    br5:
      interfaces: [enp5s0.5]
    br10:
      interfaces: [enp5s0.10]

Inside libvirt point each VM NIC at the proper bridge.
Because enp5s0 stays untagged on the wire the dumb switch is happy.

Keep one bridge, enable L2 filtering instead

Leave everything on one bridge (br0) and enable bridge–netfilter so
the host’s iptables can drop cross-subnet traffic:

echo "net.bridge.bridge-nf-call-iptables=1" | sudo tee /etc/sysctl.d/99-brfilter.conf
sudo sysctl -p /etc/sysctl.d/99-brfilter.conf

Then add rules such as:

sudo iptables -I FORWARD -i vnet4 -o vnet0 -j DROP

to stop 192.168.4.0/24 from talking to 192.168.0.0/24.

Buy a managed switch (or configure VLAN on the Windows NIC)

If you really want 802.1Q isolation, the switch (or every endpoint) must
understand VLAN tags. Either:

replace the unmanaged switch with a cheap managed one and set the port
that goes to the Ubuntu host as a trunk, or
configure the Windows NIC for VLAN 10 (Intel/Realtek panels allow
this).

Once both ends tag traffic with the same VLAN ID your original config
will work.

Pick the option that fits your hardware budget and complexity
tolerance. Without VLAN-aware gear the simplest route is multiple
bridges or host-level iptables filtering.

5

Thanks for the help unfortunately none of those worked :frowning:

But I did fix it with macvtap

First remove all bridges.

In /etc/systemd/network define each macvtap nic like below for private mode

[NetDev]
Name=mvt0
Kind=macvtap
[MACVTAP]
Mode=private

make sure the files are in the following format 00-mvt0.netdev to make sure the nics are there before netplan checks

or as below if you need to connect vms to your physical host (just do this for the nic the physical host is on).

[NetDev]
Name=net0
Kind=macvtap
[MACVTAP]
Mode=bridge

then map them in a file using the following format 00-primary.network

[Match]
Name=

[Network]
MACVTAP=net0
MACVTAP=net1
MACVTAP=mvt0
MACVTAP=mvt1

If the the macvtap nics come online the nic drops offline check the drivers on the kvm host.

if your using multicast from a KVM vm you need to enable multicast using trustGuestRxFilters=‘yes’

You can then block any remaining cross network communication using iptables INPUT chain on the KVM host.

This all works on a single NIC with unmanaged switches.

6

If the issues are resolved please mark the topic with the relevant solution.

Thanks.

7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.