Netplan 0.106 - call for testing

slyon · February 15, 2023, 1:58pm

I’m happy to announce that Netplan version 0.106 is now available in lunar-proposed.
See release announcement: https://blog.slyon.de/2023/02/15/netplan-v0-106-is-now-available/

Highlights

Highlights of this release include the new netplan status command, which queries your system for IP addresses, routes, DNS information, etc… in addition to the Netplan backend renderer (NetworkManager/networkd) in use and the relevant Netplan YAML configuration ID. It displays all this in a nicely formatted way (or alternatively in machine readable YAML/JSON format).

Furthermore, we implemented a clean libnetplan API which can be used by external tools to parse Netplan configuration, migrated away from non-inclusive language (PR#303) and improved the overall Netplan documentation. Another change that should be noted, is that the match.macaddress stanza now only matches on PermanentMACAddress= on the systemd-networkd backend, as has been the case on the NetworkManager backend ever since.

Call for testing

We’d like you to apt install -t lunar-proposed netplan.io and give the new netplan status command a try. In addition, we’d like to ask for testing of the slight change in behavior about PermanentMACAddress matching. Please see PR#278 for background information on this change and let us know in this thread if you can still match your physical devices as expected using Netplan’s match.macaddress stanza.

kajiya · April 11, 2023, 10:23pm

Incoming test results for GCE!

Testing was done for the netplan.io v0.106 package on both jammy and lunar in GCE (package versions 0.106-0ubuntu1~22.04.1~ppa1 and 0.106-0ubuntu3 respectively)

The first set of tests was CPC’s automatic testing done for all images, a.k.a cloud-test-framework. All the tests passed for these custom images (the only failures were regarding packages being “ahead” of the archive in version/PPAs being referenced in the image, which is to be expected in this situation).

Now for the manual testing…
For both suites a n2-standard-8 instance (Intel Cascade Lake) was launched with 4 NICs configured (different VPCs, internal subnets, etc). The results can be found below…

LUNAR > netplan status --all

$ netplan status --all
     Online state: online
    DNS Addresses: 127.0.0.53 (stub)
       DNS Search: c.ubuntu-kajiya.internal
                   google.internal


●  1: lo ethernet UNKNOWN/UP (unmanaged)
      MAC Address: 00:00:00:00:00:00
        Addresses: 127.0.0.1/8
                   ::1/128
           Routes: ::1 metric 256


●  2: ens4 ethernet UP (networkd: ens4)
      MAC Address: 42:01:0a:80:00:08 (Google, Inc.)
        Addresses: 10.128.0.8/32 (dhcp)
                   fe80::4001:aff:fe80:8/64 (link)
    DNS Addresses: 169.254.169.254
       DNS Search: c.ubuntu-kajiya.internal
                   google.internal
           Routes: default via 10.128.0.1 from 10.128.0.8 metric 100 (dhcp)
                   10.128.0.0/20 via 10.128.0.1 from 10.128.0.8 metric 100 (dhcp)
                   10.128.0.1 from 10.128.0.8 metric 100 (dhcp, link)
                   169.254.169.254 via 10.128.0.1 from 10.128.0.8 metric 100 (dhcp)
                   fe80::/64 metric 256


●  3: ens5 ethernet UP (networkd: id0)
      MAC Address: 42:01:15:00:00:05 (Google, Inc.)
        Addresses: 21.0.0.5/32 (dhcp)
                   fe80::4001:15ff:fe00:5/64 (link)
    DNS Addresses: 169.254.169.254
       DNS Search: c.ubuntu-kajiya.internal
                   google.internal
           Routes: 21.0.0.0/8 via 21.0.0.1 (boot)
                   21.0.0.0/8 via 21.0.0.1 from 21.0.0.5 metric 100 (dhcp)
                   21.0.0.1 (boot, link)
                   21.0.0.1 from 21.0.0.5 metric 100 (dhcp, link)
                   169.254.169.254 from 21.0.0.5 metric 100 (dhcp, link)
                   fe80::/64 metric 256


●  4: ens6 ethernet UP (networkd: id0)
      MAC Address: 42:01:16:00:00:05 (Google, Inc.)
        Addresses: 22.0.0.5/32 (dhcp)
                   fe80::4001:16ff:fe00:5/64 (link)
    DNS Addresses: 169.254.169.254
       DNS Search: c.ubuntu-kajiya.internal
                   google.internal
           Routes: 22.0.0.0/8 via 22.0.0.1 (boot)
                   22.0.0.0/8 via 22.0.0.1 from 22.0.0.5 metric 100 (dhcp)
                   22.0.0.1 (boot, link)
                   22.0.0.1 from 22.0.0.5 metric 100 (dhcp, link)
                   169.254.169.254 from 22.0.0.5 metric 100 (dhcp, link)
                   fe80::/64 metric 256


●  5: ens7 ethernet UP (networkd: id0)
      MAC Address: 42:01:13:00:00:05 (Google, Inc.)
        Addresses: 19.0.0.5/32 (dhcp)
                   fe80::4001:13ff:fe00:5/64 (link)
    DNS Addresses: 169.254.169.254
       DNS Search: c.ubuntu-kajiya.internal
                   google.internal
           Routes: 19.0.0.0/8 via 19.0.0.1 (boot)
                   19.0.0.0/8 via 19.0.0.1 from 19.0.0.5 metric 100 (dhcp)
                   19.0.0.1 (boot, link)
                   19.0.0.1 from 19.0.0.5 metric 100 (dhcp, link)
                   169.254.169.254 from 19.0.0.5 metric 100 (dhcp, link)
                   fe80::/64 metric 256


●  6: docker0 bridge DOWN/UP (unmanaged)
      MAC Address: 02:42:2b:31:58:2a
        Addresses: 172.17.0.1/16
                   fe80::42:2bff:fe31:582a/64 (link)
           Routes: 172.17.0.0/16 from 172.17.0.1 (link)
                   fe80::/64 metric 256

JAMMY > netplan status --all

$ netplan status --all
     Online state: online
    DNS Addresses: 127.0.0.53 (stub)
       DNS Search: c.ubuntu-kajiya.internal
                   google.internal


●  1: lo ethernet UNKNOWN/UP (unmanaged)
      MAC Address: 00:00:00:00:00:00
        Addresses: 127.0.0.1/8
                   ::1/128
           Routes: ::1 metric 256


●  2: ens4 ethernet UP (networkd: ens4)
      MAC Address: 42:01:0a:80:00:0d (Google, Inc.)
        Addresses: 10.128.0.13/32 (dhcp)
                   fe80::4001:aff:fe80:d/64 (link)
    DNS Addresses: 169.254.169.254
       DNS Search: c.ubuntu-kajiya.internal
                   google.internal
           Routes: default via 10.128.0.1 from 10.128.0.13 metric 100 (dhcp)
                   10.128.0.0/20 via 10.128.0.1 from 10.128.0.13 metric 100 (dhcp)
                   10.128.0.1 from 10.128.0.13 metric 100 (dhcp, link)
                   169.254.169.254 via 10.128.0.1 from 10.128.0.13 metric 100 (dhcp)
                   fe80::/64 metric 256


●  3: ens5 ethernet UP (networkd: id0)
      MAC Address: 42:01:15:00:00:06 (Google, Inc.)
        Addresses: 21.0.0.6/32 (dhcp)
                   fe80::4001:15ff:fe00:6/64 (link)
    DNS Addresses: 169.254.169.254
       DNS Search: c.ubuntu-kajiya.internal
                   google.internal
           Routes: 21.0.0.0/8 via 21.0.0.1 (boot)
                   21.0.0.0/8 via 21.0.0.1 from 21.0.0.6 metric 100 (dhcp)
                   21.0.0.1 (boot, link)
                   21.0.0.1 from 21.0.0.6 metric 100 (dhcp, link)
                   fe80::/64 metric 256


●  4: ens6 ethernet UP (networkd: id0)
      MAC Address: 42:01:16:00:00:06 (Google, Inc.)
        Addresses: 22.0.0.6/32 (dhcp)
                   fe80::4001:16ff:fe00:6/64 (link)
    DNS Addresses: 169.254.169.254
       DNS Search: c.ubuntu-kajiya.internal
                   google.internal
           Routes: 22.0.0.0/8 via 22.0.0.1 (boot)
                   22.0.0.0/8 via 22.0.0.1 from 22.0.0.6 metric 100 (dhcp)
                   22.0.0.1 (boot, link)
                   22.0.0.1 from 22.0.0.6 metric 100 (dhcp, link)
                   fe80::/64 metric 256


●  5: ens7 ethernet UP (networkd: id0)
      MAC Address: 42:01:13:00:00:06 (Google, Inc.)
        Addresses: 19.0.0.6/32 (dhcp)
                   fe80::4001:13ff:fe00:6/64 (link)
    DNS Addresses: 169.254.169.254
       DNS Search: c.ubuntu-kajiya.internal
                   google.internal
           Routes: 19.0.0.0/8 via 19.0.0.1 (boot)
                   19.0.0.0/8 via 19.0.0.1 from 19.0.0.6 metric 100 (dhcp)
                   19.0.0.1 (boot, link)
                   19.0.0.1 from 19.0.0.6 metric 100 (dhcp, link)
                   fe80::/64 metric 256


●  6: docker0 bridge DOWN/UP (unmanaged)
      MAC Address: 02:42:74:2a:5e:76
        Addresses: 172.17.0.1/16
           Routes: 172.17.0.0/16 from 172.17.0.1 (link)

LUNAR > netplan get

ubuntu@lunar-manual-netplan-io-testing:~$ sudo !!
sudo netplan get


** (process:2390): WARNING **: 16:35:18.020: Permissions for /etc/netplan/90-gce-predictable-interface-name.yaml are too open. Netplan configuration should NOT be accessible by others.
network:
  version: 2
  ethernets:
    ens4:
      match:
        macaddress: "42:01:0a:80:00:41"
      dhcp4: true
      set-name: "ens4"
    id0:
      match:
        name: "ens*"
      dhcp4: true
      dhcp6: true

JAMMY > netplan get

ubuntu@jammy-manual-netplan-io-testing:~$ netplan get


** (process:2401): WARNING **: 16:36:00.764: Permissions for /etc/netplan/50-cloud-init.yaml are too open. Netplan configuration should NOT be accessible by others.


** (process:2401): WARNING **: 16:36:00.764: Permissions for /etc/netplan/90-gce-predictable-interface-name.yaml are too open. Netplan configuration should NOT be accessible by others.
network:
  version: 2
  ethernets:
    ens4:
      match:
        macaddress: "42:01:0a:80:00:51"
      dhcp4: true
      set-name: "ens4"
    id0:
      match:
        name: "ens*"
      dhcp4: true
      dhcp6: true

I hope this is useful? Please let me know if you need any further testing/info

chad.smith · June 5, 2023, 4:06pm

It’s worth noting that cloud-init integration tests fail on Lunar and Mantic (netplan 0.106) when custom network config attempts to match: by macaddress: on LXD containers due to veth interfaces no longer being managed because netplan v. 0.106 now uses PermanentMACAddress instead of MACAddress in systemd/networkd [Match] sections.

Note: this doesn’t affect LXD VMs which have physical interfaces. It also doesn’t affect default network config generated by cloud-init on LXD containers as no match conditions are used by default.

# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        eth0:
            dhcp4: true

I’ve filed a LP: # 2022947 representing this behavior for tracking just so we can reference it and provide intended workarounds if needed. I’m not certain if this is critical or not for SRU, but it does represent that anyone providing network config which tries to match by mac for veth devices may find themselves broken and need to find an alternative network-config match clause for their environments. By default cloud-init on LXD doesn’t provide a “match” clause

fabio.martins · July 13, 2023, 12:13pm

I did some netplan 0.106 tests on Oracle cloud running a couple of Jammy instances. Overall everything is working well.

I’ve installed netplan.io 0.106 from the PPA and configured 2 VNICs in the same subnet with some policy routing, with the match.macaddress stanza. My initial tests were done with 2x VM.Standard.E4.Flex instances:

Instance 1:

Instance 2:

And everything is working well for both interfaces:

PS: In order for this to work, I had to use Oracle’s secondary_vnic_all_configure.sh [1] to query the instance metadata and get the IP address (it also configures the interface with “ip” command), and then add the settings to netplan’s yaml file.

I also tested with VNICs in different subnets, following the same logic, and all worked well:

Instance 1:

Instance 2:

I then did some tests with 1x BM.Standard.E4.128, which by default has 2 x 50 Gbps interfaces (Mellanox Technologies MT2892 Family [ConnectX-6 Dx])

First, I’ve configured it with 2 different IPs (again, using secondary_vnic_all_configure.sh to get my secondary IP, and then setting it up in netplan yaml):

And everything worked well, I can ping my both IPs from another instance in the same subnet:

I also tested a different / unusual setup in Oracle Cloud:

In bare metal instances, you need to create a VNIC in the cloud and associate it with a physical NIC in the host. That will add VLAN tagging on the Oracle Cloud infrastructure.

In order to understand how the Ubuntu setup should look like, I once again used Oracle’s secondary_vnic_all_configure.sh, and I noticed it adds a macvlan interface and then adds a VLAN ID (tag) on top of it.

Since apparently macvlan is not supported in netplan [2], we need to use some hook scripts to setup the interface. In the end it worked well, but I had to apply some workarounds:

Ping works from another instance:

Anyway, I think these workarounds are outside the scope of this test, but even in this unusual scenario, netplan 0.106 still worked well.

[1] https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/managingVNICs.htm#Linux
[2] https://bugs.launchpad.net/netplan/+bug/1664847

fabio.martins · July 13, 2023, 2:05pm

I’ve performed the netplan.io 0.106 tests in AWS instances and all worked well.

Tested with 1x m4.2xlarge instance w/ 2 ENI interfaces + 1x c5a.2xlarge w/ 2 ENI interfaces and also with 1 c6i.metal w/ 1x ENI + 1 EFA interface. Applied policy based routing and all interfaces worked well and responds to pings:

VM1:

ubuntu@ip-172-31-5-198:~$ sudo apt-cache policy netplan.io
netplan.io:
  Installed: 0.106-0ubuntu1~22.04.1~ppa1
  Candidate: 0.106-0ubuntu1~22.04.1~ppa1
  Version table:
 *** 0.106-0ubuntu1~22.04.1~ppa1 500
        500 https://ppa.launchpadcontent.net/slyon/netplan/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     0.105-0ubuntu2~22.04.3 500
        500 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     0.104-0ubuntu2 500
        500 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
ubuntu@ip-172-31-5-198:~$ sudo netplan status
     Online state: online
    DNS Addresses: 127.0.0.53 (stub)
       DNS Search: sa-east-1.compute.internal

●  1: lo ethernet UNKNOWN/UP (unmanaged)
      MAC Address: 00:00:00:00:00:00
        Addresses: 127.0.0.1/8
                   ::1/128
           Routes: ::1 metric 256

●  2: ens5 ethernet UP (networkd: ens5)
      MAC Address: 02:f4:45:9e:0d:20 (Amazon.com, Inc.)
        Addresses: 172.31.5.198/20 (dhcp)
                   fe80::f4:45ff:fe9e:d20/64 (link)
    DNS Addresses: 172.31.0.2
       DNS Search: sa-east-1.compute.internal
           Routes: default via 172.31.0.1 from 172.31.5.198 metric 100 (dhcp)
                   172.31.0.0/20 from 172.31.5.198 metric 100 (link)
                   172.31.0.1 from 172.31.5.198 metric 100 (dhcp, link)
                   172.31.0.2 from 172.31.5.198 metric 100 (dhcp, link)
                   fe80::/64 metric 256

●  3: ens6 ethernet UP (networkd: ens6)
      MAC Address: 02:62:de:c7:b2:a2 (Amazon.com, Inc.)
        Addresses: 172.31.8.248/20 (dhcp)
                   fe80::62:deff:fec7:b2a2/64 (link)
    DNS Addresses: 172.31.0.2
       DNS Search: sa-east-1.compute.internal
           Routes: default via 172.31.0.1 from 172.31.8.248 metric 200 (dhcp)
                   172.31.0.0/20 from 172.31.8.248 metric 200 (link)
                   172.31.0.1 from 172.31.8.248 metric 200 (dhcp, link)
                   172.31.0.2 from 172.31.8.248 metric 200 (dhcp, link)
                   fe80::/64 metric 256

ubuntu@ip-172-31-5-198:~$ sudo dmidecode  | grep "Product Name"
	Product Name: c5a.2xlarge
	Product Name: Not Specified
ubuntu@ip-172-31-5-198:~$ sudo lspci | grep Ethernet
00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
00:06.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)

VM2:

ubuntu@ip-172-31-2-45:~$ sudo apt-cache policy netplan.io
netplan.io:
  Installed: 0.106-0ubuntu1~22.04.1~ppa1
  Candidate: 0.106-0ubuntu1~22.04.1~ppa1
  Version table:
 *** 0.106-0ubuntu1~22.04.1~ppa1 500
        500 https://ppa.launchpadcontent.net/slyon/netplan/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     0.105-0ubuntu2~22.04.3 500
        500 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     0.104-0ubuntu2 500
        500 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
ubuntu@ip-172-31-2-45:~$ sudo netplan status
     Online state: online
    DNS Addresses: 127.0.0.53 (stub)
       DNS Search: sa-east-1.compute.internal

●  1: lo ethernet UNKNOWN/UP (unmanaged)
      MAC Address: 00:00:00:00:00:00
        Addresses: 127.0.0.1/8
                   ::1/128
           Routes: ::1 metric 256

●  2: ens3 ethernet UP (networkd: ens3)
      MAC Address: 02:6e:be:f4:46:7c (Intel Corporation)
        Addresses: 172.31.2.45/20 (dhcp)
                   fe80::6e:beff:fef4:467c/64 (link)
    DNS Addresses: 172.31.0.2
       DNS Search: sa-east-1.compute.internal
           Routes: default via 172.31.0.1 from 172.31.2.45 metric 100 (dhcp)
                   172.31.0.0/20 from 172.31.2.45 metric 100 (link)
                   172.31.0.1 from 172.31.2.45 metric 100 (dhcp, link)
                   172.31.0.2 from 172.31.2.45 metric 100 (dhcp, link)
                   fe80::/64 metric 256

●  3: ens4 ethernet UP (networkd: ens4)
      MAC Address: 02:2b:e6:24:ae:06 (Intel Corporation)
        Addresses: 172.31.14.75/20 (dhcp)
                   fe80::2b:e6ff:fe24:ae06/64 (link)
    DNS Addresses: 172.31.0.2
       DNS Search: sa-east-1.compute.internal
           Routes: default via 172.31.0.1 from 172.31.14.75 metric 200 (dhcp)
                   172.31.0.0/20 from 172.31.14.75 metric 200 (link)
                   172.31.0.1 from 172.31.14.75 metric 200 (dhcp, link)
                   172.31.0.2 from 172.31.14.75 metric 200 (dhcp, link)
                   fe80::/64 metric 256


ubuntu@ip-172-31-2-45:~$ sudo lspci | grep Ethernet
00:03.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:04.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

ubuntu@ip-172-31-2-45:~$ curl http://169.254.169.254/latest/meta-data/instance-type
m4.2xlargeubuntu@ip-172-31-2-45:~$

Bare Metal instance:

ubuntu@ip-172-31-6-38:~$ sudo netplan status
     Online state: online
    DNS Addresses: 127.0.0.53 (stub)
       DNS Search: sa-east-1.compute.internal

●  1: lo ethernet UNKNOWN/UP (unmanaged)
      MAC Address: 00:00:00:00:00:00
        Addresses: 127.0.0.1/8
                   ::1/128
           Routes: ::1 metric 256

●  2: ens1 ethernet UP (networkd: ens1)
      MAC Address: 02:a6:4a:40:39:60 (Amazon.com, Inc.)
        Addresses: 172.31.6.38/20 (dhcp)
                   fe80::a6:4aff:fe40:3960/64 (link)
    DNS Addresses: 172.31.0.2
       DNS Search: sa-east-1.compute.internal
           Routes: default via 172.31.0.1 from 172.31.6.38 metric 100 (dhcp)
                   172.31.0.0/20 from 172.31.6.38 metric 100 (link)
                   172.31.0.1 from 172.31.6.38 metric 100 (dhcp, link)
                   172.31.0.2 from 172.31.6.38 metric 100 (dhcp, link)
                   fe80::/64 metric 256

●  3: ens2 ethernet UP (networkd: ens2)
      MAC Address: 02:92:87:bc:02:b8 (Amazon.com, Inc.)
        Addresses: 172.31.4.200/20 (dhcp)
                   fe80::92:87ff:febc:2b8/64 (link)
    DNS Addresses: 172.31.0.2
       DNS Search: sa-east-1.compute.internal
           Routes: default via 172.31.0.1 from 172.31.4.200 metric 200 (dhcp)
                   172.31.0.0/20 from 172.31.4.200 metric 200 (link)
                   172.31.0.1 from 172.31.4.200 metric 200 (dhcp, link)
                   172.31.0.2 from 172.31.4.200 metric 200 (dhcp, link)
                   fe80::/64 metric 256

ubuntu@ip-172-31-6-38:~$ sudo dmidecode  | grep "Product Name"
	Product Name: Not Specified
	Product Name: c6i.metal

ubuntu@ip-172-31-6-38:~$ sudo lspci | grep Ethernet
2f:00.0 Ethernet controller: Amazon.com, Inc. Elastic Fabric Adapter (EFA)
30:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
31:00.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)

ubuntu@ip-172-31-6-38:~$ sudo apt-cache policy netplan.io
netplan.io:
  Installed: 0.106-0ubuntu1~22.04.1~ppa1
  Candidate: 0.106-0ubuntu1~22.04.1~ppa1
  Version table:
 *** 0.106-0ubuntu1~22.04.1~ppa1 500
        500 https://ppa.launchpadcontent.net/slyon/netplan/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     0.105-0ubuntu2~22.04.3 500
        500 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     0.104-0ubuntu2 500
        500 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

Ping tests (from the 1st VM pinging the 2nd VM and the baremetal instance):

ubuntu@ip-172-31-5-198:~$ ping -I ens5 172.31.2.45 -c 2
PING 172.31.2.45 (172.31.2.45) from 172.31.5.198 ens5: 56(84) bytes of data.
64 bytes from 172.31.2.45: icmp_seq=1 ttl=64 time=0.217 ms
64 bytes from 172.31.2.45: icmp_seq=2 ttl=64 time=0.149 ms

--- 172.31.2.45 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 0.149/0.183/0.217/0.034 ms

ubuntu@ip-172-31-5-198:~$ ping -I ens6 172.31.2.45 -c 2
PING 172.31.2.45 (172.31.2.45) from 172.31.8.248 ens6: 56(84) bytes of data.
64 bytes from 172.31.2.45: icmp_seq=1 ttl=64 time=0.195 ms
64 bytes from 172.31.2.45: icmp_seq=2 ttl=64 time=0.169 ms

--- 172.31.2.45 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.169/0.182/0.195/0.013 ms

ubuntu@ip-172-31-5-198:~$ ping -I ens5 172.31.14.75 -c 2
PING 172.31.14.75 (172.31.14.75) from 172.31.5.198 ens5: 56(84) bytes of data.
64 bytes from 172.31.14.75: icmp_seq=1 ttl=64 time=0.186 ms
64 bytes from 172.31.14.75: icmp_seq=2 ttl=64 time=0.156 ms

--- 172.31.14.75 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1012ms
rtt min/avg/max/mdev = 0.156/0.171/0.186/0.015 ms

ubuntu@ip-172-31-5-198:~$ ping -I ens6 172.31.14.75 -c 2
PING 172.31.14.75 (172.31.14.75) from 172.31.8.248 ens6: 56(84) bytes of data.
64 bytes from 172.31.14.75: icmp_seq=1 ttl=64 time=0.211 ms
64 bytes from 172.31.14.75: icmp_seq=2 ttl=64 time=0.147 ms

--- 172.31.14.75 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1021ms
rtt min/avg/max/mdev = 0.147/0.179/0.211/0.032 ms

ubuntu@ip-172-31-5-198:~$ ping -I ens5 172.31.6.38 -c 2
PING 172.31.6.38 (172.31.6.38) from 172.31.5.198 ens5: 56(84) bytes of data.
64 bytes from 172.31.6.38: icmp_seq=1 ttl=64 time=0.257 ms
64 bytes from 172.31.6.38: icmp_seq=2 ttl=64 time=0.297 ms

--- 172.31.6.38 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1010ms
rtt min/avg/max/mdev = 0.257/0.277/0.297/0.020 ms

ubuntu@ip-172-31-5-198:~$ ping -I ens6 172.31.6.38 -c 2
PING 172.31.6.38 (172.31.6.38) from 172.31.8.248 ens6: 56(84) bytes of data.
64 bytes from 172.31.6.38: icmp_seq=1 ttl=64 time=4.67 ms
64 bytes from 172.31.6.38: icmp_seq=2 ttl=64 time=0.295 ms

--- 172.31.6.38 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.295/2.482/4.669/2.187 ms

ubuntu@ip-172-31-5-198:~$ ping -I ens5 172.31.4.200 -c 2
PING 172.31.4.200 (172.31.4.200) from 172.31.5.198 ens5: 56(84) bytes of data.
64 bytes from 172.31.4.200: icmp_seq=1 ttl=64 time=0.427 ms
64 bytes from 172.31.4.200: icmp_seq=2 ttl=64 time=0.271 ms

--- 172.31.4.200 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1007ms
rtt min/avg/max/mdev = 0.271/0.349/0.427/0.078 ms

ubuntu@ip-172-31-5-198:~$ ping -I ens6 172.31.4.200 -c 2
PING 172.31.4.200 (172.31.4.200) from 172.31.8.248 ens6: 56(84) bytes of data.
64 bytes from 172.31.4.200: icmp_seq=1 ttl=64 time=0.307 ms
64 bytes from 172.31.4.200: icmp_seq=2 ttl=64 time=0.296 ms

--- 172.31.4.200 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1004ms
rtt min/avg/max/mdev = 0.296/0.301/0.307/0.005 ms