[Blog] Netplan developer diaries

Hi I’m the maintainer of Netplan.io, the tool for your network configuration needs.
Today I’d like to start this new topic which will contain stories and updates from the Netplan development team on a somewhat regular basis. You can expect a variety of topics to be covered, such as:

  • Announcements of (patch) releases
  • Introduction of new features
  • Interesting bugs & solutions
  • Conference presentations
  • Continued testing efforts
  • Spotlight on use cases
  • Mini tutorials

Feel free to reach out to us at any time!

Website: https://netplan.io
Documentation: netplan.rtfd.io
Questions: netplan tag on askubuntu.com
Chat: #netplan on libera.chat IRC
Code: github.com/canonical/netplan
Bugs: bugs.launchpad.net/netplan (Report)

3 Likes

In February 2023 we published our latest Netplan release v0.106, which contained an interesting new feature, called netplan status. This new sub-command queries your system for IP addresses, routes, DNS information, etc… in addition to the Netplan backend renderer (NetworkManager/networkd) in use and the relevant Netplan YAML configuration ID. It displays all this in a nicely formatted way (or alternatively in machine-readable YAML/JSON format).

This new feature was presented in more detail at the Ubuntu engineering sprint in Prague (May 2023). Please find the video below to watch this lightning talk:

2 Likes

Netplan 0.106.1 release

We are happy to announce that Netplan 0.106.1 is available for download on Ubuntu Mantic Minotaur and Debian testing.

This release includes some improvements in our documentation and CI infrastructure and a number of bug fixes.

What’s new in Netplan 0.106.1?

Documentation

Infrastructure

  • canonical/setup-lxd GitHub action. The autopkgtest environment creation was standardized to use Canonical’s setup-lxd action.
  • Snapd integrations tests with spread. A new test set for the Snapd integration with Netplan was introduced using the spread tool.
  • DBus. A number of DBus integration tests were added to the Debian package.

New features

  • Keyfile parser improvements. Our Network Manager keyfile parser (the capability of loading Network Manager configuration to Netplan YAML) was expanded to support all the types of tunnels supported by Netplan.

Misc

  • Ubuntu’s Code of Conduct 2.0 was added to the code repository.
  • We added a new bash autocompletion script with all the Netplan’s subcommands.
  • The new release package was synchronized with Debian.

Bug fixes

  • Keyfile parser. This release contains a couple of important fixes for the NetworkManager integration stability: 1) adding WPA enterprise connections is now working fine and new test cases were added to the package; 2) a WireGuard peer with allowed IPs that don’t include the network prefix are now accepted.
  • Netplan parser. A number of memory leaks and stability issues were fixed.
  • DBus. An issue related to how directory paths are built in the Netplan DBus service was causing issues in the Snapd integration and was fixed.

For the complete list of changes please consult the debian/changelog file in https://launchpad.net/ubuntu/+source/netplan.io/+changelog

1 Like

Testing Netplan’s Desktop integration with NetworkManager

Netplan.io is the “single source of truth” for controlling Ubuntu’s network stack, which means you can find all the information about your network configuration in /etc/netplan/ (or using sudo netplan get) on Ubuntu Server, Ubuntu Cloud images, Ubuntu Core IoT, etc. On Ubuntu Desktop, we deviated a bit from this approach in the past, as NetworkManager instead stored its “keyfile” settings in /etc/NetworkManager/system-connections/.

This changed in Ubuntu 23.10 “Mantic Minotaur” (the current development release), where NetworkManager is able to feed back information to Netplan’s YAML config (for example a WiFi password that was changed in the settings GUI), through a bidirectional integration where NetworkManager can talk to Netplan and vice versa.

I’d like to ask anybody running Ubuntu Mantic to help testing this new integration!
The Netplan integration will be automatically installed and activated on your system, when running NetworkManager 1.42.4-1ubuntu3 or above.

Note: When upgrading, NetworkManager will automatically create backups of your original keyfiles, stored in /var/lib/NetworkManager/backups/ and convert your connection profiles to Netplan YAML in /etc/netplan/.

Testing scenarios

Conversion on upgrade/installation

Whenever you add, modify or delete a connection in NetworkManager (e.g. via settings UI, D-Bus API, nmcli, …) it will be reflected in Netplan’s YAML configuration (i.e. sudo netplan get).

So after installation/upgrade, you should check the contents of /etc/netplan/90-NM-*.yaml and verify everything is properly converted into /run/NetworkManager/system-connections/ and properly visible in NetworkManager’s GUI, while /etc/NetworkManager/system-connections/ stays empty.

Ethernet connections

Check /etc/netplan for a .yaml file that already contains an ethernet connection and remove it. Then connect your system to a ethernet link via NetworkManager and check for the corresponding YAML to be created and make sure the contents look correct.

$ grep -ri ethernet /etc/netplan
/etc/netplan/90-NM-1b0dfe3a-2815-36aa-8e48-f883bbf16829.yaml:  ethernets:
/etc/netplan/90-NM-1b0dfe3a-2815-36aa-8e48-f883bbf16829.yaml:          ethernet._: ""
$ rm /etc/netplan/90-NM-1b0dfe3a-2815-36aa-8e48-f883bbf16829.yaml

## Plug into ethernet
$ cat /etc/netplan/90-NM-1b0dfe3a-2815-36aa-8e48-f883bbf16829.yaml
network:
  version: 2
  ethernets:
    NM-1b0dfe3a-2815-36aa-8e48-f883bbf16829:
      renderer: NetworkManager
      match:
        name: "enp0s31f6"
      dhcp4: true
      dhcp6: true
      ipv6-address-generation: "stable-privacy"
      wakeonlan: true
      networkmanager:
        uuid: "1b0dfe3a-2815-36aa-8e48-f883bbf16829"
        name: "Wired connection 1"
        passthrough:
          connection.autoconnect-priority: "-999"
          connection.timestamp: "1662490134"
          ethernet._: ""
          ipv6.ip6-privacy: "-1"
          proxy._: ""
          .nmmeta.nm-generated: "true"

WiFi connection

Using nmcli, create a new connection and look for an equivalent change under /etc/netplan/

$ nmcli con add con-name wifi1 type wifi ifname wifi1 ipv4.method manual ipv4.address 10.10.10.4/24 ipv4.gateway 10.10.10.1 ssid <ssid_name>

## should show your new wifi connection
$ nmcli con show
NAME         UUID                                  TYPE  DEVICE 
XXXXX        dffebd3b-f21e-4e0f-9ca0-59667d6ac3ac  wifi  wlp1s0 
XXXXX        ac0346ad-08f0-48d6-a071-11c7631d4374  wifi  --     
wifi1        f7adb62a-8088-4270-ac09-10755064cbb3  wifi  --

## Note there is a 90-NM-dffebd3.. for another wifi network that I've connected to
$ ls -l /etc/netplan 
total 12
-rw-r--r-- 1 root root 104 Apr 19 04:09 01-network-manager-all.yaml
-rw-r--r-- 1 root root 705 Sep 14 16:43 90-NM-dffebd3b-f21e-4e0f-9ca0-59667d6ac3ac.yaml
-rw-r--r-- 1 root root 676 Sep 12 15:28 90-NM-f7adb62a-8088-4270-ac09-10755064cbb3.yaml

See the new YAML file listed and open it up to look and make sure it looks correct

$ cat /etc/netplan/90-NM-f7adb62a-8088-4270-ac09-10755064cbb3.yaml
network:
  version: 2
  wifis:
    NM-f7adb62a-8088-4270-ac09-10755064cbb3:
      renderer: NetworkManager
      match:
        name: "wifi1"
      addresses:
      - "10.10.10.4/24"
      dhcp6: true
      ipv6-address-generation: "stable-privacy"
      access-points:
        "<ssid_name>":
          networkmanager:
            uuid: "f7adb62a-8088-4270-ac09-10755064cbb3"
            name: "wifi1"
            passthrough:
              ipv4.address1: "10.10.10.4/24,10.10.10.1"
              ipv4.method: "manual"
              ipv6.ip6-privacy: "-1"
              proxy._: ""
      networkmanager:
        uuid: "f7adb62a-8088-4270-ac09-10755064cbb3"
        name: "wifi1"

Insipration for further testing

In addition to standard WiFi and ethernet connections, we’d like to ask the community to test a wide variety of networking setups, as we cannot reproduce all the different setups that any of you might encounter in their daily lifes. Including, but not limited to:

  • Enterprise WiFi (e.g. eduroam)
  • WWAN (gsm/cdma modem connections)
  • WiFi AP mode (Fat, Fit, Cloud - and switching between them)
  • Different types of VPN connections (IPSec, WireGuard, VPNC, …)
  • Composite devices, like bridges or bonds
  • Virtual VLAN and Tunnel devices
  • DHCP address assignment (both IPv4 and IPv6)
  • Static IP assignment (both IPv4 and IPv6)
  • Shared network connectivity, e.g. a connection created with:
nmcli c add con-name shared type ethernet ifname enx000ec6e241bf ipv4.method shared ipv6.method ignore

Feedback

Please let us know about your experience using the Netplan desktop integration of NetworkManager. Be it positive or negative, your feedback is highly appreciated!

Please don’t hesitate to open a bug report and try to describe the setup you’ve been testing in detail and present any problems that might have occurred.

References

Please see Netplan’s documentation and this post for our previous, PPA based work:

2 Likes

Recent stability improvements

One of our goals for this cycle is to improve Netplan’s code quality and stability. We also want to continuously test our code to keep the quality while we work to add new features.

Below is a list of things we’ve been working on to achieve these goals.

Increasing the compiler warning level

The compiler warning level was increased from 1 to 2 in our meson.build file. With this change, our code is now compiled with the flags -Wall and -Wextra.

Because we treat warnings as errors, this change forced us to fix a few number of issues in our code, such comparisons between signed and unsigned variables, missing members in static struct initializations and other things. PR#380

Testing Netplan’s C code with cmocka

As you might know, the Netplan’s parser and configuration generation code are written in C.

Until recently, this code was tested only through our bindings for Python, which is good enough to test if it works as intended.

Some time ago, we introduced unit tests using the cmocka framework. One of the main wins with cmocka is that we can also check if the code has memory issues such as memory leaks and out of bound memory accesses by compiling it with GCC’s Address Sanitizer. PR#298

Using ASAN to detect memory issues automatically

As already mentioned, during tests, we compile and run our code with GCC’s Address Sanitizer. Apart from the unit tests, we also run the Netplan’s generator against each configuration example YAML in the examples/ directory. Even though we still don’t have 100% of code coverage in this particular test, it’s helping us to catch some issues automatically. PR#321

Static Analysis with Coverity

More recently we started a periodically code static analysis with Coverity, which helped us to identify and fix a number of issues (PR#383).

Coverity is a powerful tool that can detect a big number of issues in C code. It’s available for free for open source projects.

Configuration Fuzzing

We’ve been experimenting with fuzzing Netplan’s parser with random valid YAML configuration. To do that, we use a JSON schema with a fake data generator to create random but still valid Netplan YAML configuration. We then send it through Netplan’s parser and see what happens. For this test, Netplan is compiled with ASAN so any memory issues will cause the generator to crash so we can detect the problem.

While this project is not ready yet, it already helped us to find and fix a few issues such as memory leaks and crashes. We have plans to use this technique to continuously brute force the Netplan’s parser as part of our CI workflows.

2 Likes

Netplan v0.107 released, incl. Python bindings

I’m happy to announce that Netplan version 0.107 is now available on GitHub and is soon to be deployed into a Linux installation near you! Six months and more than 200 commits after the previous version (including a .1 stable release), this release is brought to you by 8 free software contributors from around the globe.

Highlights

Highlights of this release include the new configuration types for veth and dummy interfaces:

network:
  version: 2
  virtual-ethernets:
    veth0:
      peer: veth1
    veth1:
      peer: veth0
  dummy-devices: # loopback
    dm0:
      addresses:
        - 192.168.0.123/24
      ...

Furthermore, we implemented CFFI based Python bindings on top of libnetplan’s API, that are available as part of the python3-netplan package and can easily be consumed by 3rd party applications (see full cffi-bindings.py example):

from netplan import Parser, State, NetDefinition
from netplan import NetplanException, NetplanParserException
parser = Parser()

# Parse the full, existing YAML config hierarchy
parser.load_yaml_hierarchy(rootdir='/')

# Validate the final parser state
state = State()
try:
    # validation of current state + new settings
    state.import_parser_results(parser)
except NetplanParserException as e:
    print('Error in', e.filename, 'Row/Col', e.line, e.column, '->', e.message)
except NetplanException as e:
    print('Error:', e.message)

# Walk through ethernet NetdefIDs in the state and print their backend
# renderer, to demonstrate working with NetDefinitionIterator &
# NetDefinition
for netdef in state.ethernets.values():
    print('Netdef', netdef.id, 'is managed by:', netdef.backend)
    print('Is it configured to use DHCP?', netdef.dhcp4 or netdef.dhcp6)

Changelog:

Netplan at DebConf 2023 (Kochi/India)

We’ve recently presented Netplan and its usage within Debian at DebConf 2023 in Kochi/India:
A declarative approach to Linux networking with Netplan

Recording:
https://laotzu.ftp.acc.umu.se/pub/debian-meetings/2023/DebConf23/debconf23-53-a-declarative-approach-to-linux-networking-with-netplan.av1.webm

Download slides (PDF)

1 Like

NetworkManager integration on Ubuntu Desktop 23.10

With Ubuntu 23.10 out the door, the Netplan’s integration with NetworkManager is now enabled by default on Ubuntu Desktop. To show how this integration is working end-to-end from the graphical NetworkManager UI to the Netplan YAML settings persisted on disk, we created a mini-tutorial:

All network configuration is stored in /etc/netplan/ and made available to NetworkManager transparently. Furthermore, a passthrough method was implemented for handling unknown or new settings, making Netplan future-proof for any upcoming NetworkManager release.

1 Like

ip route add... supports dport and sport. How do I specify source or destination port using yaml routing?

Also, /bin/ip rule add to 192.5.5.241/32 lookup default, specifying default in yaml causes an error:

            routing-policy:
            -   to: 192.5.5.241
                table: default

Error in network definition: invalid unsigned int value 'default'
I have to specify table: 32767 in yaml.