[Blog] Netplan developer diaries

Hi I’m the maintainer of Netplan.io, the tool for your network configuration needs.
Today I’d like to start this new topic which will contain stories and updates from the Netplan development team on a somewhat regular basis. You can expect a variety of topics to be covered, such as:

  • Announcements of (patch) releases
  • Introduction of new features
  • Interesting bugs & solutions
  • Conference presentations
  • Continued testing efforts
  • Spotlight on use cases
  • Mini tutorials

Feel free to reach out to us at any time!

Website: https://netplan.io
Documentation: netplan.rtfd.io
Questions: netplan tag on askubuntu.com
Chat: #netplan on libera.chat IRC
Code: github.com/canonical/netplan
Bugs: bugs.launchpad.net/netplan (Report)

4 Likes

In February 2023 we published our latest Netplan release v0.106, which contained an interesting new feature, called netplan status. This new sub-command queries your system for IP addresses, routes, DNS information, etc… in addition to the Netplan backend renderer (NetworkManager/networkd) in use and the relevant Netplan YAML configuration ID. It displays all this in a nicely formatted way (or alternatively in machine-readable YAML/JSON format).

This new feature was presented in more detail at the Ubuntu engineering sprint in Prague (May 2023). Please find the video below to watch this lightning talk:

2 Likes

Netplan 0.106.1 release

We are happy to announce that Netplan 0.106.1 is available for download on Ubuntu Mantic Minotaur and Debian testing.

This release includes some improvements in our documentation and CI infrastructure and a number of bug fixes.

What’s new in Netplan 0.106.1?

Documentation

Infrastructure

  • canonical/setup-lxd GitHub action. The autopkgtest environment creation was standardized to use Canonical’s setup-lxd action.
  • Snapd integrations tests with spread. A new test set for the Snapd integration with Netplan was introduced using the spread tool.
  • DBus. A number of DBus integration tests were added to the Debian package.

New features

  • Keyfile parser improvements. Our Network Manager keyfile parser (the capability of loading Network Manager configuration to Netplan YAML) was expanded to support all the types of tunnels supported by Netplan.

Misc

  • Ubuntu’s Code of Conduct 2.0 was added to the code repository.
  • We added a new bash autocompletion script with all the Netplan’s subcommands.
  • The new release package was synchronized with Debian.

Bug fixes

  • Keyfile parser. This release contains a couple of important fixes for the NetworkManager integration stability: 1) adding WPA enterprise connections is now working fine and new test cases were added to the package; 2) a WireGuard peer with allowed IPs that don’t include the network prefix are now accepted.
  • Netplan parser. A number of memory leaks and stability issues were fixed.
  • DBus. An issue related to how directory paths are built in the Netplan DBus service was causing issues in the Snapd integration and was fixed.

For the complete list of changes please consult the debian/changelog file in https://launchpad.net/ubuntu/+source/netplan.io/+changelog

1 Like

Testing Netplan’s Desktop integration with NetworkManager

Netplan.io is the “single source of truth” for controlling Ubuntu’s network stack, which means you can find all the information about your network configuration in /etc/netplan/ (or using sudo netplan get) on Ubuntu Server, Ubuntu Cloud images, Ubuntu Core IoT, etc. On Ubuntu Desktop, we deviated a bit from this approach in the past, as NetworkManager instead stored its “keyfile” settings in /etc/NetworkManager/system-connections/.

This changed in Ubuntu 23.10 “Mantic Minotaur” (the current development release), where NetworkManager is able to feed back information to Netplan’s YAML config (for example a WiFi password that was changed in the settings GUI), through a bidirectional integration where NetworkManager can talk to Netplan and vice versa.

I’d like to ask anybody running Ubuntu Mantic to help testing this new integration!
The Netplan integration will be automatically installed and activated on your system, when running NetworkManager 1.42.4-1ubuntu3 or above.

Note: When upgrading, NetworkManager will automatically create backups of your original keyfiles, stored in /var/lib/NetworkManager/backups/ and convert your connection profiles to Netplan YAML in /etc/netplan/.

Testing scenarios

Conversion on upgrade/installation

Whenever you add, modify or delete a connection in NetworkManager (e.g. via settings UI, D-Bus API, nmcli, …) it will be reflected in Netplan’s YAML configuration (i.e. sudo netplan get).

So after installation/upgrade, you should check the contents of /etc/netplan/90-NM-*.yaml and verify everything is properly converted into /run/NetworkManager/system-connections/ and properly visible in NetworkManager’s GUI, while /etc/NetworkManager/system-connections/ stays empty.

Ethernet connections

Check /etc/netplan for a .yaml file that already contains an ethernet connection and remove it. Then connect your system to a ethernet link via NetworkManager and check for the corresponding YAML to be created and make sure the contents look correct.

$ grep -ri ethernet /etc/netplan
/etc/netplan/90-NM-1b0dfe3a-2815-36aa-8e48-f883bbf16829.yaml:  ethernets:
/etc/netplan/90-NM-1b0dfe3a-2815-36aa-8e48-f883bbf16829.yaml:          ethernet._: ""
$ rm /etc/netplan/90-NM-1b0dfe3a-2815-36aa-8e48-f883bbf16829.yaml

## Plug into ethernet
$ cat /etc/netplan/90-NM-1b0dfe3a-2815-36aa-8e48-f883bbf16829.yaml
network:
  version: 2
  ethernets:
    NM-1b0dfe3a-2815-36aa-8e48-f883bbf16829:
      renderer: NetworkManager
      match:
        name: "enp0s31f6"
      dhcp4: true
      dhcp6: true
      ipv6-address-generation: "stable-privacy"
      wakeonlan: true
      networkmanager:
        uuid: "1b0dfe3a-2815-36aa-8e48-f883bbf16829"
        name: "Wired connection 1"
        passthrough:
          connection.autoconnect-priority: "-999"
          connection.timestamp: "1662490134"
          ethernet._: ""
          ipv6.ip6-privacy: "-1"
          proxy._: ""
          .nmmeta.nm-generated: "true"

WiFi connection

Using nmcli, create a new connection and look for an equivalent change under /etc/netplan/

$ nmcli con add con-name wifi1 type wifi ifname wifi1 ipv4.method manual ipv4.address 10.10.10.4/24 ipv4.gateway 10.10.10.1 ssid <ssid_name>

## should show your new wifi connection
$ nmcli con show
NAME         UUID                                  TYPE  DEVICE 
XXXXX        dffebd3b-f21e-4e0f-9ca0-59667d6ac3ac  wifi  wlp1s0 
XXXXX        ac0346ad-08f0-48d6-a071-11c7631d4374  wifi  --     
wifi1        f7adb62a-8088-4270-ac09-10755064cbb3  wifi  --

## Note there is a 90-NM-dffebd3.. for another wifi network that I've connected to
$ ls -l /etc/netplan 
total 12
-rw-r--r-- 1 root root 104 Apr 19 04:09 01-network-manager-all.yaml
-rw-r--r-- 1 root root 705 Sep 14 16:43 90-NM-dffebd3b-f21e-4e0f-9ca0-59667d6ac3ac.yaml
-rw-r--r-- 1 root root 676 Sep 12 15:28 90-NM-f7adb62a-8088-4270-ac09-10755064cbb3.yaml

See the new YAML file listed and open it up to look and make sure it looks correct

$ cat /etc/netplan/90-NM-f7adb62a-8088-4270-ac09-10755064cbb3.yaml
network:
  version: 2
  wifis:
    NM-f7adb62a-8088-4270-ac09-10755064cbb3:
      renderer: NetworkManager
      match:
        name: "wifi1"
      addresses:
      - "10.10.10.4/24"
      dhcp6: true
      ipv6-address-generation: "stable-privacy"
      access-points:
        "<ssid_name>":
          networkmanager:
            uuid: "f7adb62a-8088-4270-ac09-10755064cbb3"
            name: "wifi1"
            passthrough:
              ipv4.address1: "10.10.10.4/24,10.10.10.1"
              ipv4.method: "manual"
              ipv6.ip6-privacy: "-1"
              proxy._: ""
      networkmanager:
        uuid: "f7adb62a-8088-4270-ac09-10755064cbb3"
        name: "wifi1"

Insipration for further testing

In addition to standard WiFi and ethernet connections, we’d like to ask the community to test a wide variety of networking setups, as we cannot reproduce all the different setups that any of you might encounter in their daily lifes. Including, but not limited to:

  • Enterprise WiFi (e.g. eduroam)
  • WWAN (gsm/cdma modem connections)
  • WiFi AP mode (Fat, Fit, Cloud - and switching between them)
  • Different types of VPN connections (IPSec, WireGuard, VPNC, …)
  • Composite devices, like bridges or bonds
  • Virtual VLAN and Tunnel devices
  • DHCP address assignment (both IPv4 and IPv6)
  • Static IP assignment (both IPv4 and IPv6)
  • Shared network connectivity, e.g. a connection created with:
nmcli c add con-name shared type ethernet ifname enx000ec6e241bf ipv4.method shared ipv6.method ignore

Feedback

Please let us know about your experience using the Netplan desktop integration of NetworkManager. Be it positive or negative, your feedback is highly appreciated!

Please don’t hesitate to open a bug report and try to describe the setup you’ve been testing in detail and present any problems that might have occurred.

References

Please see Netplan’s documentation and this post for our previous, PPA based work:

2 Likes

Recent stability improvements

One of our goals for this cycle is to improve Netplan’s code quality and stability. We also want to continuously test our code to keep the quality while we work to add new features.

Below is a list of things we’ve been working on to achieve these goals.

Increasing the compiler warning level

The compiler warning level was increased from 1 to 2 in our meson.build file. With this change, our code is now compiled with the flags -Wall and -Wextra.

Because we treat warnings as errors, this change forced us to fix a few number of issues in our code, such comparisons between signed and unsigned variables, missing members in static struct initializations and other things. PR#380

Testing Netplan’s C code with cmocka

As you might know, the Netplan’s parser and configuration generation code are written in C.

Until recently, this code was tested only through our bindings for Python, which is good enough to test if it works as intended.

Some time ago, we introduced unit tests using the cmocka framework. One of the main wins with cmocka is that we can also check if the code has memory issues such as memory leaks and out of bound memory accesses by compiling it with GCC’s Address Sanitizer. PR#298

Using ASAN to detect memory issues automatically

As already mentioned, during tests, we compile and run our code with GCC’s Address Sanitizer. Apart from the unit tests, we also run the Netplan’s generator against each configuration example YAML in the examples/ directory. Even though we still don’t have 100% of code coverage in this particular test, it’s helping us to catch some issues automatically. PR#321

Static Analysis with Coverity

More recently we started a periodically code static analysis with Coverity, which helped us to identify and fix a number of issues (PR#383).

Coverity is a powerful tool that can detect a big number of issues in C code. It’s available for free for open source projects.

Configuration Fuzzing

We’ve been experimenting with fuzzing Netplan’s parser with random valid YAML configuration. To do that, we use a JSON schema with a fake data generator to create random but still valid Netplan YAML configuration. We then send it through Netplan’s parser and see what happens. For this test, Netplan is compiled with ASAN so any memory issues will cause the generator to crash so we can detect the problem.

While this project is not ready yet, it already helped us to find and fix a few issues such as memory leaks and crashes. We have plans to use this technique to continuously brute force the Netplan’s parser as part of our CI workflows.

2 Likes

Netplan v0.107 released, incl. Python bindings

I’m happy to announce that Netplan version 0.107 is now available on GitHub and is soon to be deployed into a Linux installation near you! Six months and more than 200 commits after the previous version (including a .1 stable release), this release is brought to you by 8 free software contributors from around the globe.

Highlights

Highlights of this release include the new configuration types for veth and dummy interfaces:

network:
  version: 2
  virtual-ethernets:
    veth0:
      peer: veth1
    veth1:
      peer: veth0
  dummy-devices: # loopback
    dm0:
      addresses:
        - 192.168.0.123/24
      ...

Furthermore, we implemented CFFI based Python bindings on top of libnetplan’s API, that are available as part of the python3-netplan package and can easily be consumed by 3rd party applications (see full cffi-bindings.py example):

from netplan import Parser, State, NetDefinition
from netplan import NetplanException, NetplanParserException
parser = Parser()

# Parse the full, existing YAML config hierarchy
parser.load_yaml_hierarchy(rootdir='/')

# Validate the final parser state
state = State()
try:
    # validation of current state + new settings
    state.import_parser_results(parser)
except NetplanParserException as e:
    print('Error in', e.filename, 'Row/Col', e.line, e.column, '->', e.message)
except NetplanException as e:
    print('Error:', e.message)

# Walk through ethernet NetdefIDs in the state and print their backend
# renderer, to demonstrate working with NetDefinitionIterator &
# NetDefinition
for netdef in state.ethernets.values():
    print('Netdef', netdef.id, 'is managed by:', netdef.backend)
    print('Is it configured to use DHCP?', netdef.dhcp4 or netdef.dhcp6)

Changelog:

Netplan at DebConf 2023 (Kochi/India)

We’ve recently presented Netplan and its usage within Debian at DebConf 2023 in Kochi/India:
A declarative approach to Linux networking with Netplan

Recording:
https://laotzu.ftp.acc.umu.se/pub/debian-meetings/2023/DebConf23/debconf23-53-a-declarative-approach-to-linux-networking-with-netplan.av1.webm

Download slides (PDF)

1 Like

NetworkManager integration on Ubuntu Desktop 23.10

With Ubuntu 23.10 out the door, the Netplan’s integration with NetworkManager is now enabled by default on Ubuntu Desktop. To show how this integration is working end-to-end from the graphical NetworkManager UI to the Netplan YAML settings persisted on disk, we created a mini-tutorial:

All network configuration is stored in /etc/netplan/ and made available to NetworkManager transparently. Furthermore, a passthrough method was implemented for handling unknown or new settings, making Netplan future-proof for any upcoming NetworkManager release.

1 Like

ip route add... supports dport and sport. How do I specify source or destination port using yaml routing?

Also, /bin/ip rule add to 192.5.5.241/32 lookup default, specifying default in yaml causes an error:

            routing-policy:
            -   to: 192.5.5.241
                table: default

Error in network definition: invalid unsigned int value 'default'
I have to specify table: 32767 in yaml.

Short answer: You can’t as of now (v0.107), but you’d need to put a systemd-networkd override.conf file specifying those bits directly in the underlying renderer.

Yes, this option currently (as of v0.107) accepts only unsigned integers, and you need to pass the actual table ID.

Hi all,

NetworkManager connections with an explicit DoT (DNS over TLS) configuration are not supported with Netplan, but NetworkManager does feed back the DoT DNS info with server address and Server Name Indication (SNI) in the form server_address#SNI, e.g. 1.2.3.4#dns.myhome.com as nameserver addresses to Netplan. As a result, subsequent Netplan config applications fail because DNS servers don’t have the expected dotted decimal (IPv4) or colon’ed hex (IPv6) form.

nmcli> describe ipv4.dns

=== [dns] ===
[NM property description]
Array of IP addresses of DNS servers. For DoT (DNS over TLS), the SNI server name can be specified by appending "#example.com" to the IP address of the DNS server. This currently only has effect when using systemd-resolved.
1 Like

Feb 29, 2024 – A special day for a special release…

Let’s celebrate the v1.0 release of Netplan! :rocket:
Thanks to everybody who contributed to the project over the years, to reach this milestone!

Highlights

  • Simultaneous WPA2 & WPA3 support.
  • Introduction of stable libnetplan1 API.
  • Mellanox VF-LAG support for high performance SR-IOV networking.
  • New hairpin and port-mac-learning settings, useful for VXLAN tunnels with FRR.
  • New netplan status --diff subcommand, finding differences between configuration and system state:

Releasing v1.0

Documentation

Bug fixes

New Contributors

Full Changelog: https://github.com/canonical/netplan/compare/0.107...1.0

Show all
$ git shortlog 0.107..HEAD
Chris Aumann (2):
      networkd: fix formatting
      networkd: replace deprecated CriticalConnection= by KeepConfiguration=

Christopher (1):
      Add additional bridge port settings (#410)

Danilo Egea Gondolfo (64):
      netplan: add support for WPA3-Enterprise
      CI/Coverity: fetch tags before trying to use them
      CI/Autopkgtests: temporally use a patched version of network-manager
      wifi: support WPA2 and WPA3 Personal simultaneously
      util: don't return a placeholder netdef in the iterator
      tunnels/validation: do not error out if "local" is not defined
      tests: add some integration tests without the local address
      wireguard: ignore empty endpoints
      auth: add support for LEAP and EAP-PWD
      parse: improve the parsing of access-points (LP: #1809994)
      wifi: replace the previously defined AP with the new one
      doc: spelling check improvements
      CI: add spelling checker step
      wifi: make it possible to have a psk and an eap password simultaneously
      nm-parse: always read the PSK into the new psk variable
      netdef: simplify NetplanRoute __eq__ and __hash__
      state_diff: add the new state_diff submodule
      state_diff: add the get_diff() method
      state_diff: add support for IP addresses analysis
      state_diff: add support for nameservers analysis
      state_diff: add support for search domains analysis
      state_diff: add support for MAC addresses analysis
      state_diff: add support for routes analysis
      state_diff: add a JSON encoder for NetplanRoutes
      state_diff: add more information to the missing_interfaces result
      state_diff/get_diff(): return the interfaces sorted by their indices
      tools/diff.py: add a temporary tool to test netplan diff
      apply: bring "lo" back up if it's managed by NM
      apply: don't assume the NM loopback connection is called "lo"
      ovs: disable StartLimitBurst in the ovs-cleanup service
      ctests: stop including C files in the test files
      docs: add a topic about security
      workflow/coverity: install missing dependencies
      state_diff: fix filtering of host scoped routes
      cli/sriov: remove unused code
      tests: assert generated .service files in assert_sriov
      tests/sriov: test if the generated netplan-rebind service is correct
      sriov: don't generate duplicate entries in the rebind.service file
      state/status: add support for identifying bridge/bond members
      state/status: add support for VRF members and interface kind
      state/status: Rename Members to Interfaces
      libnetplan: add a get_vrf_link() function
      state_diff: add support for bond/bridge/vrf
      state_diff: sort the list of IPs in the diff
      state_diff: sort routes by destination address
      state_diff: do not process interfaces removed from netplan
      state_diff: filter out loopback host scoped route
      cli/utils: add a function to create a lookup table from iproute2/rt_tables
      cli/status: add netplan status --diff
      parser: accept special options for MAC address
      libnetplan: add a getter for bond mode
      sriov: move the udev logic to a service unit
      sriov: check the eswitch mode before trying to change it
      sriov_rebind: cooperate with VF LAG activation
      sriov_rebind: netplan rebind --debug setup
      tests/sriov: adapt tests to the last sr-iov related changes
      sriov_apply: execute apply --sriov-only before network-pre.target
      utils/state_diff: refactor route table lookup methods
      state_diff: adopt MAC address options
      libnetplan: expose the link-local setting
      state_diff: handle link local IPs and routes
      tools/diff.py: drop old diff script
      tests: fix the status --diff integration tests
      docs: add status --diff to the docs

Danilo Egêa Gondolfo (1):
      parse-nm/wg: append the correct prefix to IPv6 addresses (#428), LP: #2046158

Lukas Märdian (49):
      test:ovs: Avoid NetworkManager taking contol, breaking a test
      parse: allow COMMON_LINK_HANDLERS for VRFs (LP: #2031421, Closes: #1049432)
      docs: Add "Contribute Documentation" how-to
      tests: Add autopkgtest for LP#1959570
      cli:try: avoid linting error for type hints
      doc: Set-up some basic Doxygen project
      doc: Make Sphinx to handle autodoxygen project, using breathe
      doc: create libnetplan apidoc structure
      inc: Start documenting public API
      doc: Update 'Netplan everywhere' for 23.10
      networkd: move KeepConfiguration= into [Network] section
      cli:try: use state of the art type hint
      tests: skip some cases if OVS is not installed
      meson: Drop legacy generator symlink
      CI: cleanup
      CI: fix DebCI, with multiple versions in 'unstable'
      CI: adopt for usrmerge paths
      tools:run_asan.sh use local Netplan python code
      Makefile: update for tools/run_asan.sh artifacts
      tests:regressions: Drop dbus tests, which got integrated in tests/integration/dbus.py
      test:generator:base: Refactor to use new API
      test:libnetplan: Refactor to avoid deprecated API, add new ctests
      tests:parser:keyfile: Port to new API/bindings
      inc: Drop deprecated legacy API
      Drop legacy ABI compat functions
      src: Drop ABI compat exports
      abi: drop global state & parser
      Clean up 'TODO' file
      ABI: regenerate compat check for dropped symbols
      CI: adopt for 0.107.1-2 on Jammy runners
      meson: bump version and SOVER to 1.0
      CI: upgrade network-manager test to Ubuntu Noble
      src: drop legacy 'global_' data structures
      util-internal: Do not export private symbols
      src: Clearly mark internal API, using a '_netplan' prefix
      abi-compat: move to 1.0
      CI: introduce a Netplan-CI PPA, to ship NetworkManager and systemd quirks
      doc: Document public API symbols
      API: drop netplan_generate
      API: restructure headers, sorting symbols by object they relate to
      abi-compat: Update for dropped 'netplan_generate' symbol
      cli:utils: introduce systemctl_is_installed helper
      cli:ovs: Improve OvsDbServerNotInstalled debug message
      cli:state: Recognize additional tunnel types
      cli:state_diff: Display unknown interface type as 'other'
      CI: fix NetworkManager autopkgtest not using deb822
      test:cli:units: skip _permission_denied test if run as root
      tests:integration: Be less strict about systemctl daemon-reload (LP: #2048388)
      abi-compat: 1.0 ABI

Mathieu Dugal (1):
      Fix permissions on folder '/run/NetworkManager/'

Robert Krátký (5):
      Exclude changes in 'doc/' dir from code CI checks.
      Adopt Docs Starter Pack:
      Fix Doxygen config. & remove Doxyfile.
      Improve how-to docs:
      Lang. & formatting fixes in API docs.

Sahil Sagwekar (1):
      examples: update libvirt path (#442)

Sudo-Whodo (1):
      added mii-monitor-interval (#411)

Tom (1):
      Update ovs.py to check if ovsdb-server.service is installed before throwing an error
2 Likes