Deploying Charmed Kubernetes with Kube-OVN as primary CNI

Background

About Kube-OVN:

Kube-OVN is an enterprise-level cloud-native network orchestration system under CNCF that combines the capabilities of SDN with cloud-native technologies, providing the most functions, extreme performance and the easiest operation.

See the official docs for more information.

While Kube-OVN has been available as an official addon for MicroK8s for some time, compatibility with Charmed Kubernetes was lacking. However, with the newly released Kube-OVN 1.16.15 it is now possible to install Kube-OVN as the primary CNI for the k8s snap with no direct intervention in the node filesystem.

The previous incompatibility was due to how k8s bootstrap installs the default CNI plugins into /opt/cni/bin. Rather than loose binaries, the bootstrap process installs a single binary cni with multicall symlinks to it. This works fine in normal use, but the kube-ovn-cni component would attempt to install a few of the default CNI plugins over those symlinks. This would lead to the kubelet attempting to call the loopback plugin and the macvlan plugin would answer. This was breaking, and was usually solved by manually deleting the contents of /opt/cni/bin after the bootstrap process. Today, Kube-OVN has a configuration flag to prevent the automatic install of common CNI plugins and thus does not collide with the default behavior of Charmed Kubernetes.

Installation Guide

First, you will need to disable the default networking features of Charmed Kubernetes during the bootstrap process. This is easily done by providing a bootstrap.yaml file:

cluster-config:
  network:
    enabled: false
  gateway:
    enabled: false
  load-balancer:
    enabled: false
  ingress:
    enabled: false

You will then pass that as an argument to the bootstrap command:
k8s bootstrap --file bootstrap.yaml

At this point you should go ahead and join your additional control-plane nodes to the cluster, if you plan to run in HA mode.

If you are installing using the helm charts, which is recommended for production workloads, you’ll need to label your control-plane nodes for Kube-OVN.
k8s kubectl label node -l node-role.kubernetes.io/control-plane kube-ovn/role=master --overwrite

Finally, you will install from the helm chart. You’ll need helm installed and configured with a kubeconfig file of course.
helm install -n kube-system kube-ovn kube-ovn-v2 --repo https://kubeovn.github.io/kube-ovn --version v1.16.15 --set cni.installCNIPlugins=false

Alternatively, if you want to use the scripted installer you will set INSTALL_CNI_PLUGINS=false in your environment. Or, if you’re using the kube-ovn v1 chart, use --set cni_conf.INSTALL_CNI_PLUGINS=false

That’s all it takes! Now you have all the advanced VPC features of Kube-OVN running on Charmed Kubernetes.