Unable to start microk8s on ubuntu-core 22

#1

Hi All,

I am new to ubuntu-core and I tried installing ubuntu-core on a KVM following the instructions shared by canonical.

Now I am trying to install microk8s on ubuntu-core as mentioned in below article.

Even though I am able to install microk8s snap successfully as shown in below logs, I dont see microk8s up and running

cf@free-buck:~$ snap list
Name         Version         Rev    Tracking            Publisher   Notes
core         16-2.60.3       16091  latest/stable       canonical✓  core
core20       20230801        2015   latest/stable       canonical✓  base
core22       20230801        864    latest/stable       canonical✓  base
microk8s     v1.27.5         5892   1.27-strict/stable  canonical✓  -
nano-strict  5.2+pkg-353f    32     latest/stable       brlin       -
pc           22-0.3          146    22/stable           canonical✓  gadget
pc-kernel    5.15.0-86.96.1  1433   22/stable           canonical✓  kernel
snapd        2.60.3          20092  latest/stable       canonical✓  snapd

cf@free-buck:~$ sudo microk8s status 
microk8s is not running. Use microk8s inspect for a deeper inspection.
cf@free-buck:~$ sudo microk8s start
cf@free-buck:~$ 
cf@free-buck:~$ sudo microk8s status 
microk8s is not running. Use microk8s inspect for a deeper inspection.
1 Like
First steps with Ubuntu Core
#2

Well I tried the same in a physical server running ubuntu-core and I am facing same issue.

#3

Hi @codingfreak,

Unfortunately I can’t reproduce the issue - the microk8s snap starts fine on my Core 22 VM. I’m also using the same snap versions as you:

itrue@ubuntu:~$ snap list
Name          Version           Rev    Tracking            Publisher   Notes
core20        20230801          2015   latest/stable       canonical✓  base
core22        20230801          864    latest/stable       canonical✓  base
htop          3.2.2             3873   latest/stable       maxiberta   -
microk8s      v1.27.5           5892   1.27-strict/stable  canonical✓  -
pc            22-0.3            146    22/stable           canonical✓  gadget
pc-kernel     5.15.0-86.96.1    1433   22/stable           canonical✓  kernel
snapd         2.59.5            19457  latest/stable       canonical✓  snapd
snappy-debug  0.36-snapd2.59.4  704    latest/stable       canonical✓  -

Microk8s starts without issue after a few minutes:

itrue@ubuntu:~$ sudo microk8s status
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    dns                  # (core) CoreDNS
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
  disabled:
    cert-manager         # (core) Cloud native certificate management
    community            # (core) The community addons repository
    dashboard            # (core) The Kubernetes dashboard
    host-access          # (core) Allow Pods connecting to Host services smoothly
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    ingress              # (core) Ingress controller for external access
    mayastor             # (core) OpenEBS MayaStor
    metallb              # (core) Loadbalancer for your Kubernetes cluster
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    minio                # (core) MinIO object storage
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    rbac                 # (core) Role-Based Access Control for authorisation
    registry             # (core) Private image registry exposed on localhost:32000
    storage              # (core) Alias to hostpath-storage add-on, deprecated

Can you please share some more info on what you’re running it on? Is it amd64 hardware? Microk8s (k8s in general) is also very resource hungry and it can be tricky to get it running with <= 4GB of RAM.

Can you also please install the htop snap and check if there are any microk8s/kubelite processes running and consuming resources in the background, so we know it’s not totally stopped running in the snap? It should be easier to check this using htop than using standard top.

Does sudo microk8s.kubectl get pods -A show anything?

Cheers

1 Like
#4

Thanks for the reply @itrue.

I am running on a server which is brought up by the MAAS

image
image1141×628 31.8 KB

I have used following commands to setup microk8s in my server

   24  sudo snap install microk8s --channel=1.25-strict/stable
   25  sudo microk8s start
   26  sudo microk8s status --wait-ready

With respect to kubelite in htop (sorry not an expert in htop)

3846 lxd         26   6 1748M  969M  921M S   0.0  0.4  0:24.71 /snap/lxd/25945/bin/qemu-system-x86_64 -S -name mm01 -uuid 9b63c89c-1933-4cc2-ab0a-4c5783aed8ca -daemonize -cpu ho
   3847 lxd         24   4 1748M  969M  921M S   0.7  0.4  5:04.74 /snap/lxd/25945/bin/qemu-system-x86_64 -S -name mm01 -uuid 9b63c89c-1933-4cc2-ab0a-4c5783aed8ca -daemonize -cpu ho
   3943 lxd         22   2 1748M  969M  921M S   0.0  0.4  2:35.53 /snap/lxd/25945/bin/qemu-system-x86_64 -S -name mm01 -uuid 9b63c89c-1933-4cc2-ab0a-4c5783aed8ca -daemonize -cpu ho
  62645 lxd         21   1 1938M  798M  749M S   4.6  0.3 41:41.03 /snap/lxd/25945/bin/qemu-system-x86_64 -S -name mm03 -uuid 92884103-28e4-4eda-be5c-5d249657e7a1 -daemonize -cpu ho
  62646 lxd         20   0 1938M  798M  749M S   0.0  0.3  0:00.39 /snap/lxd/25945/bin/qemu-system-x86_64 -S -name mm03 -uuid 92884103-28e4-4eda-be5c-5d249657e7a1 -daemonize -cpu ho
  62651 lxd         20   0 1938M  798M  749M S   0.0  0.3  0:23.54 /snap/lxd/25945/bin/qemu-system-x86_64 -S -name mm03 -uuid 92884103-28e4-4eda-be5c-5d249657e7a1 -daemonize -cpu ho
  62656 lxd         20   0 1938M  798M  749M S   0.0  0.3  4:45.18 /snap/lxd/25945/bin/qemu-system-x86_64 -S -name mm03 -uuid 92884103-28e4-4eda-be5c-5d249657e7a1 -daemonize -cpu ho
  62735 lxd         20   0 1938M  798M  749M S   0.7  0.3  2:27.57 /snap/lxd/25945/bin/qemu-system-x86_64 -S -name mm03 -uuid 92884103-28e4-4eda-be5c-5d249657e7a1 -daemonize -cpu ho
  14107 root        37  17 1116M  369M 96304 S   5.9  0.1  1h04:27 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14241 root        20   0 1116M  369M 96304 S   1.3  0.1  4:42.75 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14242 root        20   0 1116M  369M 96304 S   0.0  0.1  2:25.62 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14243 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.12 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14244 root        20   0 1116M  369M 96304 S   0.0  0.1  2:08.08 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14245 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.00 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14246 root        20   0 1116M  369M 96304 S   0.0  0.1  0:51.68 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14247 root        20   0 1116M  369M 96304 S   0.0  0.1  2:19.02 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14248 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.09 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14249 root        20   0 1116M  369M 96304 S   0.0  0.1  1:02.12 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14250 root        20   0 1116M  369M 96304 S   0.0  0.1  1:00.72 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14251 root        20   0 1116M  369M 96304 S   0.7  0.1  0:55.72 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14252 root        20   0 1116M  369M 96304 S   0.0  0.1  2:09.73 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14253 root        20   0 1116M  369M 96304 S   0.0  0.1  2:38.24 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14254 root        20   0 1116M  369M 96304 S   0.0  0.1  0:04.93 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14255 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.12 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14256 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.57 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14257 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.47 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14258 root        20   0 1116M  369M 96304 S   0.0  0.1  2:19.61 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14259 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.89 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14260 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.29 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14261 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.13 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14262 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.00 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14264 root        20   0 1116M  369M 96304 S   0.0  0.1  1:51.30 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14265 root        20   0 1116M  369M 96304 S   0.0  0.1  1:34.62 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14266 root        20   0 1116M  369M 96304 S   0.0  0.1  0:51.31 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14297 root        20   0 1116M  369M 96304 S   0.0  0.1  0:55.04 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14298 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.11 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14299 root        20   0 1116M  369M 96304 S   0.0  0.1  0:59.46 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14300 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.28 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14301 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.07 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14302 root        20   0 1116M  369M 96304 S   2.0  0.1  2:39.64 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14303 root        20   0 1116M  369M 96304 S   0.0  0.1  1:01.83 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14304 root        20   0 1116M  369M 96304 S   0.0  0.1  2:16.96 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage
  14305 root        20   0 1116M  369M 96304 S   0.0  0.1  0:00.19 /snap/microk8s/5976/kubelite --scheduler-args-file=/var/snap/microk8s/5976/args/kube-scheduler --controller-manage

Well some of the following logs can be seen consistently in journalctl even today where CNI is not getting setup properly. I am suspecting this is the cause behind the failure in my case

Oct 11 06:45:38 mltr01 microk8s.daemon-kubelite[14107]: E1011 13:45:38.789942   14107 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 11 06:45:38 mltr01 systemd[1]: snapd.snap-repair.service: Deactivated successfully.
Oct 11 06:45:38 mltr01 systemd[1]: Finished Automatically fetch and run repair assertions.
Oct 11 06:45:41 mltr01 microk8s.daemon-apiserver-kicker[578297]: Setting up the CNI
Oct 11 06:45:43 mltr01 microk8s.daemon-kubelite[14107]: E1011 13:45:43.790836   14107 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 11 06:45:46 mltr01 microk8s.daemon-apiserver-kicker[578343]: Setting up the CNI
Oct 11 06:45:48 mltr01 microk8s.daemon-kubelite[14107]: E1011 13:45:48.792230   14107 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 11 06:45:52 mltr01 microk8s.daemon-apiserver-kicker[578393]: Setting up the CNI
Oct 11 06:45:53 mltr01 microk8s.daemon-kubelite[14107]: E1011 13:45:53.793760   14107 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 11 06:45:57 mltr01 microk8s.daemon-apiserver-kicker[578441]: Setting up the CNI
Oct 11 06:45:58 mltr01 microk8s.daemon-kubelite[14107]: E1011 13:45:58.794639   14107 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 11 06:46:03 mltr01 microk8s.daemon-apiserver-kicker[578489]: Setting up the CNI
Oct 11 06:46:03 mltr01 microk8s.daemon-kubelite[14107]: E1011 13:46:03.796303   14107 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 11 06:46:08 mltr01 systemd[1]: systemd-timedated.service: Deactivated successfully.
Oct 11 06:46:08 mltr01 microk8s.daemon-apiserver-kicker[578538]: Setting up the CNI
Oct 11 06:46:08 mltr01 microk8s.daemon-kubelite[14107]: E1011 13:46:08.798182   14107 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 11 06:46:13 mltr01 microk8s.daemon-kubelite[14107]: E1011 13:46:13.800068   14107 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 11 06:46:13 mltr01 microk8s.daemon-apiserver-kicker[578586]: Setting up the CNI
Oct 11 06:46:18 mltr01 microk8s.daemon-kubelite[14107]: E1011 13:46:18.801026   14107 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 11 06:46:19 mltr01 microk8s.daemon-apiserver-kicker[578637]: Setting up the CNI
Oct 11 06:46:23 mltr01 microk8s.daemon-kubelite[14107]: E1011 13:46:23.802650   14107 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 11 06:46:24 mltr01 microk8s.daemon-apiserver-kicker[578687]: Setting up the CNI
Oct 11 06:46:28 mltr01 microk8s.daemon-kubelite[14107]: E1011 13:46:28.804327   14107 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 11 06:46:30 mltr01 microk8s.daemon-apiserver-kicker[578735]: Setting up the CNI
~
#5

Low-hanging fruit: is your server behind some sort of a proxy? This has been reported a couple of times [1] [2] against the microk8s snap.

You may want to try doing [3].

Might be nice to see the outputs of microk8s inspect and the whole log or the journal.

1 Like
#6

What @dilyn-corner mentioned could be a good thing to look in to.

Can you please run sudo microk8s.kubectl get pods -A and see if the containers are being initialised? If the calico containers are getting stuck, this could be the cause of the issue you’re seeing.

You might also be able to find out more by running sudo microk8s.kubectl logs -n kube-system -f ds/calico-node. This should show you the log output of the calico containers.

#7

@itrue - Please find the requested output below

#  sudo microk8s.kubectl get pods -A
No resources found
#  sudo microk8s.kubectl logs -n kube-system -f ds/calico-node
Error from server (NotFound): daemonsets.apps "calico-node" not found
#8

@dilyn-corner - Thanks for the reply

I am attaching the microk8s inspect output to this reply

# microk8s inspect
Inspecting system
Inspecting Certificates
Inspecting services
  Service microk8s.daemon-cluster-agent is running
  Service microk8s.daemon-containerd is running
  Service microk8s.daemon-kubelite is running
  Service microk8s.daemon-k8s-dqlite is running
  Service microk8s.daemon-apiserver-kicker is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster
Inspecting dqlite
  Inspect dqlite

WARNING:  The memory cgroup is not enabled. 
The cluster may not be functioning properly. Please ensure cgroups are enabled 
See for example: https://microk8s.io/docs/install-alternatives#heading--arm 
Building the report tarball
  Report tarball is at /var/snap/microk8s/5976/inspection-report-20231011_101955.tar.gz

https://drive.google.com/file/d/1Ws7eIBs41A06kHScmLNegU0mpJW5YgeG/view?usp=sharing

Well with respect to network connection, currently I am using a VM where MAAS is installed and it has 2 ports, port1 connected towards servers which are brought up by MAAS, port2 is connected towards the Internet. Currently I am performing NAT in the VM so that servers can connect to the Internet. I am able to download other snaps, vm images for lxd and upgrade the underlying HOST OS in the servers without any issue. Only microk8s snap running in these servers is having the issue.

#9

This is very strange. Can you please create a bug report on the Microk8s GitHub? https://github.com/canonical/microk8s/issues/new?assignees=&labels=&projects=&template=bug_report.md

They should be able to help you out more there.

#10

@itrue

Well I have created a defect in microk8s github.