Upgrade Kubernetes

Kubernetes Upgrade v1.23 to v1.24 – Common Errors and Solutions

Enable the CRI plugin, to make containerd communicate with the kubelet directly.

On all cluster nodes enable cri plugin:

root@master:# vi /etc/containerd/config.toml
#Commented out cri from disabled_plugins
#disabled_plugins: ["cri"]
disabled_plugins: [""]
root@master:# systemctl daemon-reload
root@master:# systemctl restart containerd
root@ubmaster:# systemctl status containerd
root@master:# systemctl restart kubelet
root@master:# systemctl status kubelet

Kubeadm Upgrade (Non-Service Effecting for Pods)

apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm=1.24.9-00 && \
apt-mark hold kubeadm

Kubelet and Kubectl Upgrade (Service Effecting for Pods)

apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.24.9-00 kubectl=1.24.9-00 && \
apt-mark hold kubelet kubectl

Error

The kubelet service will not come up right away. Below is the error you will encounter, because of which your node will be in NotReady state, even though all packages are upgraded. Below the solution for this error.

root@master:# sudo systemctl status kubelet
  kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since 
master kubelet[123456]: Error: failed to parse kubelet flag: unknown flag: --network-plugin

There are a few blogs mentioning removing only “–network-plugin=cni” from the kubeadm-flags.env file in the kubelet directory. But it’s not successful as you will get below error and kubelet will remain down.

root@master:# journalctl -xe | kubelet
W0820 14:15:31.786693  154117 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {  <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial unix: missing address". Reconnecting...
Error: failed to run Kubelet: unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix: missing address"

Starting Kubelet service after failure

To resolve both Error 2 and 3, we need to comment out the existing environment variable used by the kubelet service and add updated parameters.

root@master:# vi /var/lib/kubelet/kubeadm-flags.env