While working with
kubernetes
somewhere down the line you might have faced this error and there could be many reasons behind the issue
error execution phase preflight
With my experience in working with
kubernetes
I found that this issue mostly occurs when you are trying to setup
kubernetes
cluster on your own and since you are setting it up by yourself than I can easily guess that you might miss a lot of technicalities and end up with this issue.
Tip -
First and far most look your log file very carefully because all the errors starting with
error execution phase preflight
is not the same. Look for the keyword in the error log such as
[ERROR DirAvailable]
[ERROR FileAvailable]
[ERROR Swap]
[ERROR IsPrivilegedUser]
[ERROR NumCPU]
[ERROR FileContent]
All the above-mentioned errors are different although the error message always starts with error execution phase preflight.
For troubleshooting please follow the individual issue here -
[ERROR DirAvailable-var-lib-etcd]: /var/lib/etcd is not empty
[ERROR FileAvailable-etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
[ERROR Swap]: running with swap on is not supported. Please disable swap
[ERROR IsPrivilegedUser]: user is not running as root
[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
[ERROR FileContent-proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist or [ERROR FileContent-proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
kubelet-start: configmaps "kubelet-config-1.15" is forbidden: User "system:bootstrap:g0toug" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
kubeadm token
CNI(Container Network interface) .i.e. - calico, flannel
Firewall settings
1. [ERROR DirAvailable-var-lib-etcd]: /var/lib/etcd is not empty
1error execution phase preflight: preflight] Some fatal errors occurred:
1 ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
In my experience this issue happens when you have deleted the node and trying to re-create it. If you have
setup your local kubernetes cluster on your local development machine
then it is quite often you will face this issue.
It mostly occurs when you either shutdown or restart and in some cases if you are trying to delete one the kubernetes node.
Solution
- You should delete the directory
/var/lib/etcd
. Use the following command to delete the directory
Generally speaking
kubeadm reset
should delete this diretory (/var/lib/etcd). But there are many possible cases where this directory or file is locked or do not have permission to delete, so in that cases
kubeadm reset
will not be successful.
This issue is not so frequent on cloud plateform such GCP(Google Cloud Platform), AWS Cloud(Amazon Web Service) or any other cloud service provider.
2. [ERROR FileAvailable-etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists {#file-already-exist
This error tells that you need to run the
kubeadm reset
command before initializing the kubernetes cluster
(
kubeadm init
). In other word you need to
properly recover a K8s cluster after reboot
But first let's see the error when I ran the command
$ kubeadm reset
1error execution phase preflight: preflight] Some fatal errors occurred:
2ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]:
3/etc/kubernetes/manifests/kube-apiserver.yaml already exists
4ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]:
5/etc/kubernetes/manifests/kube-controller-manager.yaml already exists
6ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]:
7/etc/kubernetes/manifests/kube-scheduler.yaml already exists
8ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]:
9/etc/kubernetes/manifests/etcd.yaml already exists
10ERROR FileContent--proc-sys-net-ipv4-ip_forward]:
11 /proc/sys/net/ipv4/ip_forward contents are not set to 1
How to fix and properly recover kubernetes cluster after reboot?
So all the above errors can be fixed with the following steps -
Stop kubelet.service - You must stop the Kubelet Service using the command $ sudo systemctl stop kubelet.
service
Kubeadm reset - Run the kubeadm reset before initializing the kubernetes cluster. The most common cause of
this issue is you have already run the kubeadm init command on your kubernetes cluster and now again you are trying to run the same command before doing the kubeadm reset.
4. [ERROR IsPrivilegedUser]: user is not running as root
As the error say "user is not running as root", which mean you are not running kubeadm init command as root user. To fix the issue you should run the kubeadm init as root.
Solution - Add sudo before the running the kubeadm init command.
1$ sudo kubeadm init --apiserver-advertise-address=100.0.0.1 --pod-network-cidr=10.244.0.0/16
5. [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
As the error says you are trying to run Kubernetes cluster with 1 CPU which is not sufficient enough to install
Kubernetes cluster.
Although [ERROR NumCPU] can be ignored by setting the --ignore-preflight-errors=NumCPU flag in the kube init
command but it is not recommended because if you have very limited computing power then running the kubernetes
cluster will always be troublesome.
How to increase CPU if you are using Vagrant for installing kubernetes cluster?
If you are using the Vagrant then increase the CPUs count from more than 1.
Here is sample vagrant file which you could refer (look for similar vagrant file which you have with you and update the CPU count in it)
Look for the configuration cpus inside the vagrant file
1Vagrant.configure("2") do |config|
2 config.vm.define "amaster" do |amaster|
3 amaster.vm.box_download_insecure = true
4 amaster.vm.box = "hashicorp/bionic64"
5 amaster.vm.network "private_network", ip: "100.0.0.1"
6 amaster.vm.hostname = "amaster"
7 amaster.vm.provider "virtualbox" do |v|
8 v.name = "amaster"
9 v.memory = 2048
10 v.cpus = 2
11 end
12 end
How to increase CPU if you are using only Virtual Box for installing kubernetes cluster?
In case you are using the Virtual Box then you can go to Virtual Box settings and increase the CPUs, so that it is more than 1.
Please refer to following screenshot -
Increase the CPU if you are using AWS?
It always recommended to use Amazon Elastic Kubernetes Service (Amazon EKS) because Amazon EKS service will
take care of CPU and its computing power so you do not have to worry about it. Here is a guide if you do not know
how to setup AWS EKS using eksctl
6. [ERROR FileContent-proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist or [ERROR FileContent-proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
In order to fix this error you need to edit /etc/sysctl.conf and then set the value for /proc/sys/net/bridge/bridge-nf-call-iptables
Here are following command which you can use -
1$ sudo vi /etc/sysctl.conf
After opening the file sysctl.conf in edit mode, add the following line if its not there.
1net.bridge.bridge-nf-call-iptables = 1
Then you need to execute
The above changed please do the kubeadm reset and then initialize your Kubernetes cluster with kubeadm init command
7. kubelet-start: configmaps "kubelet-config-1.15" is forbidden: User "system:bootstrap:g0toug" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
1error execution phase kubelet-start: configmaps "kubelet-config-1.15" is forbidden: User "system:bootstrap:g0toug" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
Well if you are facing this issue then I could easily say there is problem with the kublet version and kubelet-config-map.
The above error signifies that you are trying to join kubelet which is 1.15 and which has kublet-config-1.14, so that is not supported at this moment.
But as per the github it should be fixed in 1.16 for more details you can refer to git issue – #1582
8. kubeadm token
The first and far most guess would be to verify your kubeadm token and its validity. Because it quite obvious that you might have generated the token sometimes back and now it has expired.
So if you are still trying to connect to kubernetes master node with expired token than you will end up with the error.
I would recommend you to use following command to verify the kubeadm token.
Check the token validity
1$ kubeadm token list
The above command will return you with all the token which has been generated previously during the kubernetes cluster setup.
Carefully look on the EXPIRES column and see if there are any token which has expired.
How to fix the expired kubeadm token?
kubeadm(kubernetes admin) provides commands for generating/creating new token out of the box, so you can use following command to generate/create new token
1$ kubeadm token create --print-join-command
1kubeadm join 100.0.0.2:6443 --token xg7pkc.atu3g3nlanmajc8j --discovery-token-ca-cert-hash sha256:d19178104c84f7876602747a3b6e23c1df76f374fe9ba232ab406fd56d0b768e
kubeadm token create -print-join-command
The above command will create a new fresh token as well as print kubeadm join command, which you can copy and run from any node.
9. CNI(Container Network Interface) - calico, flannel
Well if you think that token validity of your cluster is okay and you do not have any expired token than I would recommend checking the CNI(container network interface) setting within your kubernetes cluster.
But in my experience troubleshooting CNI is needed when you setup kubernetes cluster with VMs(virtual machine) using Vagrant
But anyways this is what happens when you use the CNI either from calico of flannel. There will be two ethernet port on your kubernetes master node.
Use the following command to check the ethernet port
1eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
2 inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
3 inet6 fe80::a00:27ff:febb:1475 prefixlen 64 scopeid 0x20<link>
4 ether 08:00:27:bb:14:75 txqueuelen 1000 (Ethernet)
5 RX packets 704152 bytes 948370649 (948.3 MB)
6 RX errors 0 dropped 0 overruns 0 frame 0
7 TX packets 122139 bytes 7772979 (7.7 MB)
8 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
10eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
11 inet 100.0.0.2 netmask 255.255.255.0 broadcast 100.0.0.255
12 inet6 fe80::a00:27ff:fe36:b458 prefixlen 64 scopeid 0x20<link>
13 ether 08:00:27:36:b4:58 txqueuelen 1000 (Ethernet)
14 RX packets 100821 bytes 64254917 (64.2 MB)
15 RX errors 0 dropped 0 overruns 0 frame 0
16 TX packets 65066 bytes 23079670 (23.0 MB)
17 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
And if the command returns with 2 ethernet adapter name .i.e. eth0 and eth1. Then i can easily guess your CNI(container network interface) took the default ethernet adapter .i.e. eth0 and which is wrong.
Ideally it should take the eth1 as its preferred communication channel.
How to fix or setup correct ethernet adapter name for preferred CNI(Container Network Interface) ?
(Note - I tried following fix for the flannel but you can do similar steps for calico or any other CNI)
Look for the kube-flannel.yml inside your kubernetes node. You can search the kube-flannel.yml using following command
1find / -name 'kube-flannel.yml' 2>/dev/null
After that you need to edit the file and and search for the text - flanneld
In the args section add : – –iface=eth1
1- --iface=eth1
2 args:
3 - --ip-masq
4 - --kube-subnet-mgr
5 - --iface=eth1
Apply the flannel configuration
1$ kubectl apply -f kube-flannel.yml
1podsecuritypolicy.policy/psp.flannel.unprivileged created
2clusterrole.rbac.authorization.k8s.io/flannel created
3clusterrolebinding.rbac.authorization.k8s.io/flannel created
4serviceaccount/flannel created
5configmap/kube-flannel-cfg created
6daemonset.apps/kube-flannel-ds-amd64 created
7daemonset.apps/kube-flannel-ds-arm64 created
8daemonset.apps/kube-flannel-ds-arm created
9daemonset.apps/kube-flannel-ds-ppc64le created
10daemonset.apps/kube-flannel-ds-s390x created
10. Firewall settings
This could be one of the potential reason for this error because by default Firewall's are not disabled in any Linux distribution(Ubuntu, Centos, Fedora or RedHat).
The best way for you to check the firewall status before, use the following command for checking the firewall status.
Ubuntu
1$ sudo ufw status
Disable the firewall
But suppose you do not wanna disable the firewall due to security reason than it is highly recommended for you to check the firewall rule or in other words check iptables
Use the following command to verify the iptables
Note - The above command will tell you all the ports which are open for communication. Look for the port 443 because it the most default port for communication inside kubernetes cluster
Now the next question comes in my mind, how to setup a firewall rule or entry in iptables for port 443 inside kubernetes cluster?
Well use the following command for setting up the iptable rules.(Keep in mind the following firewall rules is for the port 443, but if you have any other port then please update the port accordingly)
1$ iptables -A INPUT -p tcp --dport 443 -j ACCEPT
CentOS
The theory is pretty much same for the CentOS also, but the commands are little different when we talk about CentOS.
Use the following command to check the status of the CentOS where your kubernetes cluster is running.
1$ sudo systemctl status firewalld
The above command will tell you about the firewall status.
If the firewall is up and running then i would recommend you to disable the firewall using the following command
1$ sudo systemctl enable firewalld
But just in case you not able to disable the firewall due to security reasons then you need to check the ports which are open using following command
1$ firewall-cmd --list-ports
If you didn't find any entry in the list then please add port in the firewall (change the port number if you have some other ports)
1$ firewall-cmd --permanent --add-port 443/tcp
I hope this firewall troubleshooting will help you to fix the issue "error execution phase preflight" inside your kubernetes cluster.
Learn more On Kubernetes -
Setup kubernetes on Ubuntu
Setup Kubernetes on CentOs
Setup HA Kubernetes Cluster with Kubespray
Setup HA Kubernetes with Minikube
Setup Kubernetes Dashboard for local kubernetes cluster
Setup Kubernetes Dashboard On GCP(Google Cloud Platform)
How to use Persistent Volume and Persistent Volume Claims in Kubernetes
Deploy Spring Boot Microservice on local Kubernetes cluster
Deploy Spring Boot Microservice on Cloud Platform(GCP)
Setting up Ingress controller NGINX along with HAproxy inside Kubernetes cluster
CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
kubectl export YAML | Get YAML for deployed kubernetes resources(service, deployment, PV, PVC....)
How to setup kubernetes jenkins pipeline on AWS?
Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
How to fix kubernetes pods getting recreated?
How to delete all kubernetes PODS?
How to use Kubernetes secrets?
Share kubernetes secrets between namespaces?
How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
Delete Kubernetes POD stuck in terminating state?
Kubernetes Cheat Sheet for day to day DevOps operations?
Delete Kubernetes POD stuck in terminating state?
How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
Share kubernetes secrets between namespaces?
How to use Kubernetes secrets?
How to delete all kubernetes PODS?
kubernetes pods getting recreated?
Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
kubectl export yaml OR How to generate YAML for deployed kubernetes resources
Kubernetes Updates
CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
Kubernetes cluster setup with Jenkins
How to use Persistent Volume and Persistent Claims | Kubernetes
How to fix ProvisioningFailed persistentvolume controller no volume plugin matched
Fixing – Cannot bind to requested volume: storageClasseName does not match
Fixing – pod has unbound immediate persistentvolumeclaims or cannot bind to requested volume incompatible accessmode
How to fix kubernetes dashboard forbidden 403 error – message services https kubernetes-dashboard is forbidden User
How to fix Kubernetes – error execution phase preflight [preflight]
Deploy Spring Boot microservices on kubernetes?
How to fix – ansible_memtotal_mb minimal_master_memory_mb
How to use kubespray – 12 Steps for Installing a Production Ready Kubernetes Cluster
How to setup kubernetes on CentOS 8 and CentOS 7
How to fix – How to fix - ERROR Swap running with swap on is not supported. Please disable swap
14 Steps to Install kubernetes on Ubuntu 20.04(bento/ubuntu-20.04), 18.04(hashicorp/bionic64)
Kubernetes Dashboard | Kubernetes Admin GUI | Kubernetes Desktop Client
Install Kubernetes with Minikube
Fixing the Terraform Error creating IAM Role. MalformedPolicyDocument Has prohibited field Resource
In terraform how to handle null value with default value?
Terraform use module output variables as inputs for another module?
How to Reference a Resource Created by a Terraform Module?
Its all about Open Source and DevOps, here I talk about Kubernetes, Docker, Java, Spring boot and practices.
Read More
