[Issue&Solution] When we upgrading kube v1.16.12 > v1.17.17

Thanaphat Nuangjumnong
2 min readJan 26, 2021

##Wrong repository when run kubeadm upgrade apply v1.17.17 but doesn’t have the kube latest version

Solution: kubectl edit cm kubeadm-config -n kube-system -o yaml

imageRepository: gcr.io/google-containers > imageRepository: k8s.gcr.io

## Error KubeletNotReady Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: the server could not find the requested resource
Solution: Edit /var/lib/kubelet/config.yaml on All master node

featureGates:
CSIMigration: false

ref. https://stackoverflow.com/questions/59291108/worker-start-to-fail-csinodeifo-error-updating-csinode-annotation

# kubelet error log “node xxx not found”
Solution: check /etc/hostname and systemctl restart kubelet

# kube apiserver / controller / schduler failed
Solution: remove or comment the flag — port=0 flag in /etc/kubernetes/manifests/kube-controler-manager.yaml and kube-scheduler.yaml)

ref. https://github.com/kubernetes/kubeadm/issues/2156
ref. https://github.com/kubernetes/kubeadm/issues/2207

# kubelet error log “kubelet failed to get imageFs info: non-existent label “crio-images””

Solution: edit /usr/lib/systemd/system/crio.service

# Request to be run before kubelet.service
Before=kubelet.service

**Need to be systemctl daemon-reload , systemctl restart crio , systemctl restart kubelet

ref . https://github.com/cri-o/cri-o/issues/4437

## Cannot use kubectl get componentstatuses

Solution:

Modify the following files on all master nodes:
$ sudo vi /etc/kubernetes/manifests/kube-scheduler.yaml

Clear the line (spec->containers->command) containing this phrase: — — port=0
$ sudo vi /etc/kubernetes/manifests/kube-controller-manager.yaml

Ref. https://stackoverflow.com/questions/64296491/how-to-resolve-scheduler-and-controller-manager-unhealthy-state-in-kubernetes

## We found the Weave-Pod FastDP drop too low on Weavenet the problem on that node

Solution: 1. Delete weave-net-xxx on the node xx (kubectl delete pod weave-net-ppx82 -n kube-system)

##if Solution: 1 but not resolve go to step in below.
Solution: 2. Clear weave.db and wait for re-sync new data.

cd /var/lib/weave/
mv weave-netdata.db weave-netdata.db_bk
kubectl delete pod weave-net-xx -n kube-system

Good Luck!!

--

--