【背景】

        今天测试K8s集群的node节点扩容，扩容的整个过程都很顺利，但是后来发现在新扩容的node节点（k8s-node04）上，一直有一个calico-node的pod实例报错、不断重启。

【现象】

        从下面的pod实例的运行状态查询结果来看，可以发现有一个pod实例（calico-node-xl9bc）在不断的重启。

[root@k8s-master01 ~]# kubectl get pods -A| grep calico
kube-system            calico-kube-controllers-78d6f96c7b-tv2g6               1/1     Running     0          75m
kube-system            calico-node-6dk7g                                      1/1     Running     0          75m
kube-system            calico-node-dlf26                                      1/1     Running     0          75m
kube-system            calico-node-s5phd                                      1/1     Running     0          75m
kube-system            calico-node-xl9bc                                      0/1     Running     30          3m28s

【排查】

查询pod的日志

[root@k8s-master01 ~]# kubectl logs calico-node-xl9bc -n kube-system -f

-----省略部分日志----------------
2021-09-04 12:32:45.011 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:46.025 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:47.038 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:48.050 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:49.061 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:50.072 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:51.079 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:52.093 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:53.104 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:54.114 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:55.127 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:56.138 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:57.148 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:58.162 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:32:59.176 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:33:00.186 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:33:01.199 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:33:02.211 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:33:03.225 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host
2021-09-04 12:33:04.238 [ERROR][69] felix/health.go 246: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 114.114.114.114:53: no such host

这个报错在网上搜寻了很久，基本也没发现有针对性的解决方案。

        后来根据报错信息，在网上的一篇文章中发现了一个一模一样的报错案例，案例的原因是因为/etc/hosts文件中缺少了两行ipv4和ipv6的回环地址，回头来查看我本地node节点的/etc/hosts文件，竟然真的没有这两行，不知道昨晚我自己装这台虚拟机的时候，到底做了什么诡异的操作。

### /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

在k8s-node04的/etc/hosts文件中增加了这两行配置以后，重启网络，发现pod实例出现CrashLoopBackOff。

[root@k8s-master01 ~]# kubectl get pods -A| grep calico
kube-system            calico-kube-controllers-78d6f96c7b-tv2g6               1/1     Running            0          80m
kube-system            calico-node-6dk7g                                      1/1     Running            0          80m
kube-system            calico-node-dlf26                                      1/1     Running            0          80m
kube-system            calico-node-s5phd                                      1/1     Running            0          80m
kube-system            calico-node-xl9bc                                      0/1     CrashLoopBackOff   7          8m24s

删除这个pod实例，发现重新创建的pod实例的运行状态终于恢复正常了。

[root@k8s-master01 ~]# kubectl delete pod calico-node-xl9bc -n kube-system
pod "calico-node-xl9bc" deleted
[root@k8s-master01 ~]# kubectl get pods -A| grep calico
kube-system            calico-kube-controllers-78d6f96c7b-tv2g6               1/1     Running     0          81m
kube-system            calico-node-6dk7g                                      1/1     Running     0          81m
kube-system            calico-node-dlf26                                      1/1     Running     0          81m
kube-system            calico-node-mz58r                                      0/1     Running     0          5s
kube-system            calico-node-s5phd                                      1/1     Running     0          81m
[root@k8s-master01 ~]# kubectl get pods -A| grep calico
kube-system            calico-kube-controllers-78d6f96c7b-tv2g6               1/1     Running     0          81m
kube-system            calico-node-6dk7g                                      1/1     Running     0          81m
kube-system            calico-node-dlf26                                      1/1     Running     0          81m
kube-system            calico-node-mz58r                                      0/1     Running     0          7s
kube-system            calico-node-s5phd                                      1/1     Running     0          81m
[root@k8s-master01 ~]# kubectl get pods -A| grep calico
kube-system            calico-kube-controllers-78d6f96c7b-tv2g6               1/1     Running     0          81m
kube-system            calico-node-6dk7g                                      1/1     Running     0          81m
kube-system            calico-node-dlf26                                      1/1     Running     0          81m
kube-system            calico-node-mz58r                                      0/1     Running     0          8s
kube-system            calico-node-s5phd                                      1/1     Running     0          81m
[root@k8s-master01 ~]# kubectl get pods -A| grep calico
kube-system            calico-kube-controllers-78d6f96c7b-tv2g6               1/1     Running     0          81m
kube-system            calico-node-6dk7g                                      1/1     Running     0          81m
kube-system            calico-node-dlf26                                      1/1     Running     0          81m
kube-system            calico-node-mz58r                                      1/1     Running     0          11s
kube-system            calico-node-s5phd                                      1/1     Running     0          81m

参考：

http://www.manongjc.com/detail/15-tufllpwnqavcxef.html

https://www.lmonkey.com/t/DExg407BK