ingress-nginx部署状态为CrashLoopBackOff 问题排查
| 阿里云国内75折 回扣 微信号：monov8 |
| 阿里云国际，腾讯云国际，低至75折。AWS 93折 免费开户实名账号 代冲值 优惠多多 微信号：monov8 飞机：@monov6 |
ingress-nginx部署状态为CrashLoopBackOff 问题排查，说起来这个问题挺坑的，kubernetes集群部署的时候，没有开启ipvs转发，所以会有这个问题
系统版本：CentOs8.1
k8s版本：v1.21.3
containerd版本：ctr containerd.io 1.4.3
ingress部署后状态为CrashLoopBackOff ,不停的重启
[root@ck8s1 ingress]# kubectl get pod -n ingress-nginx
NAME READY STATUS RESTARTS AGE
default-http-backend-7b6d9847f6-crgs8 1/1 Running 0 46m
nginx-ingress-controller-7bbb744996-rd9d6 0/1 CrashLoopBackOff 19 46m
describe查看信息如下：
[root@ck8s1 ingress]# kubectl describe pod nginx-ingress-controller-7bbb744996-rd9d6 -n ingress-nginx
Name: nginx-ingress-controller-7bbb744996-rd9d6
Namespace: ingress-nginx
Priority: 0
Node: ck8s2/192.168.43.152
Start Time: Sun, 15 Aug 2021 12:06:08 +0800
Labels: app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
pod-template-hash=7bbb744996
Annotations: cni.projectcalico.org/containerID: f59aa02e0654127673fc968533873d70d3b172c6f9cea0f7cab027efc662ee46
cni.projectcalico.org/podIP: 192.168.135.151/32
cni.projectcalico.org/podIPs: 192.168.135.151/32
prometheus.io/port: 10254
prometheus.io/scrape: true
Status: Running
IP: 192.168.135.151
IPs:
IP: 192.168.135.151
Controlled By: ReplicaSet/nginx-ingress-controller-7bbb744996
Containers:
nginx-ingress-controller:
Container ID: containerd://f0f3c6cb2aafb5bc7474fc44c7b680aede4a481ef0b0c8826a71af88ffa0e663
Image: registry.aliyuncs.com/kubernetes/nginx-ingress-controller:0.20.0
Image ID: registry.aliyuncs.com/kubernetes/nginx-ingress-controller@sha256:3f06079f7727b2fb7ad5c97d8152eb622ae504674395dfa71fda7ce315aaaf30
Ports: 80/TCP, 443/TCP
Host Ports: 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--default-backend-service=$(POD_NAMESPACE)/default-http-backend
--configmap=$(POD_NAMESPACE)/nginx-configuration
--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
--udp-services-configmap=$(POD_NAMESPACE)/udp-services
--publish-service=$(POD_NAMESPACE)/ingress-nginx
--annotations-prefix=nginx.ingress.kubernetes.io
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 143
Started: Sun, 15 Aug 2021 12:44:59 +0800
Finished: Sun, 15 Aug 2021 12:45:38 +0800
Ready: False
Restart Count: 17
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: nginx-ingress-controller-7bbb744996-rd9d6 (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mj82d (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-mj82d:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 43m default-scheduler Successfully assigned ingress-nginx/nginx-ingress-controller-7bbb744996-rd9d6 to ck8s2
Normal Created 42m (x2 over 43m) kubelet Created container nginx-ingress-controller
Normal Started 42m (x2 over 43m) kubelet Started container nginx-ingress-controller
Normal Pulled 41m (x3 over 43m) kubelet Container image "registry.aliyuncs.com/kubernetes/nginx-ingress-controller:0.20.0" already present on machine
Warning Unhealthy 41m (x10 over 43m) kubelet Readiness probe failed: Get "http://192.168.135.151:10254/healthz": dial tcp 192.168.135.151:10254: connect: connection refused
Normal Killing 41m (x2 over 42m) kubelet Container nginx-ingress-controller failed liveness probe, will be restarted
Warning Unhealthy 22m (x34 over 42m) kubelet Liveness probe failed: Get "http://192.168.135.151:10254/healthz": dial tcp 192.168.135.151:10254: connect: connection refused
Warning BackOff 2m56s (x147 over 40m) kubelet Back-off restarting failed container
[root@ck8s1 ingress]#
查看pod日志
[root@ck8s1 ingress]# kubectl logs nginx-ingress-controller-7bbb744996-rd9d6 -n ingress-nginx
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.20.0
Build: git-e8d8103
Repository: https://github.com/kubernetes/ingress-nginx.git
-------------------------------------------------------------------------------
nginx version: nginx/1.15.5
W0815 04:44:59.166129 8 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0815 04:44:59.166490 8 main.go:196] Creating API client for https://10.10.0.1:443
发现是443端口的问题，测试端口不通
[root@ck8s1 ingress]# telnet 10.0.0.1 443
Trying 10.0.0.1...
1，解决方法：
修改kube-proxy配置把流量转发该成ipvs模式
2，编辑配置：
kubectl edit configmap kube-proxy -n kube-system
将 mode: “” 为空 改成 mode: “ipvs”
将 masqueradeAll: null 改成 masqueradeAll: yes
3，删除所有kube-proxy的pod
kubectl get pod -n kube-system | grep kube-proxy | awk '{system(" kubectl delete pod "$1" -n kube-system")}'4，校验
kubectl logs kube-proxy-xxx -n kube-system 检查是否日志出现Using ipvs Proxier
5，重新部署ingress-nginx
[root@ck8s1 ingress]# kubectl get pod -n ingress-nginx
NAME READY STATUS RESTARTS AGE
default-http-backend-7b6d9847f6-5jt5b 1/1 Running 0 13m
nginx-ingress-controller-7bbb744996-2c7tp 1/1 Running 0 13m
启动正常，没有报错
| 阿里云国内75折 回扣 微信号：monov8 |
| 阿里云国际，腾讯云国际，低至75折。AWS 93折 免费开户实名账号 代冲值 优惠多多 微信号：monov8 飞机：@monov6 |
标签: nginx