欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识！
您现在的位置是： 首页

nginx ingress最后的倔强: admission webhook

程序员文章站 2022-05-29 22:33:51
...

背景

k8s中大多使用nginx-ingress-controller来实现ingress, 但是脆弱的nginx-controller通过ingress解析出nginx配置, 对于某些annotation会reload nignx配置失败, 然后controller就卡死了, 不断重启, 除非删除对应的ingress.

问题复现

创建有问题的ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "false"
    nginx.ingress.kubernetes.io/auth-tls-verify-client: optional
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Host $targethost;
      proxy_buffering     off;
      proxy_pass          http://$targetbackend;
      proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
      proxy_redirect      off;
      proxy_set_header    X-SSL-Client-Verify $ssl_client_verify;
      proxy_set_header    X-SSL-Client-DN $ssl_client_s_dn;
      proxy_set_header    X-Real-IP       $remote_addr;
      proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
  creationTimestamp: "2020-03-23T04:57:22Z"
  generation: 1
  name: example-ingress
  namespace: kube-system
  resourceVersion: "57681168"
  selfLink: /apis/extensions/v1beta1/namespaces/kube-system/ingresses/example-ingress
  uid: c7f66385-6cc2-11ea-b6a8-246e96d4b538
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: example-svc
          servicePort: 8008
        path: /
  tls:
  - hosts:
    - example.com
    secretName: example-tls
status:
  loadBalancer: {}

查看nginx-ingress-controller状态全部为CrashLoopBackOff

# kubectl get po -n kube-system -owide |grep ingress
nginx-ingress-controller-ftfbg                        1/2     CrashLoopBackOff   6          8m27s
nginx-ingress-controller-hp4pf                        1/2     CrashLoopBackOff   11         24m  
nginx-ingress-controller-qlb4l                        1/2     CrashLoopBackOff   11         24m

查看nginx-ingress-controller日志, 显示reload失败"proxy_pass" directive is duplicate in /tmp/nginx-cfg911768424:822

-------------------------------------------------------------------------------
W0403 10:26:14.716246       1 queue.go:130] requeuing kube-system/nginx-ingress-controller-4txfk, err 
-------------------------------------------------------------------------------
Error: exit status 1
2020/04/03 10:26:14 [notice] 137#137: ModSecurity-nginx v1.0.0
2020/04/03 10:26:14 [warn] 137#137: duplicate value "error" in /tmp/nginx-cfg911768424:815
nginx: [warn] duplicate value "error" in /tmp/nginx-cfg911768424:815
2020/04/03 10:26:14 [warn] 137#137: duplicate value "timeout" in /tmp/nginx-cfg911768424:815
nginx: [warn] duplicate value "timeout" in /tmp/nginx-cfg911768424:815
2020/04/03 10:26:14 [emerg] 137#137: "proxy_pass" directive is duplicate in /tmp/nginx-cfg911768424:822
nginx: [emerg] "proxy_pass" directive is duplicate in /tmp/nginx-cfg911768424:822
nginx: configuration file /tmp/nginx-cfg911768424 test failed

-------------------------------------------------------------------------------
W0403 10:26:16.998897       1 nginx_status.go:207] unexpected error obtaining nginx status info: unexpected error scraping nginx status page: unexpected error scraping nginx : Get http://0.0.0.0:18080/nginx_status: dial tcp 0.0.0.0:18080: connect: connection refused
I0403 10:26:17.526801       1 main.go:167] Received SIGTERM, shutting down
I0403 10:26:17.526827       1 nginx.go:364] Shutting down controller queues
I0403 10:26:17.526845       1 status.go:200] updating status of Ingress rules (remove)
I0403 10:26:17.537511       1 status.go:219] removing address from ingress status ([])
I0403 10:26:17.537593       1 nginx.go:372] Stopping NGINX process
2020/04/03 10:26:17 [notice] 141#141: signal process started
I0403 10:26:20.547669       1 nginx.go:385] NGINX process has stopped
I0403 10:26:20.547692       1 main.go:175] Handled quit, awaiting Pod deletion
I0403 10:26:30.547824       1 main.go:178] Exiting with 0

解决方案

创建一个有问题的ingress, 会影响所有新创建的ingress规则, 又一个集群级别的Bug诞生了.那么有没有办法, 提前检验ingress配置, 有问题就不去reload. 那验证步骤肯定要在请求到达nginx-controller之前来做, 是不是想到了k8s-admission-webhook, 可以在apiserver持久化对象前拦截请求, 去实现自定义的验证规则. 好在新版本的nginx-ingress-controller(v0.25.0+)已经实现了相关的功能, 只需开启对应配置就行.

ApiServer配置

Apiserver开启webhook相关配置, 必须包含MutatingAdmissionWebhook与ValidatingAdmissionWebhook

--admission-control=MutatingAdmissionWebhook,ValidatingAdmissionWebhook

创建webhook相关配置

启用ValidatingAdmissionWebhook必须使用https, 需要配置对应证书

  • 手动生成:
    openssl req -x509 -newkey rsa:2048 -keyout certificate.pem -out key.pem -days 365 -nodes -subj "/CN=ingress-validation-webhook.ingress-nginx.svc"
  • CertificateSigningRequest
    通过k8s CertificateSigningRequest来创建(controller-manager需要开启--cluster-signing-cert-file与--cluster-signing-key-file)
    可通过如下脚本创建, namespace与service替换成自己的
    SERVICE_NAME=ingress-nginx
NAMESPACE=ingress-nginx

TEMP_DIRECTORY=$(mktemp -d)
echo "creating certs in directory ${TEMP_DIRECTORY}"

cat <<EOF >> ${TEMP_DIRECTORY}/csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${SERVICE_NAME}
DNS.2 = ${SERVICE_NAME}.${NAMESPACE}
DNS.3 = ${SERVICE_NAME}.${NAMESPACE}.svc
EOF

openssl genrsa -out ${TEMP_DIRECTORY}/server-key.pem 2048
openssl req -new -key ${TEMP_DIRECTORY}/server-key.pem \
    -subj "/CN=${SERVICE_NAME}.${NAMESPACE}.svc" \
    -out ${TEMP_DIRECTORY}/server.csr \
    -config ${TEMP_DIRECTORY}/csr.conf

cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${SERVICE_NAME}.${NAMESPACE}.svc
spec:
  request: $(cat ${TEMP_DIRECTORY}/server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

kubectl certificate approve ${SERVICE_NAME}.${NAMESPACE}.svc

for x in $(seq 10); do
    SERVER_CERT=$(kubectl get csr ${SERVICE_NAME}.${NAMESPACE}.svc -o jsonpath='{.status.certificate}')
    if [[ ${SERVER_CERT} != '' ]]; then
        break
    fi
    sleep 1
done
if [[ ${SERVER_CERT} == '' ]]; then
    echo "ERROR: After approving csr ${SERVICE_NAME}.${NAMESPACE}.svc, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
    exit 1
fi
echo ${SERVER_CERT} | openssl base64 -d -A -out ${TEMP_DIRECTORY}/server-cert.pem

kubectl create secret generic ingress-nginx.svc \
    --from-file=key.pem=${TEMP_DIRECTORY}/server-key.pem \
    --from-file=cert.pem=${TEMP_DIRECTORY}/server-cert.pem \
    -n ${NAMESPACE}

配置ingress controller

ingress controller需要启用如下参数, 挂载需要的tls证书

flag description example usage
--validating-webhook admission webhook的地址 :8080
--validating-webhook-certificate webhook证书 /usr/local/certificates/validating-webhook.pem
--validating-webhook-key webhook私钥 /usr/local/certificates/validating-webhook-key.pem

验证

更新后, 创建有问题的ingress则会拦截, 符合预期

# kubectl apply -f ing.yaml
Error from server: error when creating "ing.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: 
-------------------------------------------------------------------------------
Error: exit status 1
2020/04/02 10:26:04 [emerg] 331#331: directive "proxy_pass" is not terminated by ";" in /tmp/nginx-cfg461116913:2165
nginx: [emerg] directive "proxy_pass" is not terminated by ";" in /tmp/nginx-cfg461116913:2165
nginx: configuration file /tmp/nginx-cfg461116913 test failed

引用

  • https://kubernetes.io/zh/docs/reference/access-authn-authz/extensible-admission-controllers/
  • https://kubernetes.github.io/ingress-nginx/deploy/validating-webhook/
  • https://qingwave.github.io/ingress-nginx-controller-admission-webhook/
相关标签： 云原生

上一篇： Pod memory usage in k8s

下一篇： 用Photoshop给图片加局部马赛克将logo盖上

推荐阅读