[ตอนที่ 2] How to install ingress-nginx Via the host network on kubernetes cluster

เนื่องจาก K8S Cluster จำเป็นต้องมี Ingress (ทางเข้า) เพื่อให้ User เรียกใช้งาน Application เฉพาะ HTTP(80) / HTTPS(443)

พวกเราจึงเรียก Ingress-nginx มาช่วย Controller traffic ที่เข้ามาให้ถูกส่งต่อไปยัง Ingress บน Namespace ที่ถูก set hostRules ไว้ อ่านต่อที่ https://kubernetes.io/docs/concepts/services-networking/ingress/

ในบทความนี้ ผมได้ Design ให้มี External Loadbalance == 10.95.108.248
โดยมี Member 2 server เป็น Woker node ที่จะ deploy ingress-nginx controller ไปเกิด (ไม่จำเป็นว่าทุก Worker node ต้องมี Ingress-nginx)

Download mandatory file

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml

Edit deployment add config affinity

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: nginx-ingress-controller
  namespace: ingress-nginx
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: it.net/ingress
                operator: In
                values:
                - nginx
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/part-of
                operator: In
                values:
                - ingress-nginx
            topologyKey: kubernetes.io/hostname
      containers:
      - args:
        - /nginx-ingress-controller
        - --configmap=$(POD_NAMESPACE)/nginx-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
        - --annotations-prefix=nginx.ingress.kubernetes.io
        - --publish-status-address=10.95.108.248
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - /wait-shutdown
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        name: nginx-ingress-controller
        ports:
        - containerPort: 80
          hostPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          hostPort: 443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        resources: {}
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
          runAsUser: 101
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirstWithHostNet
      hostNetwork: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: nginx-ingress-serviceaccount
      serviceAccountName: nginx-ingress-serviceaccount
      terminationGracePeriodSeconds: 300

Deploy Ingress-nginx

kubectl apply -f mandatory.yaml

Deploy Serivce Ingress-nginx

Ref. https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/baremetal.md

!!! note This approach does not leverage any Service object to expose the NGINX Ingress controller. If the ingress-nginx Service exists in the target cluster, it is recommended to delete it.

Add labels for support param “affinity” in deployment file

kubectl label nodes {nodename with ingress-nginx} it.net/ingress=nginx
kubectl label nodes {nodename with ingress-nginx} it.net/ingress=nginx

Verify installation

kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx

Detect installed version

POD_NAMESPACE=ingress-nginx
POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version

DNS resolution

Pods configured with hostNetwork: true do not use the internal DNS resolver (i.e. kube-dns or CoreDNS), unless their dnsPolicy spec field is set to ClusterFirstWithHostNet. Consider using this setting if NGINX is expected to resolve internal names for any reason.

##edit deployment
dnsPolicy: ClusterFirst   > dnsPolicy: ClusterFirstWithHostNet

Ingress status

!!! note Alternatively, it is possible to override the address written to Ingress objects using the — publish-status-address flag. See Command line arguments.

## edit deployment
      containers:
      - args:
        - /nginx-ingress-controller
        - --configmap=$(POD_NAMESPACE)/nginx-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
   ##     - --publish-service=$(POD_NAMESPACE)/ingress-nginx
        - --annotations-prefix=nginx.ingress.kubernetes.io
        - --publish-status-address=10.95.108.248    ## Add VIP

Example

!!Note that all ingress in the kube-cluster using External IP (VIP)

How to test ingress work

## how to test deploy ingress / svc / pod
## deploy app
kubectl run nginx --image=nginx --dry-run -o yaml > nginx.yaml
kubectl apply -f nginx.yaml
## deploy svc
kubectl expose deployment nginx --port=80 --target-port=80
## deploy ing
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
spec:
  rules:
  - host: nginx.defualt.k8s
    http:
      paths:
        - backend:
            serviceName: nginx
            servicePort: 80
          path: /
##map host test
10.95.108.248   nginx.defualt.k8s

[root@marine1 ~]# curl -v nginx.defualt.k8s
* About to connect() to nginx.defualt.k8s port 80 (#0)
*   Trying 10.95.108.248...
* Connected to nginx.defualt.k8s (10.95.108.248) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: nginx.defualt.k8s
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.17.8
< Date: Thu, 05 Mar 2020 04:48:52 GMT
< Content-Type: text/html
< Content-Length: 612
< Connection: keep-alive
< Vary: Accept-Encoding
< Last-Modified: Tue, 03 Mar 2020 14:32:47 GMT
< ETag: "5e5e6a8f-264"
< Accept-Ranges: bytes
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Connection #0 to host nginx.defualt.k8s left intact

Ref. https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/baremetal.md

https://github.com/kubernetes/ingress-nginx