NGINX Architecture for EKS

This is the known good design for deploying the Ingress NGINX Controller on Amazon EKS.

Overview

Ingress NGINX Overview

The Ingress NGINX Controller is a Kubernetes-native load balancer that manages external access to services in a cluster. On EKS, it works seamlessly with Amazon Elastic Load Balancers (ELBs) to provide robust HTTP(S) ingress capabilities.

Key Features

  • Supports HTTP and HTTPS: Handles secure and non-secure traffic.
  • Rule-Based Routing: Route traffic based on host, path, or headers.
  • Custom Annotations: Allows fine-grained control over NGINX configurations.
  • Integration with ACM: Automates TLS certificate provisioning via AWS Certificate Manager.

For more details, refer to the AWS Documentation.

Design of NGINX Ingress Controller on EKS

AWS Load Balancer Behavior

How AWS Deploys a Load Balancer

When a service is configured with type: LoadBalancer in an EKS cluster, AWS automatically creates an Elastic Load Balancer (ELB) outside the cluster. This load balancer forwards external traffic to the node ports associated with the service.

  • The Load Balancer maps to the cluster nodes and routes traffic to the relevant pods via node ports.
  • By default, an Application Load Balancer (ALB) is used for HTTP/HTTPS traffic, while a Network Load Balancer (NLB) is preferred for TCP traffic.

Example:

  • Service with ports 80 and 443.
  • ELB forwards traffic to random node ports, such as 31567 (HTTP) and 31568 (HTTPS), on each node.

For more information, see the AWS Load Balancers Documentation.

Helm Deployment Example

Step 1: Install the Ingress NGINX Controller with Helm

  1. Add the Helm Repository:

    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

  2. Install the Ingress NGINX Controller: Use Helm to install the controller in the ingress-nginx namespace:

    helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.service.type=LoadBalancer

  3. Verify the Installation: Check the pods and services in the ingress-nginx namespace:

    kubectl get pods -n ingress-nginx
kubectl get svc -n ingress-nginx

Expected output for the service:

NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.245.89.53    123.45.67.89     80:31567/TCP,443:31568/TCP    2m

ArgoCD Deployment Example

Deploying the Ingress NGINX Controller with ArgoCD

The following is an example of deploying the Ingress NGINX Controller using ArgoCD:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ingress-nginx
  namespace: argocd
spec:
  destination:
    namespace: ingress-nginx
    server: https://kubernetes.default.svc
  project: cluster-services
  source:
    chart: ingress-nginx
    helm:
      values: |
        controller:
          service:
            type: LoadBalancer
          publishService:
            enabled: true
          metrics:
            enabled: true
            prometheus:
              enabled: true
          allowSnippetAnnotations: true
          config:
            ssl-ciphers: "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:AES128-SHA256"
            ssl-protocols: "TLSv1.2 TLSv1.3"          
            proxy-connect-timeout: '15'
            proxy-read-timeout: '600'
            proxy-send-timeout: '600'
            hsts-include-subdomains: 'false'
            body-size: 64m
            server-name-hash-bucket-size: '256'
            client-max-body-size: 50m
          replicaCount: 3
          podAnnotations:
            "prometheus.io/scrape": "true"
            "prometheus.io/port": "10254"        
    repoURL: https://kubernetes.github.io/ingress-nginx
    targetRevision: 4.10.1
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      backoff:
        duration: 30m
        factor: 2
        maxDuration: 5m
      limit: 3
    syncOptions:
      - CreateNamespace=true

Key Features in the Configuration

  • Load Balancer: Configures the service type as LoadBalancer for external access.
  • Prometheus Metrics: Enables metrics collection for monitoring with Prometheus.
  • Timeouts and Limits: Configures proxy and client body limits to handle large requests.
  • Replicas: Deploys three replicas of the Ingress NGINX controller for high availability.
  • Automated Sync: Ensures that the deployment self-heals and prunes unused resources automatically.
  • Namespace Creation: Automatically creates the ingress-nginx namespace if it doesn’t exist.

Using Annotations to Customize the Load Balancer

AWS Load Balancers can be customized using annotations on the Kubernetes service resource. The following example demonstrates commonly used annotations:

Example Service with Annotations

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:region:account-id:certificate/certificate-id"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: "my-bucket"
    service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: "nginx-logs"
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
  ports:
    - port: 80
      targetPort: http
    - port: 443
      targetPort: https

Annotation Details

  • service.beta.kubernetes.io/aws-load-balancer-backend-protocol: Specifies the protocol for the backend (http or tcp).
  • service.beta.kubernetes.io/aws-load-balancer-ssl-cert: Associates an ACM certificate for HTTPS traffic.
  • service.beta.kubernetes.io/aws-load-balancer-ssl-ports: Defines which ports should use SSL.
  • service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: Enables access logs for the load balancer.
  • service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: Specifies the S3 bucket for storing access logs.
  • service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: Defines the log prefix within the S3 bucket.

For a complete list of available annotations, see the AWS Load Balancers Documentation.

References