4

I'm trying to write an autoscaling configuration for ingress-nginx, deployed via helm chart.

my goals are:

  • 3 minimum replicas (because I have 3 nodes minimum)
  • ensure only one nginx per node, but:
  • be elastic, if autoscale says we need 4 nginx allow for a node in the cluster to have 2
  • if a fourth node gets added, ensure a new nginx gets spawned

https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml#L326 https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml#L343 https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml#L256

I tried playing with the below settings and a combination of them but there's always something off, for example right now I have a fourth nginx that want to spawn for some reason and it can't because of the antiaffinity rule.

Can anyone share some ideas how to achieve that?

  • always one nginx per node, if a new node gets created, a new nginx gets created
  • preserve autoscaling, if hpa wants to spawn a fourth nginx on a 3 node cluster it should be free to do so
      replicaCount: 3
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                - ingress-nginx
              - key: app.kubernetes.io/instance
                operator: In
                values:
                - ingress-nginx
              - key: app.kubernetes.io/component
                operator: In
                values:
                - controller
            topologyKey: "kubernetes.io/hostname"

  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels:
          app.kubernetes.io/instance: ingress-nginx

  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 6
    targetCPUUtilizationPercentage: 75
    targetMemoryUtilizationPercentage: 100
John Smith
  • 288

1 Answers1

3

If you need only one nginx instance per node, wouldn't it be easier to deploy it as DaemonSet ?

As you can read in the official docs:

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

And this seems like the right solution in your use case.

As you rightly suggested in your comment, nginx-ingress official helm chart can be deployed both as a Deployment or as a DaemonSet. It can be done by adjusting its values.yaml file.

mario
  • 635