I am using Cilium and Kubernetes Gateway API in Amazon EKS.
I am trying to expose one of my service (Goldilocks in this case at hm-goldilocks-dashboard.production-hm-goldilocks.svc:80) at https://hm-goldilocks.internal.mydomain.com
Here is my setup:
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: staging-lets-encrypt-cluster-issuer
namespace: production-hm-cert-manager
spec:
acme:
email: me@mydomain.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: staging-lets-encrypt-cluster-issuer-account-secret
solvers:
- selector:
dnsZones:
- internal.mydomain.com
dns01:
route53:
region: us-west-2
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: staging-hm-cilium-gateway-certificate
namespace: production-hm-gateway-api
spec:
commonName: "*.internal.mydomain.com"
secretName: staging-hm-cilium-gateway-certificate
dnsNames:
- "*.internal.mydomain.com"
issuerRef:
kind: ClusterIssuer
name: staging-lets-encrypt-cluster-issuer
# https://letsencrypt.org/certificates
privateKey:
algorithm: ECDSA
size: 384
rotationPolicy: Always
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: hm-cilium-gateway
namespace: production-hm-gateway-api
labels:
app.kubernetes.io/name: hm-cilium-gateway
app.kubernetes.io/part-of: production-hm-gateway-api
spec:
gatewayClassName: cilium
infrastructure:
annotations:
# https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations
# https://cloud-provider-aws.sigs.k8s.io/service_controller
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-xx,subnet-xx,subnet-xx,subnet-xx
listeners:
- name: http
protocol: HTTP
hostname: "*.internal.mydomain.com"
port: 80
allowedRoutes:
namespaces:
from: All
- name: https
protocol: HTTPS
hostname: "*.internal.mydomain.com"
port: 443
allowedRoutes:
namespaces:
from: All
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: staging-hm-cilium-gateway-certificate
namespace: production-hm-gateway-api
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: hm-goldilocks-http-httproute
namespace: production-hm-goldilocks
spec:
parentRefs:
- name: hm-cilium-gateway
namespace: production-hm-gateway-api
sectionName: http
hostnames:
- hm-goldilocks.internal.mydomain.com
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
port: 443
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: hm-goldilocks-https-httproute
namespace: production-hm-goldilocks
spec:
parentRefs:
- name: hm-cilium-gateway
namespace: production-hm-gateway-api
sectionName: https
hostnames:
- hm-goldilocks.internal.mydomain.com
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: hm-goldilocks-dashboard
namespace: production-hm-goldilocks
kind: Service
port: 80
I can see that External DNS successfully registered with Route 53.
And Gateway's corresponding Load Balancer:
Here is my Gateway status:
➜ kubectl describe gateway hm-cilium-gateway -n production-hm-gateway-api
Name: hm-cilium-gateway
Namespace: production-hm-gateway-api
Labels: app.kubernetes.io/name=hm-cilium-gateway
app.kubernetes.io/part-of=production-hm-gateway-api
argocd.argoproj.io/instance=hm-gateway-api
Annotations: <none>
API Version: gateway.networking.k8s.io/v1
Kind: Gateway
Metadata:
Creation Timestamp: 2024-08-01T21:07:08Z
Generation: 1
Resource Version: 1237234
UID: c17f7a60-38ff-462a-90c6-b725066068c5
Spec:
Gateway Class Name: cilium
Infrastructure:
Annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: true
service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-xx,subnet-xx,subnet-xx,subnet-xx
service.beta.kubernetes.io/aws-load-balancer-type: nlb
Listeners:
Allowed Routes:
Namespaces:
From: All
Hostname: *.internal.mydomain.com
Name: http
Port: 80
Protocol: HTTP
Allowed Routes:
Namespaces:
From: All
Hostname: *.internal.mydomain.com
Name: https
Port: 443
Protocol: HTTPS
Tls:
Certificate Refs:
Group:
Kind: Secret
Name: staging-hm-cilium-gateway-certificate
Namespace: production-hm-gateway-api
Mode: Terminate
Status:
Addresses:
Type: Hostname
Value: aa3ff376273144bd089a876b9417e2fd-3fadba626fbed2b5.elb.us-west-2.amazonaws.com
Conditions:
Last Transition Time: 2024-08-01T21:07:08Z
Message: Gateway successfully scheduled
Observed Generation: 1
Reason: Accepted
Status: True
Type: Accepted
Last Transition Time: 2024-08-01T21:07:12Z
Message: Gateway successfully reconciled
Observed Generation: 1
Reason: Programmed
Status: True
Type: Programmed
Listeners:
Attached Routes: 1
Conditions:
Last Transition Time: 2024-08-01T21:41:06Z
Message: Listener Programmed
Observed Generation: 1
Reason: Programmed
Status: True
Type: Programmed
Last Transition Time: 2024-08-01T21:41:06Z
Message: Listener Accepted
Observed Generation: 1
Reason: Accepted
Status: True
Type: Accepted
Last Transition Time: 2024-08-01T21:41:06Z
Message: Resolved Refs
Reason: ResolvedRefs
Status: True
Type: ResolvedRefs
Name: http
Supported Kinds:
Group: gateway.networking.k8s.io
Kind: HTTPRoute
Attached Routes: 1
Conditions:
Last Transition Time: 2024-08-01T21:41:06Z
Message: Listener Programmed
Observed Generation: 1
Reason: Programmed
Status: True
Type: Programmed
Last Transition Time: 2024-08-01T21:41:06Z
Message: Listener Accepted
Observed Generation: 1
Reason: Accepted
Status: True
Type: Accepted
Last Transition Time: 2024-08-01T21:41:06Z
Message: Resolved Refs
Reason: ResolvedRefs
Status: True
Type: ResolvedRefs
Name: https
Supported Kinds:
Group: gateway.networking.k8s.io
Kind: HTTPRoute
Events: <none>
My http HTTPRoute status:
➜ kubectl describe httproute hm-goldilocks-http-httproute -n production-hm-goldilocks
Name: hm-goldilocks-http-httproute
Namespace: production-hm-goldilocks
Labels: app.kubernetes.io/name=hm-goldilocks-http-httproute
app.kubernetes.io/part-of=production-hm-goldilocks
argocd.argoproj.io/instance=hm-goldilocks
Annotations: argocd.argoproj.io/hook: PostSync
API Version: gateway.networking.k8s.io/v1
Kind: HTTPRoute
Metadata:
Creation Timestamp: 2024-08-01T21:41:06Z
Generation: 1
Resource Version: 1237228
UID: 33184787-1469-473f-85f7-4e2d5df491d4
Spec:
Hostnames:
hm-goldilocks.internal.mydomain.com
Parent Refs:
Group: gateway.networking.k8s.io
Kind: Gateway
Name: hm-cilium-gateway
Namespace: production-hm-gateway-api
Section Name: http
Rules:
Filters:
Request Redirect:
Port: 443
Scheme: https
Status Code: 302
Type: RequestRedirect
Matches:
Path:
Type: PathPrefix
Value: /
Status:
Parents:
Conditions:
Last Transition Time: 2024-08-01T21:41:06Z
Message: Accepted HTTPRoute
Observed Generation: 1
Reason: Accepted
Status: True
Type: Accepted
Last Transition Time: 2024-08-01T21:41:06Z
Message: Service reference is valid
Observed Generation: 1
Reason: ResolvedRefs
Status: True
Type: ResolvedRefs
Controller Name: io.cilium/gateway-controller
Parent Ref:
Group: gateway.networking.k8s.io
Kind: Gateway
Name: hm-cilium-gateway
Namespace: production-hm-gateway-api
Section Name: http
Events: <none>
My https HTTPRoute status:
➜ kubectl describe httproute hm-goldilocks-https-httproute -n production-hm-goldilocks
Name: hm-goldilocks-https-httproute
Namespace: production-hm-goldilocks
Labels: app.kubernetes.io/name=hm-goldilocks-https-httproute
app.kubernetes.io/part-of=production-hm-goldilocks
argocd.argoproj.io/instance=hm-goldilocks
Annotations: argocd.argoproj.io/hook: PostSync
API Version: gateway.networking.k8s.io/v1
Kind: HTTPRoute
Metadata:
Creation Timestamp: 2024-08-01T21:41:06Z
Generation: 1
Resource Version: 1237227
UID: af2c2c67-4ff4-4c5d-963f-0c2c6222b04f
Spec:
Hostnames:
hm-goldilocks.internal.mydomain.com
Parent Refs:
Group: gateway.networking.k8s.io
Kind: Gateway
Name: hm-cilium-gateway
Namespace: production-hm-gateway-api
Section Name: https
Rules:
Backend Refs:
Group:
Kind: Service
Name: hm-goldilocks-dashboard
Namespace: production-hm-goldilocks
Port: 80
Weight: 1
Matches:
Path:
Type: PathPrefix
Value: /
Status:
Parents:
Conditions:
Last Transition Time: 2024-08-01T21:41:06Z
Message: Accepted HTTPRoute
Observed Generation: 1
Reason: Accepted
Status: True
Type: Accepted
Last Transition Time: 2024-08-01T21:41:06Z
Message: Service reference is valid
Observed Generation: 1
Reason: ResolvedRefs
Status: True
Type: ResolvedRefs
Controller Name: io.cilium/gateway-controller
Parent Ref:
Group: gateway.networking.k8s.io
Kind: Gateway
Name: hm-cilium-gateway
Namespace: production-hm-gateway-api
Section Name: https
Events: <none>
I can see http redirects to https, however, the https failed to point the Service in the Kubernetes:
My Cilium Helm values.yaml
eni:
enabled: true
ipam:
mode: eni
egressMasqueradeInterfaces: eth0
routingMode: native
kubeProxyReplacement: true
k8sServiceHost: xx.gr7.us-west-2.eks.amazonaws.com
k8sServicePort: 443
rollOutCiliumPods: true
operator:
enabled: true
rollOutPods: true
l7Proxy: true
gatewayAPI:
enabled: true
gatewayClass:
create: "true"
hubble:
enabled: true
relay:
enabled: true
ui:
enabled: true
Cilium Version
➜ cilium version --helm-release-name=hm-cilium --namespace=production-hm-cilium
cilium-cli: v0.16.13 compiled with go1.22.5 on darwin/arm64
cilium image (default): v1.15.6
cilium image (stable): v1.16.0
cilium image (running): 1.15.7
Kubernetes Version
➜ kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.2-eks-db838b0
Any guide would be appreciate, thanks!
