6/30/2024
I found what I posted on 6/27/2024 is misleading. Even it repaired the issue in the question, but only on the surface. The Certificate still failed to issue.
It took me some time to make everything including cert-manager, ExternalDNS, Traefik work well.
Also, it turns out acme.cert-manager.io/http01-edit-in-place: "true" annotation is not necessary to fix the issue in the question.
Note for the solution in my case, I am using DNS01 challenge.
I also published all my deployment codes here. Hopefully it can give a bigger picture how it get deployed.
cert-manager AWS IAM role CertManagerRole
Trust relationships
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:aud": "sts.amazonaws.com",
"oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:sub": "system:serviceaccount:production-hm-cert-manager:hm-cert-manager"
}
}
}
]
}
Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"route53:GetChange"
],
"Effect": "Allow",
"Resource": "arn:aws:route53:::change/*"
},
{
"Action": [
"route53:ListResourceRecordSets"
],
"Effect": "Allow",
"Resource": "arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx"
},
{
"Action": [
"route53:ChangeResourceRecordSets"
],
"Condition": {
"ForAllValues:StringEquals": {
"route53:ChangeResourceRecordSetsNormalizedRecordNames": [
"_acme-challenge.*"
],
"route53:ChangeResourceRecordSetsRecordTypes": [
"TXT"
]
}
},
"Effect": "Allow",
"Resource": "arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx"
},
{
"Action": [
"route53:ListHostedZonesByName"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
ExternalDNS AWS IAM role ExternalDNSRole
Trust relationships
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:sub": "system:serviceaccount:production-hm-external-dns:hm-external-dns",
"oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:aud": "sts.amazonaws.com"
}
}
}
]
}
Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"route53:ChangeResourceRecordSets"
],
"Effect": "Allow",
"Resource": [
"arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx"
]
},
{
"Action": [
"route53:ListHostedZones",
"route53:ListResourceRecordSets",
"route53:ListTagsForResource"
],
"Effect": "Allow",
"Resource": [
"*"
]
}
]
}
cert-manager Helm chart custom values.yaml
crds:
enabled: true
keep: true
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/CertManagerRole
securityContext:
fsGroup: 1001
ExternalDNS Helm chart custom values.yaml
provider: aws
aws:
region: us-west-2
zoneType: public
txtOwnerId: Z1XXXXXXXXXXXXXXXXXXX
domainFilters:
- internal.example.com
registry: txt
policy: sync
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/ExternalDNSRole
Traefik Helm chart custom values.yaml
providers:
kubernetesIngress:
publishedService:
enabled: true
service:
enabled: true
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: production-lets-encrypt-cluster-issuer
namespace: production-hm-cert-manager
spec:
acme:
email: me@example.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: production-lets-encrypt-cluster-issuer-account-secret
solvers:
- selector:
dnsZones:
- internal.example.com
dns01:
route53:
region: us-west-2
Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hm-airbyte-ingress
namespace: production-hm-airbyte
annotations:
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
cert-manager.io/cluster-issuer: production-lets-encrypt-cluster-issuer
spec:
rules:
- host: hm-airbyte.internal.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hm-airbyte-airbyte-webapp-svc
port:
number: 80
tls:
- hosts:
- hm-airbyte.internal.example.com
secretName: production-hm-airbyte-certificate
6/27/2024 (Misleading Answer)
After a lot of experiments, I found if I add acme.cert-manager.io/http01-edit-in-place: "true" annotation, it helps resolve the issue.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hm-airbyte-ingress
namespace: production-hm-airbyte
annotations:
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
cert-manager.io/cluster-issuer: production-lets-encrypt-issuer
acme.cert-manager.io/http01-edit-in-place: "true" # <- Added this
labels:
app.kubernetes.io/name: hm-airbyte-ingress
app.kubernetes.io/part-of: production-hm-airbyte
spec:
rules:
- host: hm-airbyte.internal.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hm-airbyte-airbyte-webapp-svc
port:
number: 80
tls:
- hosts:
- hm-airbyte.internal.example.com
secretName: hm-airbyte-ingress-tls
So now here is what happens after I apply this Ingress file
- A temporary
Secret: hm-airbyte-ingress-tls-7w9z7 with random suffix will be created
- After about half minute, a
Secret: hm-airbyte-ingress-tls without suffix will be created
- The one will suffix will be automatically deleted.
Without this annotation, based on my experiment, it will be stuck at step 1 forever.
This is the Secret: hm-airbyte-ingress-tls without suffix, you can see the difference between the temporary secret I posted in the question.
Here is the explanation of this annotation:
acme.cert-manager.io/http01-edit-in-place: "true": this controls whether the ingress is modified 'in-place', or a new one is created specifically for the HTTP01 challenge. If present, and set to "true", the existing ingress will be modified. Any other value, or the absence of the annotation assumes "false". This annotation will also add the annotation "cert-manager.io/issue-temporary-certificate": "true" onto created certificates which will cause a temporary certificate to be set on the resulting Secret until the final signed certificate has been returned. This is useful for keeping compatibility with the ingress-gce component.
P.S. I cannot explain why this annotation helps resolve the issue. Because I actually expect if with this annotation, the temporary secret will not be created at all. Hope someone can further explain in future, thanks!
