1

I am currently facing an issue with cert-manager while trying to obtain a Let's Encrypt certificate for my Kubernetes cluster running on Scaleway Kapsule.

I have successfully installed cert-manager from the helm chart. However, I am now encountering a problem with the challenges for domain validation.

I created an issuer:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: my@email.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - selector: {}
      http01:
        ingress:
          ingressClassName: nginx

and annotated the ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /api/(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: backend
            port:
              number: 8080
      - path: /?(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: frontend
            port:
              number: 3000
  tls:
  - hosts:
    - example.com
    secretName: frontend-ingress-cert

This creates the cm-acme-http-solver:

NAME                        CLASS   HOSTS                         ADDRESS          PORTS     AGE
cm-acme-http-solver-t8pb9   nginx   example.com                   51.158.131.xxx   80        11m

This seems to work fine, Certificate, -Request and Order are created, but the challenge is stuck performing the self check:

Name:         frontend-ingress-cert-1-1328491808-3444115809
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2024-04-02T15:07:31Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  frontend-ingress-cert-1-1328491808
    UID:                   e85e8c53-5965-4be4-87fa-50359217539f
  Resource Version:        10398552697
  UID:                     73e25122-6340-406f-aa68-632e92adec4b
Spec:
  Authorization URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/123456789
  Dns Name:           example.com
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   Issuer
    Name:   letsencrypt-prod
  Key:      <redacted>>.NaUj9lmjGqlcRba77ZLjqqNfT_qWVn2WdlTDo0K4NMI
  Solver:
    http01:
      Ingress:
        Ingress Class Name:  nginx
    Selector:
  Token:     <redacted>>
  Type:      HTTP-01
  URL:       https://acme-v02.api.letsencrypt.org/acme/chall-v3/123456789/ZaFzbg
  Wildcard:  false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://example.com/.well-known/acme-challenge/<redacted>>': Get "http://example.com/.well-known/acme-challenge/<redacted>>": EOF
  State:       pending
Events:
  Type    Reason     Age    From                     Message
  ----    ------     ----   ----                     -------
  Normal  Started    5m40s  cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  5m39s  cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism

Accessing the challenge url from outside the cluster works fine, but from within the cluster, the request fails:

❯ kubectl exec -it frontend-775d68f5-7rq7x -- sh
/app $ wget http://example.com/.well-known/acme-challenge/<redacted>>
Connecting to mydomain.com (51.158.131.xxx:80)
wget: error getting response: Resource temporarily unavailable

Nginx logs this:

2024/04/02 15:16:03 [error] 3555#3555: *60767 broken header: "GET /.well-known/acme-challenge/<redacted>> HTTP/1.1" while reading PROXY protocol, client: 100.64.0.110, server: 0.0.0.0:80
cvk
  • 111
  • 2

1 Answers1

0

For anyone running into the same problem: cert-manager's issue tracker has an excellent discussion about the problem dating back to 2018.

The solution for me was to install compumike's hairpin-proxy to enable PROXY protocol support for internal-to-LoadBalancer traffic.

cvk
  • 111
  • 2