kubernetes cert-manager failing with certificate chain is malformed or broken

Question

I have setup cert-manager to sign the certificate with the private ca issuer. the private CA secret is setup correctly and before adding to the TLS secret i have verified the chain with OpenSSL verify command and they all verify to the root correctly. But when the same is done with cert-manager, the certificate is not signed and i get the error " Message: The certificate request has failed to complete and will be retried: Error signing certificate: certificate chain is malformed or broken"

Here is the yaml for the cert issuer

   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
    name: my-ca-issuer
    namespace: default
   spec:
   ca:
      secretName: my-tls-secret
   ---
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: cert-test
     namespace: default
   spec:
     secretName: my-tls-secret
   issuerRef:
     name: my-ca-issuer
     kind: ClusterIssuer
   commonName: cert-test.com
   renewBefore: 12h # Renew the certificate when it's 12 hours from expiration

my-tls-secret is in the cert-manager namespace

the ca issuer verifies correctly

  kubectl get clusterissuers -o wide
NAME                        READY   STATUS                AGE
my-ca-issuer            True    Signing CA verified   27h

the my-tls secret was created using this command

 kubectl create secret generic my-tls-secret --from-file=tls.crt=cert-chain.pem --from-file=tls.key=myca.key --from-file=ca.crt=ca-chain.pem -n cert-manager

Any clues on why the chain is malformed with cert-manager whereas openssl verify doesnt have any issue with the chain

openssl crl2pkcs7 -nocrl -certfile ca-chain.pem | openssl pkcs7 -print_certs -noout
subject=C = US, ST = TX, L = Austin, O = org, CN = app-TLS-Sub-CA
issuer=C = US, ST = TX, L = Austin, O = org, CN = Issuing-Sub-CA
subject=C = US, ST = TX, L = Austin, O = org, CN = Issuing-Sub-CA
issuer=C = US, ST = TX, L = Austin, O = org, CN = Root
subject=C = US, ST = TX, L = Austin, O = org, CN = Root
issuer=C = US, ST = TX, L = Austin, O = org, CN = Root

score 3 · Accepted Answer · answered Nov 26 '23 at 00:21

From your comments, the issue persists even after ensuring the correct order and concatenation of the certificates.

[ Kubernetes Cluster ]
         |
   [ cert-manager ]
         | \
         |  [ Increased Log Verbosity ]
         |
[ my-ca-issuer ClusterIssuer ]
         | \
         |  [ Check CA Permissions ]
         |
[ my-tls-secret (CA secret) ]
         | \
         |  [ Verify Certificate Chain Formatting ]
         |
[ cert-test Certificate ]
         |
   [ CertificateRequest Debugging ]

While OpenSSL is able to verify the chain, cert-manager in the Kubernetes context might interpret it differently. That could be due to how cert-manager handles certificate parsing.
Consider using a tool like kubectl describe on the Certificate and CertificateRequest resources to get more clues.

kubectl describe certificate cert-name -n default
kubectl describe certificaterequest -n default
kubectl describe clusterissuer my-ca-issuer

Sometimes, certificate files may contain hidden characters or formatting issues not easily visible. Make sure the PEM files do not contain any extraneous characters or incorrect line endings. You can use tools like cat -v or a text editor with hidden character visibility to inspect the files.

Make sure the CA represented by my-tls-secret has the necessary permissions to sign certificates. Some CAs might be restricted to only sign certain types or levels of certificates. That is particularly relevant since you have multiple levels in your CA hierarchy.

Increase the log verbosity of cert-manager to get more detailed information about what is happening during the signing process.

kubectl get deployments -n cert-manager
kubectl edit deployment <cert-manager-deployment-name> -n cert-manager
spec:
  containers:

args:
--v=4  # This sets log verbosity to level 4

...

Just in case, the depth of your certificate chain might be causing issues. While not common, some systems have limits on the chain depth they can process. If possible, test with a simpler chain (fewer levels) to see if the issue persists.

score 0 · Answer 2 · answered Nov 25 '23 at 07:30

It seems like the issue might be related to the format or content of the certificate chain in the my-tls-secret secret. You should ensure the chain in tls.crt is correctly ordered and contains all necessary intermediate certificates.

Try concatenating your certificates in the correct order: your leaf certificate first, then the intermediates (if any), and the root certificate last. Then update the my-tls-secret with the correctly ordered chain.

Like this:

kubectl create secret generic my-tls-secret --from-file=tls.crt=combined-chain.pem --from-file=tls.key=myca.key --namespace=cert-manager

Then combine the certificates:

cat leaf-cert.pem intermediate-cert.pem root-cert.pem > combined-chain.pem

Let me know how this goes!

kubernetes cert-manager failing with certificate chain is malformed or broken

2 Answers2