4

In order to learn a bit more about K8S I started running a single server/node K3S cluster as a home lab. But I believe I've reached an impasse on my understanding of the network model, maybe specific to K3S.

So far so good, except I wanted to apply a TLS certificate to some services+ingresses I have set up.

In one of them, I have configured a TLS certificate as a secret and applied it to the Ingress associated with the service, however I always get the TRAEFIK DEFAULT CERT, which you can see here.

As I understand it, K3S comes with Traefik and ServiceLB pre-packaged in order to not rely on cloud-services' external load balancers (AWS etc). My first guess was that Traefik would "discover" the routes setup in my Ingress and proxy the TLS traffic, thus using the certificate I set up. This is clearly not the case, so I suppose I need to set up a TLS certificate for the Traefik instance itself.

My questions are then

  • How can I set up this certificate in K3S, and do I need a wildcard certificate if I'm planning on having multiple projects/domains hitting this cluster? (I would prefer managing the certificates per project)
  • What are the roles of ServiceLB and Traefik in K3S network model? If Traefik is getting the 80 and 443 traffic, is it just forwarding the traffic to ServiceLB which then forwards it to the Ingress resource?

In case it's needed, here's my ingress/service configuration


resource kubernetes_service_v1 snitch_service {
  metadata {
    name = "snitch"
    namespace = module.namespace.name
  }

spec {
    selector = {
      app = "snitch"
    }


type = "LoadBalancer"

port {
  name = "main"
  port = 3010
  target_port = 3000
  node_port = 30003
}



}
}


resource kubernetes_secret_v1 tls_secret {
  metadata {
    name = "snitch-tls-cert"
    namespace = "my-namespace"
  }


type = "kubernetes.io/tls"


data = {
    "tls.crt" = base64encode(my_certificate)
    "tls.key" = base64encode(my_certificate_private_key)
  }
}


resource kubernetes_ingress_v1 snitch_ingress {
  metadata {
    name = "snitch"
    namespace = "my-namespace"


annotations = {
  "ingress.kubernetes.io/ssl-redirect" = "false"
}



}


spec {
    tls {
      hosts = [local.subdomain]
      secret_name = "snitch-tls-cert"
    }


rule {
  host = local.subdomain

  http {
    path {
      path = "/"
      path_type = "Prefix"

      backend {
        service {
          name = "snitch"
          port {
            number = 3010
          }
        }
      }
    }
  }
}



}
}

Thank you!

Jo Colina
  • 171

2 Answers2

3

The ingress service is generally where TLS terminations happens -- that is, when you have a client like a web browser accessing an https:// url that points at your kubernetes cluster, the client is connecting to the ingress service, which negotiates the secure connection and then proxies the connection to your backend service (many ingress servers do have support for "pass-through" tls, where termination actually happens on the backend, but it's generally easier and more performant to have termination handled by the ingress service).

From the kubernetes documentation:

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.

k3s uses traefik as the ingress service, so you need to configure your ssl certificates in traefik.

Configuring the default certificate

In the absence of an ingress-specific tls certificate, traefik will use a default certificate for securing tls traffic. Out of the box, traefik will be using a self-signed certificate. Assuming that my k3s cluster is available at the hostname k3s.virt, we can see it like this:

$ openssl s_client -showcerts -connect k3s.virt:443 < /dev/null 2> /dev/null | openssl x509 -noout -subject -issuer
subject=CN = TRAEFIK DEFAULT CERT
issuer=CN = TRAEFIK DEFAULT CERT

We can follow the instructions in this article to replace the default certificate. If most of our services are going to be hosted in the same domain, it makes sense to use a wildcard certificate. If I create a new certificate for *.example.com and set up a TLSStore as described in the linked article, we can see the Traefik is now using the updated certificate:

$ openssl s_client -showcerts -connect k3s.virt:443 < /dev/null 2> /dev/null | openssl x509 -noout -subject -issuer -ext subjectAltName
subject=CN = default-router.example.com
issuer=CN = default-router.example.com
X509v3 Subject Alternative Name: 
    DNS:*.example.com

If I deploy a pod, service, and ingress, like this:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: whoami
  name: whoami
spec:
  ports:
  - name: http
    port: 80
    targetPort: http
  selector:
    app: whoami
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: whoami
  name: whoami
spec:
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
      - image: docker.io/traefik/whoami:latest
        name: whoami
        ports:
        - containerPort: 80
          name: http
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: whoami
  name: example
spec:
  ingressClassName: traefik
  rules:
  - host: whoami.example.com
    http:
      paths:
      - backend:
          service:
            name: whoami
            port:
              name: http
        path: /
        pathType: Prefix

Then (assuming that whoami.example.com resolves to the cluster) we will see that the ingress is using the default certificate:

$ curl -sk -vvI https://whoami.example.com
...
* Server certificate:
*  subject: CN=default-router.example.com
*  start date: Mar 14 12:02:54 2024 GMT
*  expire date: Mar 12 12:02:54 2034 GMT
*  issuer: CN=default-router.example.com
...

Configuring per-Ingress certificates

If we don't want to rely on the default certificate (e.g., if we need to use a hostname that is not part of the domain covered by the wildcard), we can configure a unique certificate for our service. This is described in the traefik ingress documentation.

First, we need to create a certificate and stuff it into a kubernetes secret. Then we need to update our Ingress to refer to the certificate by adding a spec.tls section:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: whoami
  name: example
spec:
  ingressClassName: traefik
  rules:
  - host: whoami.example.com
    http:
      paths:
      - backend:
          service:
            name: whoami
            port:
              name: http
        path: /
        pathType: Prefix
  tls:
  - secretName: whoami-certificate

With this resource in place, we see that the ingress is no longer using the default certificate:

$ curl -sk -vvI https://whoami.example.com
...
* Server certificate:
*  subject: CN=whoami.example.com
*  start date: Mar 14 12:07:16 2024 GMT
*  expire date: Mar 12 12:07:16 2034 GMT
*  issuer: CN=whoami.example.com
...

Hopefully that clears things up a bit. The documentation links I've provided here cover things in much more detail.

I have placed all the manifests I used to test this configuration in this repository.

Responding to your comments:

ServiceLB is the thing that watches for Services with type: LoadBalancer, and exposes those ports directly on cluster nodes (see e.g. these docs). For example, Traefik as a service in the kube-system namespace that looks like this:

ports:
  - name: web
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    port: 443
    protocol: TCP
    targetPort: websecure
type: LoadBalancer

So ServiceLB maps ports 80 and 443 on your cluster nodes to this service.

larsks
  • 47,453
1

I've managed to make the TLS work in part thanks @larsks' answer, it led me to understand that the K8S config is playnig around with the Traefik instance, and thus was able to find the logs and see the problem.

I'm writing an answer in case it becomes helpful for someone else in the future.

Essentially when you add/modify an Ingress ressource, K8S tells the Ingress controller to "configure" itself. In our case, the Traefik instance launches a configuration on itself depending on the defined Ingress (ie: adding routes), and if there is TLS defined it tries to add the certificate in the secret to its TLSStore.

Since the Traefik instance is applying new configuration, a ton of necessary information can be found on the logs. In my case, there were different issues preventing the certificate from being added to the TLSStore. Which means that Traefik only had the "DEFAULT TLS CERT" available to terminate. I was expecting it to use whatever I put into the secret, even if they were random strings.

For K3S, you can follow the logs with:

kubectl logs -f <the_traefik_pod_id> -n kube-system

Creating a self-signed certificate allowed me to validate the theory, since now my HTTPS traffic was being terminated by this self-signed certificate and not the default cert. @larsks answer validated that my config made sense. In case it's helpful to anyone, you can create a self-signed certificate and a new key with this command:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes

As a final touch, multiple tutorials talked about entering the "base64 encoded certificates" into the secret, which just means entering them in PEM format, and not base64-encoding the PEM itself (which contains a b64 block inside). The Traefik logs allowed me to detect this issue as well. You can see in the original question I was using base64encode on the secret. And if you happen to use Let's Encrypt, you can concatenate the issuer_pem and the certificate_pem in order to create the chain (in my case certificate_pem + issuer_pem in that order does the job).

Jo Colina
  • 171