3

I am new to deploying on-premise Kubernetes clusters. I managed to deploy one using kubeadm with weave CNI.

Internet connection inside Pods is somewhat sluggish / non-working. I am trying to deploy Self-Hosted GitHub Actions Runner which is accessing GitHub API. For example accessing *.github.com FQDN from Pods will result in this error (coredns Pod Logs): [ERROR] plugin/errors: 2 api.github.com. A: dns: buffer size too small

I also observed that Pods won't connect to Ubuntu or Debian servers but nslookup inside pod works (using dnsutils pod from K8s).

Host machine can access every site so I assume that there is no problem in Firewall.

ufw and iptables are disabled.

I set bufsize to 512, 1024 and 4096 but still the same error. By default bufsize in ConfigMap is not defined, but It gave me the same error.

CoreDNS ConfigMap ($ kubectl describe configmap coredns -n kube-system):

Name:         coredns
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Data


Corefile:


.:53 {
    bufsize 4096
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}
BinaryData
====
Events:  <none>

How I deployed Cluster:

kubeadm init --pod-network-cidr=172.30.0.0/16 --service-cidr=172.16.0.0/16 --cri-socket=unix:/run/cri-dockerd.sock

kubectl taint nodes --all node-role.kubernetes.io/control-plane-


Install weave


sudo curl -L git.io/weave -o /usr/local/bin/weave
sudo chmod a+x /usr/local/bin/weave


Download weave DaemonSet from Github


wget https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

Edit to:

...
containers:
  - name: weave
    env:
      - name: IPALLOC_RANGE
    value: 172.30.0.0/16
...

Finally: kubectl apply -f ./weave-daemonset-k8s.yaml

Versions:

$ kubectl version
    Client Version: v1.28.2
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
    Server Version: v1.28.2

$ kubectl describe configmap coredns -n kube-system
    image: registry.k8s.io/coredns/coredns:v1.10.1

What can be the cause?

Tried:

  • kubeadm reset
  • Different CIDR for Node and Service
  • Default weave config
  • Edit weaveconfig - IPALLOC_RANGE to match Node CIDR
  • dnsutils Pods
    • github.com
      • nslookup & dig - Success
      • ping - Failure - unknown host - buffer too small Error
    • api.github.com
      • nslookup - buffer too small Error
      • dig - Success
    • kubernetes.io
      • nslookup & dig - Success

Expectations:

  • Working network connection in Pods
  • Successful DNS resolution

EDIT

I updated CoreDNS to version 1.11.0 and now there is no buffer too small error but instead it shows Request Timed out. At least I can see that it is correctly trying to Query from my main DNS which I tested multiple times and works as expected.

WhenGrill
  • 31

1 Answers1

0

Seems like the latest CoreDNS does not support DNS compression and the response goes over the limit 512 bytes (UDP limit).

First ping/wget... the address that does not work.

To trace the bag run:

kubectl get pods -n kube-system -l k8s-app=kube-dns

kubectl logs coredns-xxxxxxx -n kube-system

You will see the following in the results

[ERROR] plugin/errors: 2 checkip.amazonaws.com. A: dns: buffer size too small.

Modify 'forward' in the config

kubectl edit configmap coredns -n kube-system

forward .  /etc/resolv.conf {
          force_tcp
          max_concurrent 1000
        }

Save & Restart Docker Desktop Here is more about forward attribute https://coredns.io/plugins/forward/

James B.
  • 1