0

I've got an on premise kubernetes cluster with a NodePort service running on port 32001. But in 80% of the cases requests fail:

curl localhost:32001

times out and sometimes it works, results are returned

Als starting an nginx pod (ng) in the cluster and calling

wget -O- http://ng:80

from a busybox does not work ever. While in another cluster it works.

We use flannel CNI so I checked logs of flannel pods and kube-proxy but can't find anything weird.

Any ideas what to check for? What can cause a sometimes timeout and sometimes success?

Serve Laurijssen
  • 594
  • 2
  • 8
  • 17

1 Answers1

1

Found the cause, maybe someone will find it useful in the future.

This happened because flannel is not correctly set up for an on premise kubernetes environment.

I installed the default kube-flannel.yaml but two things need to be changed first in on premise environments.

  1. The specific ip of the network interface must be passed to flanneld as a commandline argument. Otherwise flannel takes the first network interface.

     command:
 - /opt/bin/flanneld
 args:
 - --ip-masq
 - --kube-subnet-mgr
 - --iface=enp0s8

  2. The default config has "backend type" VXLAN which causes the vxlan.go package code to be triggered in flannel. This works in azure, gcloud and AWS etc. but not on premise. Type host-gw must be configured so the hostgw code is run which tries to connect to other machines with layer2 routing.

    net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "host-gw" } }

Checking out flannel's source code for hostgw:

https://github.com/flannel-io/flannel/blob/master/pkg/backend/hostgw/hostgw.go

every backend is added with it's "constructor" New function during startup so in the case of host-gw where ip address of extIface is the one passed with --iface option.

func init() {
    backend.Register("host-gw", New)
}

func New(sm subnet.Manager, extIface *backend.ExternalInterface) (backend.Backend, error) {
    if !extIface.ExtAddr.Equal(extIface.IfaceAddr) {
        return nil, fmt.Errorf("PublicIP differs from interface IP probably on NAT is not supported by host-gw")
    }


be := &HostgwBackend{
    sm:       sm,
    extIface: extIface,
}
return be, nil



}

The HostgwBackend is then instantiated in main.go and RegisterNetwork is called.

bm := backend.NewManager(ctx, sm, extIface)
be, err := bm.GetBackend(config.BackendType)
bn, err := be.RegisterNetwork(ctx, &wg, config)

And in RegisterNetwork the function is set up which communicates the pod routes to the "pod-network-cidr" (10.244.0.0/24) of the cluster to the kernel with the netlink package when it calls AcquireLease and Lease. Then to retrieve the routes it uses netlink again to query for the routes.

    n.GetRoute = func(lease *lease.Lease) *netlink.Route {
        return &netlink.Route{
            Dst:       lease.Subnet.ToIPNet(),
            Gw:        lease.Attrs.PublicIP.ToIP(),
            LinkIndex: n.LinkIndex,
        }
    }

So after initialization 'ip route' shows all the created routes.

ip route show

10.244.1.0/24 via 10.203.32.120 dev ens160 
10.244.2.0/24 via 10.203.32.98 dev ens160 
10.244.3.0/24 via 10.203.32.111 dev ens160 
10.244.4.0/24 via 10.203.32.99 dev ens160 
10.244.5.0/24 via 10.203.32.110 dev ens160 
10.244.6.0/24 via 10.203.32.121 dev ens160
Serve Laurijssen
  • 594
  • 2
  • 8
  • 17