1

I've been reading through The Book of Kubernetes by Alan Hohn and decided I would try to get the application described in the final chapter working with the Flannel CNI rather than Calico, which is what the application uses by default. The Longhorn storage system refuses to work with Flannel though. Only one longhorn-manager and longhorn-csi-plugin pods start (longhorn-manager-c46fd and longhorn-csi-plugin-nzw5v), both on node02 in example I'm providing. The longhorn-csi-plugin pods are attempting to connect to a socket called /csi/csi.sock (/var/lib/kubelet/plugins/driver.longhorn.io/csi.sock on the host system), which is only available on host02.

NAME                                                READY   STATUS             RESTARTS          AGE   IP            NODE     NOMINATED NODE   READINESS GATES
csi-attacher-698d78f74c-5lwfz                       0/1     CrashLoopBackOff   50 (4m57s ago)    46h   10.244.5.36   host06   <none>           <none>
csi-attacher-698d78f74c-cqdp7                       0/1     Error              53 (5m50s ago)    46h   10.244.3.31   host04   <none>           <none>
csi-attacher-698d78f74c-t68pd                       1/1     Running            51 (5m30s ago)    46h   10.244.4.30   host05   <none>           <none>
csi-provisioner-67f89896c9-7twtp                    1/1     Running            51 (5m22s ago)    46h   10.244.4.31   host05   <none>           <none>
csi-provisioner-67f89896c9-9jlsk                    0/1     CrashLoopBackOff   48 (4m39s ago)    46h   10.244.5.34   host06   <none>           <none>
csi-provisioner-67f89896c9-tmxdq                    0/1     Error              52 (5m42s ago)    46h   10.244.3.32   host04   <none>           <none>
csi-resizer-7966f575f5-27s6b                        1/1     Running            50 (5m25s ago)    46h   10.244.4.32   host05   <none>           <none>
csi-resizer-7966f575f5-67wcq                        0/1     CrashLoopBackOff   50 (4m50s ago)    46h   10.244.5.35   host06   <none>           <none>
csi-resizer-7966f575f5-zcv8m                        0/1     CrashLoopBackOff   51 (12s ago)      46h   10.244.3.35   host04   <none>           <none>
csi-snapshotter-845c75b5b6-8rp7f                    0/1     CrashLoopBackOff   49 (4m30s ago)    46h   10.244.5.32   host06   <none>           <none>
csi-snapshotter-845c75b5b6-trswb                    0/1     Error              51 (5m33s ago)    46h   10.244.3.33   host04   <none>           <none>
csi-snapshotter-845c75b5b6-z2kzn                    1/1     Running            49 (5m16s ago)    46h   10.244.4.33   host05   <none>           <none>
engine-image-ei-b0369a5d-6ccqj                      1/1     Running            3 (27m ago)       46h   10.244.0.15   host01   <none>           <none>
engine-image-ei-b0369a5d-6x486                      1/1     Running            3 (26m ago)       46h   10.244.2.15   host03   <none>           <none>
engine-image-ei-b0369a5d-fgnns                      1/1     Running            3 (25m ago)       46h   10.244.5.30   host06   <none>           <none>
engine-image-ei-b0369a5d-qj2xv                      1/1     Running            3 (26m ago)       46h   10.244.3.36   host04   <none>           <none>
engine-image-ei-b0369a5d-vg5n2                      1/1     Running            3 (27m ago)       46h   10.244.1.42   host02   <none>           <none>
engine-image-ei-b0369a5d-wlcbv                      1/1     Running            3 (25m ago)       46h   10.244.4.36   host05   <none>           <none>
instance-manager-37f197c012829700a256d86ef4357100   1/1     Running            0                 26m   10.244.1.47   host02   <none>           <none>
longhorn-csi-plugin-58wgq                           1/3     CrashLoopBackOff   105 (3m14s ago)   45h   10.244.3.29   host04   <none>           <none>
longhorn-csi-plugin-k4mmd                           1/3     CrashLoopBackOff   104 (3m9s ago)    45h   10.244.5.29   host06   <none>           <none>
longhorn-csi-plugin-mnz8z                           1/3     CrashLoopBackOff   108 (28s ago)     45h   10.244.2.14   host03   <none>           <none>
longhorn-csi-plugin-nzw5v                           3/3     Running            52 (12m ago)      46h   10.244.1.45   host02   <none>           <none>
longhorn-csi-plugin-r54wv                           1/3     CrashLoopBackOff   107 (2m57s ago)   45h   10.244.4.34   host05   <none>           <none>
longhorn-csi-plugin-v6b24                           1/3     CrashLoopBackOff   108 (39s ago)     45h   10.244.0.13   host01   <none>           <none>
longhorn-driver-deployer-56f94fbbd7-psgsk           1/1     Running            17 (25m ago)      46h   10.244.1.44   host02   <none>           <none>
longhorn-manager-c46fd                              1/1     Running            6 (26m ago)       46h   10.244.1.40   host02   <none>           <none>
longhorn-manager-fnp7s                              0/1     CrashLoopBackOff   38 (61s ago)      45h   10.244.5.31   host06   <none>           <none>
longhorn-manager-jpxrv                              0/1     CrashLoopBackOff   38 (97s ago)      45h   10.244.3.34   host04   <none>           <none>
longhorn-manager-kzlsm                              0/1     CrashLoopBackOff   37 (118s ago)     45h   10.244.0.14   host01   <none>           <none>
longhorn-manager-lnmtv                              0/1     CrashLoopBackOff   38 (110s ago)     45h   10.244.2.13   host03   <none>           <none>
longhorn-manager-s54g6                              0/1     CrashLoopBackOff   38 (68s ago)      45h   10.244.4.29   host05   <none>           <none>
longhorn-ui-74b6b85c94-7k4v5                        1/1     Running            5 (26m ago)       46h   10.244.1.46   host02   <none>           <none>
longhorn-ui-74b6b85c94-gfb22                        1/1     Running            6 (26m ago)       46h   10.244.1.41   host02   <none>           <none>

Logs from failing pods

longhorn-manager-fnp7s

W0624 19:29:48.578629       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2024-06-24T19:29:48Z" level=info msg="Starting longhorn conversion webhook server" func=webhook.StartWebhook file="webhook.go:24"
time="2024-06-24T19:29:48Z" level=info msg="Waiting for conversion webhook to become ready" func=webhook.StartWebhook file="webhook.go:43"
time="2024-06-24T19:29:48Z" level=warning msg="Failed to check endpoint https://localhost:9501/v1/healthz" func=webhook.isServiceAvailable file="webhook.go:78" error="Get \"https://localhost:9501/v1/healthz\": dial tcp [::1]:9501: connect: connection refused"
time="2024-06-24T19:29:48Z" level=info msg="Active TLS secret longhorn-system/longhorn-webhook-tls (ver=5907) (count 2): map[listener.cattle.io/cn-longhorn-admission-webhook.longhor-59584d:longhorn-admission-webhook.longhorn-system.svc listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc listener.cattle.io/fingerprint:SHA1=A5D6010EC627A8877305D8828712E103B103C5B3]" func="memory.(*memory).Update" file="memory.go:42"
time="2024-06-24T19:29:48Z" level=info msg="Listening on :9501" func=server.ListenAndServe.func2 file="server.go:77"
time="2024-06-24T19:29:48Z" level=info msg="Starting apiregistration.k8s.io/v1, Kind=APIService controller" func="controller.(*controller).run" file="controller.go:144"
time="2024-06-24T19:29:48Z" level=info msg="Starting apiextensions.k8s.io/v1, Kind=CustomResourceDefinition controller" func="controller.(*controller).run" file="controller.go:144"
time="2024-06-24T19:29:48Z" level=info msg="Starting /v1, Kind=Secret controller" func="controller.(*controller).run" file="controller.go:144"
time="2024-06-24T19:29:48Z" level=info msg="Building conversion rules..." func="server.(*WebhookServer).runConversionWebhookListenAndServe.func1" file="server.go:195"
time="2024-06-24T19:29:48Z" level=info msg="Updating TLS secret for longhorn-system/longhorn-webhook-tls (count: 2): map[listener.cattle.io/cn-longhorn-admission-webhook.longhor-59584d:longhorn-admission-webhook.longhorn-system.svc listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc listener.cattle.io/fingerprint:SHA1=A5D6010EC627A8877305D8828712E103B103C5B3]" func="kubernetes.(*storage).saveInK8s" file="controller.go:225"
time="2024-06-24T19:29:50Z" level=info msg="Started longhorn conversion webhook server on localhost" func=webhook.StartWebhook file="webhook.go:47"
time="2024-06-24T19:29:51Z" level=warning msg="Failed to check endpoint https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/healthz" func=webhook.isServiceAvailable file="webhook.go:78" error="Get \"https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
time="2024-06-24T19:29:54Z" level=warning msg="Failed to check endpoint https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/healthz" func=webhook.isServiceAvailable file="webhook.go:78" error="Get \"https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

longhorn-csi-plugin-58wgq

I0624 19:28:16.188843   64939 main.go:135] Version: v2.9.2
I0624 19:28:16.188921   64939 main.go:136] Running node-driver-registrar in mode=
I0624 19:28:16.188948   64939 main.go:157] Attempting to open a gRPC connection with: "/csi/csi.sock"
W0624 19:28:26.189744   64939 connection.go:232] Still connecting to unix:///csi/csi.sock
W0624 19:28:36.189205   64939 connection.go:232] Still connecting to unix:///csi/csi.sock
W0624 19:28:46.190515   64939 connection.go:232] Still connecting to unix:///csi/csi.sock
E0624 19:28:46.190535   64939 main.go:160] error connecting to CSI driver: context deadline exceeded
Robert
  • 63

1 Answers1

0
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: longhorn-manager
  namespace: longhorn-system
spec:
  selector:
    matchLabels:
      app: longhorn-manager
  template:
    metadata:
      labels:
        app: longhorn-manager
    spec:
      hostNetwork: true
      containers:
        - name: longhorn-manager
          image: longhornio/longhorn-manager:v1.2.0
          volumeMounts:
            - mountPath: /var/lib/rancher/longhorn
              name: longhorn
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
      volumes:
        - name: longhorn
          hostPath:
            path: /var/lib/rancher/longhorn
            type: DirectoryOrCreate
warashi nguyen
  • 182