1

I've been doing some work with k3s, and everything was working fine. Now I'm getting the following error when I restart and I try to run the port-forward command I had previously run to get to the service I want,

E0921 19:32:10.085091 3083215 portforward.go:400] an error occurred forwarding 3000 -> 3000: error forwarding port 3000 to pod ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df, uid : failed to execute portforward in network namespace "/var/run/netns/cni-00a42412-9f1f-6713-07bd-0872bbdf6de5": failed to connect to localhost:3000 inside namespace "ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df", IPv4: dial tcp4 127.0.0.1:3000: connect: connection refused IPv6 dial tcp6 [::1]:3000: connect: connection refused

I can see my service running,

kubectl get svc -n monitoring prometheus-grafana
NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
prometheus-grafana   ClusterIP   10.43.34.93   <none>        80/TCP    45h

And when I run this command

kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

I see

Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000
Handling connection for 3000
E0921 19:37:03.012976 3085961 portforward.go:400] an error occurred forwarding 3000 -> 3000: error forwarding port 3000 to pod ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df, uid : failed to execute portforward in network namespace "/var/run/netns/cni-00a42412-9f1f-6713-07bd-0872bbdf6de5": failed to connect to localhost:3000 inside namespace "ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df", IPv4: dial tcp4 127.0.0.1:3000: connect: connection refused IPv6 dial tcp6 [::1]:3000: connect: connection refused
Evan Carroll
  • 2,921
  • 6
  • 37
  • 85

1 Answers1

1

I was getting this because I had a pod in CrashLoopBackOff. In my specific case because yaml file had changed (through the Grafana interface) and Grafana could no longer start.

Error: ✗ Datasource provisioning error: datasource.yaml config is invalid. Only one datasource per organization can be marked as default

So I couldn't port forward :3000 because nothing was running on it.

Evan Carroll
  • 2,921
  • 6
  • 37
  • 85