I went through the setup in the K8s documentation and chose flannel as the network plugin. I came up with the steps below on Ubuntu VMs. All the pods come up and I can deploy new pods but I can't communicate with pods when they're on different nodes. For example, if I deploy nginx, I can't curl the pods on port 80. I know my yaml works because everything works as expected on EKS. I have the k8s nodes (1 master 2 workers) deployed in a single VPC in a single region and kubectl exec -it -- sh works on all the pods. Not sure where to look. There are no errors in kube-proxy or flannel pods. No errors in kubelet.
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
EOF
Apply sysctl params without reboot
sudo sysctl --system
#Install Container runtime
https://github.com/containerd/containerd/blob/main/docs/getting-started.md
wget https://github.com/containerd/containerd/releases/download/v1.7.21/containerd-1.7.21-linux-amd64.tar.gz
sudo tar Cxzvf /usr/local containerd-1.7.21-linux-amd64.tar.gz
#download containerd.service
wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
sudo mkdir -p /usr/local/lib/systemd/system
sudo mv containerd.service /usr/local/lib/systemd/system/containerd.service
sudo systemctl daemon-reload
sudo systemctl enable --now containerd
wget https://github.com/opencontainers/runc/releases/download/v1.1.14/runc.amd64
sudo install -m 755 runc.amd64 /usr/local/sbin/runc
wget https://github.com/containernetworking/plugins/releases/download/v1.5.1/cni-plugins-linux-amd64-v1.5.1.tgz
sudo mkdir -p /opt/cni/bin
sudo tar Cxzvf /opt/cni/bin
sudo mkdir -p /etc/containerd
sudo sh -c '/usr/local/bin/containerd config default > /etc/containerd/config.toml'
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
sudo mkdir -p -m 755 /etc/apt/keyringsvi
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl enable --now kubelet
