I've been trying to set up Kubernetes in Amazon Linux 2023 with an ARM64 architecture (t4g.medium). The idea is to set up Kubernetes inside a single EC2 instance for POC work.
Since dnf is the preferred package manager for AL2023, I'm using that.
1. Installed docker
[ec2-user@ip-172-31-22-247 ~]$ sudo dnf install -y docker
... Installation logs ...
[ec2-user@ip-172-31-22-247 ~]$ sudo systemctl start docker
[ec2-user@ip-172-31-22-247 ~]$ sudo systemctl enable docker
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
[ec2-user@ip-172-31-22-247 ~]$ sudo usermod -aG docker $USER
[ec2-user@ip-172-31-22-247 ~]$ newgrp docker
2. Tried installing Kubectl, Kublet, and Kubeadm by pulling repositories from Yum
[ec2-user@ip-172-31-22-247 ~]$ mkdir setup
[ec2-user@ip-172-31-22-247 ~]$ cd setup/
[ec2-user@ip-172-31-22-247 setup]$ cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
> [kubernetes]
> name=Kubernetes
> baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
> enabled=1
> gpgcheck=1
> repo_gpgcheck=1
> gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
> EOF
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
[ec2-user@ip-172-31-22-247 setup]$ sudo dnf install -y kubectl kubelet kubeadm
Kubernetes 8.8 kB/s | 1.4 kB 00:00
Errors during downloading metadata for repository 'kubernetes':
- Status code: 404 for https://packages.cloud.google.com/yum/repos/kubernetes-el7-aarch64/repodata/repomd.xml (IP: 142.250.182.206)
Error: Failed to download metadata for repo 'kubernetes': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.
Ignoring repositories: kubernetes
Last metadata expiration check: 0:20:04 ago on Wed Aug 21 06:13:49 2024.
No match for argument: kubectl
No match for argument: kubelet
No match for argument: kubeadm
Error: Unable to find a match: kubectl kubelet kubeadm
That didn't work because Yum does not have the ARM version of the package (kuberentes-e17-aarch64).
3. Manually installed Kubenetes (kubectl, kubelet, kubeadm)
[ec2-user@ip-172-31-22-247 setup]$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl"
... Installation logs ...
[ec2-user@ip-172-31-22-247 setup]$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubelet"
... Installation logs ...
[ec2-user@ip-172-31-22-247 setup]$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubeadm"
... Installation logs ...
[ec2-user@ip-172-31-22-247 setup]$ chmod +x kubectl kubelet kubeadm
[ec2-user@ip-172-31-22-247 setup]$ ls
kubeadm kubectl kubelet
[ec2-user@ip-172-31-22-247 setup]$ sudo mv kubectl kubelet kubeadm /usr/local/bin/
[ec2-user@ip-172-31-22-247 setup]$ kubectl version --client
ubeadm version
Client Version: v1.31.0
Kustomize Version: v5.4.2
[ec2-user@ip-172-31-22-247 setup]$ kubelet --version
Kubernetes v1.31.0
[ec2-user@ip-172-31-22-247 setup]$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"31", GitVersion:"v1.31.0", GitCommit:"9edcffcde5595e8a5b1a35f88c421764e575afce", GitTreeState:"clean", BuildDate:"2024-08-13T07:35:57Z", GoVersion:"go1.22.5", Compiler:"gc", Platform:"linux/arm64"}
Then I tried to - 4. Add and enable kubelet as service
[ec2-user@ip-172-31-22-247 setup]$ sudo tee /etc/systemd/system/kubelet.service <<EOF
> [Unit]
> Description=kubelet: The Kubernetes Node Agent
> Documentation=https://kubernetes.io/docs/
> After=network-online.target
> Wants=network-online.target
>
> [Service]
> ExecStart=/usr/local/bin/kubelet
> Restart=always
> StartLimitInterval=0
> RestartSec=10
>
> [Install]
> WantedBy=multi-user.target
> EOF
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/local/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
[ec2-user@ip-172-31-22-247 setup]$ sudo systemctl daemon-reload
[ec2-user@ip-172-31-22-247 setup]$ sudo systemctl enable kubelet
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /etc/systemd/system/kubelet.service.
[ec2-user@ip-172-31-22-247 setup]$ sudo systemctl start kubelet
[ec2-user@ip-172-31-22-247 setup]$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: disabled)
Active: active (running) since Wed 2024-08-21 06:56:04 UTC; 3s ago
Docs: https://kubernetes.io/docs/
Main PID: 28070 (kubelet)
Tasks: 11 (limit: 4507)
Memory: 16.8M
CPU: 1.087s
CGroup: /system.slice/kubelet.service
└─28070 /usr/local/bin/kubelet
Aug 21 06:56:05 ip-172-31-22-247.ap-south-1.compute.internal kubelet[28070]: I0821 06:56:05.851119 28070 cpu_manager.go:215] "Reconciling" reconcilePeriod="10s"
Aug 21 06:56:05 ip-172-31-22-247.ap-south-1.compute.internal kubelet[28070]: I0821 06:56:05.851180 28070 state_mem.go:36] "Initialized new in-memory state store"
Aug 21 06:56:05 ip-172-31-22-247.ap-south-1.compute.internal kubelet[28070]: I0821 06:56:05.852450 28070 policy_none.go:49] "None policy: Start"
Aug 21 06:56:05 ip-172-31-22-247.ap-south-1.compute.internal kubelet[28070]: I0821 06:56:05.853351 28070 memory_manager.go:170] "Starting memorymanager" policy="None"
Aug 21 06:56:05 ip-172-31-22-247.ap-south-1.compute.internal kubelet[28070]: I0821 06:56:05.853467 28070 state_mem.go:35] "Initializing new in-memory state store"
Aug 21 06:56:05 ip-172-31-22-247.ap-south-1.compute.internal kubelet[28070]: I0821 06:56:05.917341 28070 manager.go:510] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
Aug 21 06:56:05 ip-172-31-22-247.ap-south-1.compute.internal kubelet[28070]: I0821 06:56:05.917953 28070 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 21 06:56:05 ip-172-31-22-247.ap-south-1.compute.internal kubelet[28070]: I0821 06:56:05.917984 28070 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 21 06:56:05 ip-172-31-22-247.ap-south-1.compute.internal kubelet[28070]: I0821 06:56:05.918126 28070 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 21 06:56:05 ip-172-31-22-247.ap-south-1.compute.internal kubelet[28070]: I0821 06:56:05.919167 28070 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
5. Initialize Kubernetes cluster
[ec2-user@ip-172-31-22-247 setup]$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.31.0
[preflight] Running pre-flight checks
[WARNING FileExisting-crictl]: crictl not found in system path
[WARNING FileExisting-socat]: socat not found in system path
[WARNING FileExisting-tc]: tc not found in system path
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileExisting-conntrack]: conntrack not found in system path
[ERROR Port-10250]: Port 10250 is in use
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
That didn't work as some of the packages required for this command were not installed-
6. Install conntrack,socat,iproute-tc
[ec2-user@ip-172-31-22-247 setup]$ VERSION="v1.28.0" # Replace with the desired version
[ec2-user@ip-172-31-22-247 setup]$ curl -LO "https://github.com/kubernetes-sigs/cri-tools/releases/download/${VERSION}/crictl-${VERSION}-linux-arm64.tar.gz"
... Installation logs ...
[ec2-user@ip-172-31-22-247 setup]$ sudo tar -C /usr/local/bin -xzf "crictl-${VERSION}-linux-arm64.tar.gz"
[ec2-user@ip-172-31-22-247 setup]$ crictl --version
crictl version v1.28.0
[ec2-user@ip-172-31-22-247 setup]$ sudo dnf install -y conntrack socat iproute-tc
... Installation logs ...
Installed:
conntrack-tools-1.4.6-2.amzn2023.0.2.aarch64 iproute-tc-5.10.0-2.amzn2023.0.5.aarch64 libnetfilter_cthelper-1.0.0-21.amzn2023.0.2.aarch64 libnetfilter_cttimeout-1.0.0-19.amzn2023.0.2.aarch64 libnetfilter_queue-1.0.5-2.amzn2023.0.2.aarch64
socat-1.7.4.2-1.amzn2023.0.2.aarch64
Complete!
7. kubeadm init again, with --v=5, and with --ignore-preflight-errors=port-10250
[ec2-user@ip-172-31-22-247 setup]$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.31.0
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR Port-10250]: Port 10250 is in use
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
[ec2-user@ip-172-31-22-247 setup]$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --v=5 --ignore-preflight-errors=Port-10250
I0821 06:58:58.733231 28979 initconfiguration.go:123] detected and using CRI socket: unix:///var/run/containerd/containerd.sock
I0821 06:58:58.733493 28979 interface.go:432] Looking for default routes with IPv4 addresses
I0821 06:58:58.733510 28979 interface.go:437] Default route transits interface "ens5"
I0821 06:58:58.733623 28979 interface.go:209] Interface ens5 is up
I0821 06:58:58.733665 28979 interface.go:257] Interface "ens5" has 2 addresses :[172.31.22.247/20 fe80::4a4:85ff:fea1:b411/64].
I0821 06:58:58.733684 28979 interface.go:224] Checking addr 172.31.22.247/20.
I0821 06:58:58.733705 28979 interface.go:231] IP found 172.31.22.247
I0821 06:58:58.733722 28979 interface.go:263] Found valid IPv4 address 172.31.22.247 for interface "ens5".
I0821 06:58:58.733733 28979 interface.go:443] Found active IP 172.31.22.247
I0821 06:58:58.733752 28979 kubelet.go:195] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd"
I0821 06:58:58.733791 28979 version.go:192] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt
[init] Using Kubernetes version: v1.31.0
[preflight] Running pre-flight checks
I0821 06:58:59.065953 28979 checks.go:561] validating Kubernetes and kubeadm version
I0821 06:58:59.065995 28979 checks.go:166] validating if the firewall is enabled and active
I0821 06:58:59.080990 28979 checks.go:201] validating availability of port 6443
I0821 06:58:59.081192 28979 checks.go:201] validating availability of port 10259
I0821 06:58:59.081223 28979 checks.go:201] validating availability of port 10257
I0821 06:58:59.081246 28979 checks.go:278] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml
I0821 06:58:59.081263 28979 checks.go:278] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml
I0821 06:58:59.081274 28979 checks.go:278] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml
I0821 06:58:59.081282 28979 checks.go:278] validating the existence of file /etc/kubernetes/manifests/etcd.yaml
I0821 06:58:59.081292 28979 checks.go:428] validating if the connectivity type is via proxy or direct
I0821 06:58:59.081314 28979 checks.go:467] validating http connectivity to first IP address in the CIDR
I0821 06:58:59.081333 28979 checks.go:467] validating http connectivity to first IP address in the CIDR
I0821 06:58:59.081360 28979 checks.go:102] validating the container runtime
I0821 06:58:59.082191 28979 checks.go:637] validating whether swap is enabled or not
I0821 06:58:59.082257 28979 checks.go:368] validating the presence of executable crictl
I0821 06:58:59.082285 28979 checks.go:368] validating the presence of executable conntrack
I0821 06:58:59.082303 28979 checks.go:368] validating the presence of executable ip
I0821 06:58:59.082452 28979 checks.go:368] validating the presence of executable iptables
I0821 06:58:59.082478 28979 checks.go:368] validating the presence of executable mount
I0821 06:58:59.082501 28979 checks.go:368] validating the presence of executable nsenter
I0821 06:58:59.082572 28979 checks.go:368] validating the presence of executable ebtables
I0821 06:58:59.082602 28979 checks.go:368] validating the presence of executable ethtool
I0821 06:58:59.082618 28979 checks.go:368] validating the presence of executable socat
I0821 06:58:59.082634 28979 checks.go:368] validating the presence of executable tc
I0821 06:58:59.082648 28979 checks.go:368] validating the presence of executable touch
I0821 06:58:59.082665 28979 checks.go:514] running all checks
I0821 06:58:59.089380 28979 checks.go:399] checking whether the given node name is valid and reachable using net.LookupHost
I0821 06:58:59.089877 28979 checks.go:603] validating kubelet version
I0821 06:58:59.141251 28979 checks.go:128] validating if the "kubelet" service is enabled and active
I0821 06:58:59.157598 28979 checks.go:201] validating availability of port 10250
[WARNING Port-10250]: Port 10250 is in use
I0821 06:58:59.157733 28979 checks.go:327] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0821 06:58:59.157790 28979 checks.go:201] validating availability of port 2379
I0821 06:58:59.157835 28979 checks.go:201] validating availability of port 2380
I0821 06:58:59.157863 28979 checks.go:241] validating the existence and emptiness of directory /var/lib/etcd
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
I0821 06:58:59.160079 28979 checks.go:832] using image pull policy: IfNotPresent
W0821 06:58:59.161088 28979 checks.go:846] detected that the sandbox image "registry.k8s.io/pause:3.8" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image.
I0821 06:58:59.161808 28979 checks.go:871] pulling: registry.k8s.io/kube-apiserver:v1.31.0
I0821 06:59:02.605498 28979 checks.go:871] pulling: registry.k8s.io/kube-controller-manager:v1.31.0
I0821 06:59:05.869342 28979 checks.go:871] pulling: registry.k8s.io/kube-scheduler:v1.31.0
I0821 06:59:08.644167 28979 checks.go:871] pulling: registry.k8s.io/kube-proxy:v1.31.0
I0821 06:59:11.654486 28979 checks.go:871] pulling: registry.k8s.io/coredns/coredns:v1.11.1
I0821 06:59:14.616718 28979 checks.go:871] pulling: registry.k8s.io/pause:3.10
I0821 06:59:16.324448 28979 checks.go:871] pulling: registry.k8s.io/etcd:3.5.15-0
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0821 06:59:21.674284 28979 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0821 06:59:21.918553 28979 certs.go:473] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [ip-172-31-22-247.ap-south-1.compute.internal kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.31.22.247]
[certs] Generating "apiserver-kubelet-client" certificate and key
I0821 06:59:22.520101 28979 certs.go:112] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I0821 06:59:23.451630 28979 certs.go:473] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I0821 06:59:24.201139 28979 certs.go:112] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I0821 06:59:24.631585 28979 certs.go:473] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [ip-172-31-22-247.ap-south-1.compute.internal localhost] and IPs [172.31.22.247 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [ip-172-31-22-247.ap-south-1.compute.internal localhost] and IPs [172.31.22.247 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I0821 06:59:26.743735 28979 certs.go:78] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0821 06:59:26.971434 28979 kubeconfig.go:111] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I0821 06:59:27.475109 28979 kubeconfig.go:111] creating kubeconfig file for super-admin.conf
[kubeconfig] Writing "super-admin.conf" kubeconfig file
I0821 06:59:27.648101 28979 kubeconfig.go:111] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I0821 06:59:27.873964 28979 kubeconfig.go:111] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0821 06:59:28.836644 28979 kubeconfig.go:111] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0821 06:59:29.158606 28979 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I0821 06:59:29.158647 28979 manifests.go:103] [control-plane] getting StaticPodSpecs
I0821 06:59:29.158837 28979 certs.go:473] validating certificate period for CA certificate
I0821 06:59:29.158904 28979 manifests.go:129] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I0821 06:59:29.158919 28979 manifests.go:129] [control-plane] adding volume "etc-pki-ca-trust" for component "kube-apiserver"
I0821 06:59:29.158925 28979 manifests.go:129] [control-plane] adding volume "etc-pki-tls-certs" for component "kube-apiserver"
I0821 06:59:29.158937 28979 manifests.go:129] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I0821 06:59:29.159775 28979 manifests.go:158] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I0821 06:59:29.159799 28979 manifests.go:103] [control-plane] getting StaticPodSpecs
I0821 06:59:29.159981 28979 manifests.go:129] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I0821 06:59:29.159997 28979 manifests.go:129] [control-plane] adding volume "etc-pki-ca-trust" for component "kube-controller-manager"
I0821 06:59:29.160003 28979 manifests.go:129] [control-plane] adding volume "etc-pki-tls-certs" for component "kube-controller-manager"
I0821 06:59:29.160008 28979 manifests.go:129] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I0821 06:59:29.160012 28979 manifests.go:129] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I0821 06:59:29.160016 28979 manifests.go:129] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I0821 06:59:29.160749 28979 manifests.go:158] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I0821 06:59:29.160775 28979 manifests.go:103] [control-plane] getting StaticPodSpecs
I0821 06:59:29.160951 28979 manifests.go:129] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I0821 06:59:29.161420 28979 manifests.go:158] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
I0821 06:59:29.161442 28979 kubelet.go:68] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 502.138004ms
[api-check] Waiting for a healthy API server. This can take up to 4m0s
[api-check] The API server is not healthy after 4m0.000126863s
Unfortunately, an error has occurred:
context deadline exceeded
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs CONTAINERID'
could not initialize a Kubernetes cluster
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase.func1
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:112
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:132
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).Run.func1
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:259
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).visitAll
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:446
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).Run
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:128
github.com/spf13/cobra.(Command).execute
github.com/spf13/cobra@v1.8.1/command.go:985
github.com/spf13/cobra.(Command).ExecuteC
github.com/spf13/cobra@v1.8.1/command.go:1117
github.com/spf13/cobra.(Command).Execute
github.com/spf13/cobra@v1.8.1/command.go:1041
k8s.io/kubernetes/cmd/kubeadm/app.Run
k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:47
main.main
k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
runtime/proc.go:271
runtime.goexit
runtime/asm_arm64.s:1222
error execution phase wait-control-plane
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).Run.func1
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:260
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).visitAll
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:446
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).Run
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:128
github.com/spf13/cobra.(Command).execute
github.com/spf13/cobra@v1.8.1/command.go:985
github.com/spf13/cobra.(Command).ExecuteC
github.com/spf13/cobra@v1.8.1/command.go:1117
github.com/spf13/cobra.(Command).Execute
github.com/spf13/cobra@v1.8.1/command.go:1041
k8s.io/kubernetes/cmd/kubeadm/app.Run
k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:47
main.main
k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
runtime/proc.go:271
runtime.goexit
runtime/asm_arm64.s:1222
8. When I checked status of kubelet, it is both active (running), and enabled
[ec2-user@ip-172-31-22-247 setup]$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: disabled)
Active: active (running) since Wed 2024-08-21 06:59:29 UTC; 5min ago
Docs: https://kubernetes.io/docs/
Main PID: 29264 (kubelet)
Tasks: 10 (limit: 4507)
Memory: 20.6M
CPU: 2.202s
CGroup: /system.slice/kubelet.service
└─29264 /usr/local/bin/kubelet
Aug 21 06:59:29 ip-172-31-22-247.ap-south-1.compute.internal kubelet[29264]: I0821 06:59:29.804878 29264 policy_none.go:49] "None policy: Start"
Aug 21 06:59:29 ip-172-31-22-247.ap-south-1.compute.internal kubelet[29264]: I0821 06:59:29.805593 29264 memory_manager.go:170] "Starting memorymanager" policy="None"
Aug 21 06:59:29 ip-172-31-22-247.ap-south-1.compute.internal kubelet[29264]: I0821 06:59:29.805622 29264 state_mem.go:35] "Initializing new in-memory state store"
Aug 21 06:59:29 ip-172-31-22-247.ap-south-1.compute.internal kubelet[29264]: I0821 06:59:29.805809 29264 state_mem.go:75] "Updated machine memory state"
Aug 21 06:59:29 ip-172-31-22-247.ap-south-1.compute.internal kubelet[29264]: I0821 06:59:29.806489 29264 manager.go:510] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
Aug 21 06:59:29 ip-172-31-22-247.ap-south-1.compute.internal kubelet[29264]: I0821 06:59:29.806928 29264 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 21 06:59:29 ip-172-31-22-247.ap-south-1.compute.internal kubelet[29264]: I0821 06:59:29.806953 29264 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 21 06:59:29 ip-172-31-22-247.ap-south-1.compute.internal kubelet[29264]: I0821 06:59:29.807063 29264 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 21 06:59:29 ip-172-31-22-247.ap-south-1.compute.internal kubelet[29264]: I0821 06:59:29.868811 29264 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 21 07:00:19 ip-172-31-22-247.ap-south-1.compute.internal kubelet[29264]: W0821 07:00:19.664597 29264 watcher.go:93] Error while processing event ("/sys/fs/cgroup/system.slice/sysstat-collect.service": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch />
10. crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a
[ec2-user@ip-172-31-22-247 setup]$ sudo crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
11. cat /proc/cmdline
[ec2-user@ip-172-31-22-247 setup]$ cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt1)/boot/vmlinuz-6.1.102-111.182.amzn2023.aarch64 root=UUID=80c901d5-82c8-4093-a9f3-b467c777b080 ro console=tty0 console=ttyS0,115200n8 nvme_core.io_timeout=4294967295 rd.emergency=poweroff rd.shell=0 selinux=1 security=selinux quiet numa_cma=1:64M
It is said that Kubernetes is ready and available for ARM, but getting it installed in AL2023 with ARM64 looks pretty complex.
I couldn't find any online documents, forums, or SO posts related to the same.
I've tried multiple ways, tried with GPTs, inspected logs... but nothing worked. Please take a look into the above given logs and let me know if there's any way to fix this.
