How can I release previously allocated resources of a deleted pod?

Question

I already had 3 Cassandra node/pods running. I deleted them and tried to re create them again using the same YAML file as following, on the same Kind cluster, but it stuck pending status:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra
        image: gcr.io/google-samples/cassandra:v13
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.default.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8Demo"
          - name: CASSANDRA_DC
            value: "DC1-K8Demo"
          - name: CASSANDRA_RACK
            value: "Rack1-K8Demo"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - /ready-probe.sh
          initialDelaySeconds: 15
          timeoutSeconds: 5
        # These volume mounts are persistent. They are like inline claims,
        # but not exactly because the names need to match exactly one of
        # the stateful pod volumes.
        volumeMounts:
        - name: cassandra-data
          mountPath: /cassandra_data
  # These are converted to volume claims by the controller
  # and mounted at the paths mentioned above.
  # do not use these in production until ssd GCEPersistentDisk or other ssd pd
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: fast
      resources:
        requests:
          storage: 1Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast
provisioner: k8s.io/minikube-hostpath
parameters:
  type: pd-ssd
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  clusterIP: None
  ports:
  - port: 9042
  selector:
    app: cassandra

As I searched through the internet I believe that the problem is because of lack of resources, but I guess this is happening because previously allocated resources to deleted nodes/pods are still occupied. But I don't know how can I release them?

I tried kubectl top nodes:

NAME                 CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
kind-control-plane   205m         2%     1046Mi          6%        
kind-worker          171m         2%     2612Mi          16%

It seems everything is fine?

Maybe the problem is with hard disc allocation that I don't know how to check?

Dharani Dhar Golladasari · Answer 1 · 2023-01-17T17:09:56.650

If a pod is in a pending state then typically, this is a result of a lack of resources. First check the events of the pod to know the reason why pod is in pending state. For this use following command

Kubectl describe pod-name

This events will give the insight why the pod is in a pending state. A common reason for a pod going into a pending state is lack of memory or storage. you may have exhausted the resources available in nodes. One way to get the exhausted resources back is to clean up the nodes by deleting the unwanted pods and deployments.

This official document contains information to debug the pods in kubernetes.

This document helps to debug the statefulsets in k8s.

If you need a example to deploy cassandra in k8s, this offical k8s document will help.

How can I release previously allocated resources of a deleted pod?

1 Answers1