How to Fix Kubernetes Node NotReady

“This guide is part of the Production Kubernetes Debugging Handbook — a complete reference for debugging production Kubernetes clusters.”

What Does Node NotReady Mean?

When a Kubernetes node enters NotReady state, it means the control plane can no longer confirm the node is healthy and capable of running workloads. Every pod scheduled on that node is at risk — some will be evicted, others will be rescheduled onto remaining nodes, and if those nodes are already under pressure, you can quickly turn a single node failure into a cluster-wide incident.

bash

kubectl get nodes
# NAME                   STATUS     ROLES   AGE   VERSION
# aks-nodepool-001       NotReady   agent   30d   v1.28.0
# aks-nodepool-002       Ready      agent   30d   v1.28.0
# aks-nodepool-003       Ready      agent   30d   v1.28.0

Unlike a pod failure which affects one workload, a node failure affects everything running on it simultaneously. This is why node issues need a different, faster triage approach.

This guide walks through every major cause of Node NotReady — kubelet failure, memory pressure, disk pressure, network issues — with step-by-step diagnosis and fixes for each.

How Kubernetes Detects a NotReady Node

Understanding the detection mechanism helps you interpret what you see and how fast to expect recovery.

The kubelet on each node sends a heartbeat to the API server every 10 seconds (default nodeStatusUpdateFrequency). The node controller on the control plane monitors these heartbeats. If no heartbeat is received for 40 seconds (default node-monitor-grace-period), the node transitions from Ready to Unknown. After 5 minutes without recovery (default pod-eviction-timeout), pods on that node begin to be evicted.

This timeline matters during incidents:

T+0s    Node stops sending heartbeats
T+40s   Node status changes to Unknown / NotReady
T+5min  Pod evictions begin
T+5min  Pods rescheduled onto healthy nodes

If you see a node flip to Unknown rather than NotReady, it means the control plane has completely lost contact with the node — network failure, VM crash, or kubelet process died. NotReady with a specific condition (MemoryPressure, DiskPressure) means the kubelet is still running and reporting, just reporting a problem.

Step 1 — The 5-Minute Node Triage

Run these four commands in order before touching anything.

bash

# 1. Confirm which nodes are affected and their status
kubectl get nodes

# 2. Check node conditions in detail
kubectl describe node <node-name>

# 3. Check system pods on the affected node
kubectl get pods -n kube-system -o wide | grep <node-name>

# 4. Check resource usage
kubectl top node <node-name>

In the kubectl describe node output, go straight to the Conditions section:

Conditions:
  Type                 Status  Reason                Message
  ----                 ------  ------                -------
  MemoryPressure       False   KubeletHasSufficientMemory
  DiskPressure         False   KubeletHasSufficientDisk
  PIDPressure          False   KubeletHasSufficientPID
  Ready                False   KubeletNotReady        PLEG is not healthy

Each condition tells a specific story. The table below maps conditions to root causes.

Condition	Status	Meaning	Jump to
Ready	False	Kubelet reporting a problem	Kubelet Failure section
Ready	Unknown	Control plane lost contact with node	Network / VM crash
MemoryPressure	True	Node running out of memory	Memory Pressure section
DiskPressure	True	Node running out of disk	Disk Pressure section
PIDPressure	True	Too many processes on node	PID Pressure section
NetworkUnavailable	True	CNI not configured correctly	Network Unavailable section

Cause 1 — Kubelet Failure

The kubelet is the agent that runs on every node and reports its health to the control plane. If the kubelet process stops, crashes, or cannot communicate with the container runtime, the node will enter NotReady.

This is the most common cause of NotReady in production.

How to Diagnose

You need SSH access to the affected node for this. In AKS you can use kubectl debug node:

bash

# Open a privileged debug session on the node without SSH
kubectl debug node/<node-name> -it --image=ubuntu

# Or SSH directly if you have access
ssh <node-ip>

Once on the node:

bash

# Check kubelet service status
systemctl status kubelet

# Read kubelet logs — last 100 lines
journalctl -u kubelet -n 100 --no-pager

# Check if the container runtime is running
systemctl status containerd
# or
systemctl status docker

Common Kubelet Log Errors and What They Mean

PLEG is not healthy

PLEG is not healthy: pleg was last seen active 3m45s ago; threshold is 3m0s

PLEG stands for Pod Lifecycle Event Generator. It is the kubelet component that watches the container runtime for pod state changes. When PLEG is unhealthy, it usually means the container runtime (containerd or Docker) is unresponsive or overloaded.

bash

# Check container runtime
systemctl status containerd
journalctl -u containerd -n 50 --no-pager

# Restart container runtime if it has crashed
systemctl restart containerd

# Then restart kubelet
systemctl restart kubelet

Failed to load kubeconfig

Failed to load kubeconfig file: stat /etc/kubernetes/kubelet.conf: no such file or directory

The kubelet configuration file is missing or was deleted. In managed clusters (AKS, EKS), this usually means the node needs to be reimaged or replaced.

Certificate errors

x509: certificate has expired or is not yet valid

Kubelet client certificates have expired. In self-managed clusters, certificates need manual rotation. In managed clusters, this is handled by the cloud provider — but if you recently performed a certificate rotation, nodes may need to be drained and rejoined.

bash

# Check certificate expiry on the node
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

Fix

bash

# Most cases: restart kubelet
systemctl restart kubelet

# Watch the node recover
kubectl get nodes -w

# If kubelet fails to start, check config files
cat /etc/kubernetes/kubelet.conf
cat /var/lib/kubelet/config.yaml

# Check kubelet systemd unit for the correct flags
systemctl cat kubelet

Cause 2 — Memory Pressure

When a node is running low on available memory, Kubernetes sets MemoryPressure: True. The node stops receiving new pod scheduling and may begin evicting existing pods to reclaim memory.

How to Diagnose

bash

kubectl describe node <node-name> | grep -A3 MemoryPressure
# MemoryPressure   True   KubeletHasInsufficientMemory

# Check actual memory usage
kubectl top node <node-name>

# SSH into the node for detailed view
free -h
cat /proc/meminfo | grep -E "MemTotal|MemFree|MemAvailable"

Find what is consuming memory on the node:

bash

# Top memory-consuming pods on this node
kubectl top pods --all-namespaces \
  --field-selector spec.nodeName=<node-name> \
  --sort-by=memory

# Find pods without memory limits (the most dangerous ones)
kubectl get pods --all-namespaces -o json | \
  jq '.items[] | select(.spec.nodeName=="<node-name>") |
  select(.spec.containers[].resources.limits.memory == null) |
  .metadata.name'

Common Causes

Pod without memory limits consuming all available memory. A single pod without a memory limit can grow until the node is exhausted. The Linux kernel will then begin OOMKilling processes — sometimes killing kubelet or system processes rather than the offending pod.

Memory leak in a long-running application. The pod appears healthy but its memory usage grows slowly over hours or days until the node is pressured.

Too many pods on a single node. Even if individual pods have limits, the combined usage can exceed available node memory.

Fix

bash

# Immediately: cordon the node to stop new pod scheduling
kubectl cordon <node-name>

# Find and evict the memory hog
kubectl top pods --all-namespaces --sort-by=memory | head -10

# Delete or restart the offending pod
kubectl delete pod <pod-name> -n <namespace>

# Add memory limits to all pods without them
# Check your deployments for missing limits
kubectl get deployment <name> -o yaml | grep -A10 resources

# After resolving the cause, uncordon the node
kubectl uncordon <node-name>

Long-term fix: enforce memory limits through LimitRange at the namespace level so no pod can be created without limits:

yaml

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: production
spec:
  limits:
  - type: Container
    defaultRequest:
      memory: "128Mi"
      cpu: "100m"
    default:
      memory: "256Mi"
      cpu: "500m"
    max:
      memory: "2Gi"

Cause 3 — Disk Pressure

When a node runs low on disk space, Kubernetes sets DiskPressure: True. Kubelet evicts pods and stops accepting new ones. If disk fills completely, kubelet itself can fail.

How to Diagnose

bash

kubectl describe node <node-name> | grep -A3 DiskPressure
# DiskPressure   True   KubeletHasDiskPressure

# SSH into the node
df -h

# Find the largest directories
du -sh /var/lib/containerd/*    # containerd image storage
du -sh /var/lib/docker/*        # docker image storage
du -sh /var/log/*               # system logs
du -sh /tmp/*                   # temp files

Common Causes in Production

Accumulated container images. Every time a new image is pulled for a deployment, the old image stays on the node unless garbage collected. Over weeks, hundreds of old image layers accumulate. This is the most common cause of disk pressure in long-running clusters.

Container logs without rotation. If log rotation is not configured, container logs in /var/log/pods grow indefinitely.

Core dumps. A crashing process may write large core dump files to disk repeatedly.

Large emptyDir volumes. Pods using emptyDir volumes write to the node’s disk. A pod writing large temporary files without cleanup fills node disk directly.

Fix

bash

# Immediate: clean up unused container images
crictl rmi --prune

# Or for Docker-based nodes
docker image prune -af

# Clean up stopped containers
crictl rm $(crictl ps -a -q)

# Clean up unused volumes
docker volume prune -f   # Docker only

# Check log sizes
du -sh /var/log/pods/*/* | sort -rh | head -20

# Truncate a specific large log (emergency only)
truncate -s 0 /var/log/pods/<pod-uid>/<container>/0.log

Long-term fix: configure kubelet image garbage collection:

yaml

# /var/lib/kubelet/config.yaml
imageGCHighThresholdPercent: 75   # start GC when disk hits 75%
imageGCLowThresholdPercent: 60    # GC until disk is at 60%

Default thresholds are 85% high and 80% low — too conservative for nodes that run batch jobs with large images. Lower them.

Cause 4 — Network Unavailable

NetworkUnavailable: True means the CNI (Container Network Interface) plugin has not configured networking on the node correctly. New pods cannot get IP addresses.

How to Diagnose

bash

kubectl describe node <node-name> | grep NetworkUnavailable
# NetworkUnavailable   True   CniPluginNotReady

# Check CNI plugin pods on the affected node
kubectl get pods -n kube-system -o wide | grep <node-name> | grep -E "calico|flannel|cilium|azure"

# Get CNI plugin logs
kubectl logs -n kube-system <cni-pod-name>

# On the node: check CNI config directory
ls /etc/cni/net.d/
cat /etc/cni/net.d/<config-file>

Common Causes

CNI plugin pod not running on the node. DaemonSet pods occasionally fail to start on specific nodes after a node restart or upgrade.

CNI config file missing or corrupted. The /etc/cni/net.d/ directory may be empty or contain an invalid config after a CNI upgrade.

IP address pool exhausted. In Calico or Azure CNI, the IP pool for the node or subnet can be exhausted. New pods cannot get IPs even though the CNI plugin is running.

Fix

bash

# Restart the CNI plugin DaemonSet pod on the affected node
kubectl delete pod -n kube-system <cni-pod-on-affected-node>

# If CNI config is missing, re-apply the CNI manifest
# For Calico:
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

# For Azure CNI (AKS) — check IPAM pool
az aks show --resource-group <rg> --name <cluster> \
  --query "networkProfile.podCidr"

# Check available IPs per node in AKS
kubectl get node <node-name> -o jsonpath='{.status.capacity.pods}'

Cause 5 — Node Completely Unreachable (Ready: Unknown)

When the node status is Unknown rather than False, the control plane has not received a heartbeat in over 40 seconds. The node is likely unreachable — VM crash, network partition, or underlying infrastructure failure.

How to Diagnose

bash

kubectl get nodes
# NAME               STATUS    ROLES   AGE
# aks-nodepool-001   Unknown   agent   30d

# Check when the node was last seen
kubectl describe node <node-name> | grep "Last Heartbeat"

# In cloud environments — check the VM status directly
# AKS:
az vm show --resource-group <node-rg> --name <vm-name> --query "provisioningState"

# Check if the node is reachable via ping (from another node or bastion)
ping <node-internal-ip>

Fix Options

bash

# Option 1: Wait — in cloud environments the node may self-recover
# AKS will attempt to restart the VM automatically within a few minutes

# Option 2: Drain and delete the node to force rescheduling
kubectl drain <node-name> --ignore-daemonsets \
  --delete-emptydir-data --force

kubectl delete node <node-name>

# The cloud provider will provision a replacement node
# In AKS, the node pool autoscaler creates a replacement automatically

# Option 3: Manually trigger node replacement in AKS
az aks nodepool delete --resource-group <rg> \
  --cluster-name <cluster> --name <nodepool> \
  --no-wait

Cordon vs Drain vs Delete — When to Use Each

This is a decision you will make under pressure. Know it before the incident.

Action	Command	Effect	When to use
Cordon	`kubectl cordon <node>`	Stops new pod scheduling. Existing pods keep running.	Investigating an issue, not ready to move workloads yet
Drain	`kubectl drain <node> --ignore-daemonsets --delete-emptydir-data`	Evicts all pods safely, then cordons.	Planned maintenance or replacing the node
Delete	`kubectl delete node <node>`	Removes node from cluster. Cloud provider may replace it.	Node is unrecoverable
Uncordon	`kubectl uncordon <node>`	Returns node to schedulable state.	After fix is confirmed

Warning: kubectl drain evicts all pods including stateful ones. Always verify your StatefulSets have enough replicas on other nodes before draining, or you will cause downtime.

Real Production Example — Two Nodes Hit DiskPressure Simultaneously

The alert: Monday morning. Two out of five nodes enter NotReady. Pod scheduling is failing across 12 namespaces.

bash

kubectl describe node aks-nodepool-001 | grep DiskPressure
# DiskPressure: True

kubectl describe node aks-nodepool-002 | grep DiskPressure
# DiskPressure: True

# SSH into node 1
df -h
# /dev/sda1   99%   /

du -sh /var/lib/containerd/*
# 47G   /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs

crictl images | wc -l
# 312 images

A nightly batch job had been pulling a new 8GB ML model image on each run without cleaning up. Over three weeks, 312 images accumulated. Both nodes were provisioned at the same time and hit the threshold simultaneously.

bash

# Fix: clean images on both nodes
crictl rmi --prune
# Freed 43GB on each node

kubectl uncordon aks-nodepool-001
kubectl uncordon aks-nodepool-002

# Long-term: lower GC thresholds in kubelet config
# imageGCHighThresholdPercent: 75
# imageGCLowThresholdPercent: 60

Time to resolution: 31 minutes.

Quick Reference

bash

# Check all node conditions
kubectl describe node <node-name> | grep -A10 Conditions

# Check system pods on node
kubectl get pods -n kube-system -o wide | grep <node-name>

# Check node resource usage
kubectl top node <node-name>

# SSH via kubectl debug (no SSH required)
kubectl debug node/<node-name> -it --image=ubuntu

# Kubelet status on node
systemctl status kubelet
journalctl -u kubelet -n 100 --no-pager

# Clean up images on node
crictl rmi --prune

# Cordon node
kubectl cordon <node-name>

# Drain node safely
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# Uncordon after fix
kubectl uncordon <node-name>

Summary

Node NotReady issues follow a predictable pattern. Check conditions first, then go one level deeper into the specific cause:

Ready: Unknown — node is unreachable, check VM status in cloud console
Ready: False with PLEG — container runtime failure, restart containerd then kubelet
MemoryPressure — find the pod without memory limits and evict it
DiskPressure — run crictl rmi --prune, configure GC thresholds permanently
NetworkUnavailable — restart the CNI DaemonSet pod on the affected node

Always cordon before investigating so no new pods land on the troubled node while you are working on it.

Related guides:

Post Views: 4

How to Fix Kubernetes Node NotReady

What Does Node NotReady Mean?

How Kubernetes Detects a NotReady Node

Step 1 — The 5-Minute Node Triage

Cause 1 — Kubelet Failure

How to Diagnose

Common Kubelet Log Errors and What They Mean

Fix

Cause 2 — Memory Pressure

How to Diagnose

Common Causes

Fix

Cause 3 — Disk Pressure

How to Diagnose

Common Causes in Production

Fix

Cause 4 — Network Unavailable

How to Diagnose

Common Causes

Fix

Cause 5 — Node Completely Unreachable (Ready: Unknown)

How to Diagnose

Fix Options

Cordon vs Drain vs Delete — When to Use Each

Real Production Example — Two Nodes Hit DiskPressure Simultaneously

Quick Reference

Summary

About The Author

Shamsher Khan

Leave a Comment Cancel Reply

What Does Node NotReady Mean?

How Kubernetes Detects a NotReady Node

Step 1 — The 5-Minute Node Triage

Cause 1 — Kubelet Failure

How to Diagnose

Common Kubelet Log Errors and What They Mean

Fix

Cause 2 — Memory Pressure

How to Diagnose

Common Causes

Fix

Cause 3 — Disk Pressure

How to Diagnose

Common Causes in Production

Fix

Cause 4 — Network Unavailable

How to Diagnose

Common Causes

Fix

Cause 5 — Node Completely Unreachable (Ready: Unknown)

How to Diagnose

Fix Options

Cordon vs Drain vs Delete — When to Use Each

Real Production Example — Two Nodes Hit DiskPressure Simultaneously

Quick Reference

Summary

About The Author

Shamsher Khan

Related Posts

Leave a Comment Cancel Reply