From 243 Pods to 5 Problems: A Kubernetes Operational Triage Investigation

243 pods. 28 namespaces. Thousands of metrics.

When the investigation started, only five things actually mattered.

This article documents what those five things were, why every monitoring dashboard in the cluster missed them, and what an operational triage layer looks like in practice. It is not a tool tutorial. It is a pattern argument with real findings.

1. The Cluster on Paper

At 11:42 AM on a Tuesday, the monitoring dashboard for the cluster I was investigating showed everything green. Pod count steady at 243. CPU utilization at 11%. Memory at 23%. No active alerts. No customer-facing incidents.

By every signal the platform team monitored, this cluster was healthy.

Here is what it looked like to that team:

Cluster:       Production-grade AKS, single region
Node Pools:    2 (system + user workloads)
Namespaces:    28
Pods:          243 (240 Running, 3 in various states)
Deployments:   71
Services:      88
PVCs:          47
Network:       Standard CNI, no service mesh

The Grafana dashboards monitoring this cluster tracked the usual signals:

• CPU and memory utilization (per node pool and per namespace)
• Pod count and deployment replica satisfaction
• Service-level latency for the namespaces exposing HTTP endpoints
• Cluster autoscaler events
• Persistent storage capacity per StorageClass

These dashboards had been running for over a year. They had caught real production incidents. They had triggered useful alerts. By any reasonable definition, the cluster had observability.

What the cluster did not have was operational triage.

2. The Difference Between Visibility and Triage

I want to be careful about the language here, because “observability” has become a marketing term that covers everything from raw metrics to AI-powered anomaly detection.

For the purposes of this investigation, here is the distinction that matters:

Metrics dashboards are excellent at the first question. They aggregate time-series data, apply threshold alerts, and tell you whether the cluster is keeping its promises about latency and availability.

kubectl and visual cluster explorers are excellent at the second question. Given a specific pod or namespace, they show you exactly what is happening.

Neither answers the third question. Neither tells an on-call engineer where to look first when 243 pods need triage.

This is the gap operational triage fills.

Methodology note: The findings shown in this investigation were collected from a real non-production Azure Kubernetes Service cluster. Resource names and identifying details have been anonymized. Cluster state was observed using read-only tooling and no changes were made during the investigation.

3. What the Scan Actually Found

A read-only scan against this cluster returned the following:

Critical findings (13 total)

4 pods crash-looping
  └─ team-a/data-pipeline-7fd867f4b-pwpvn (1,824 restarts)
  └─ team-a/event-processor-9c5d6b8d7-tzrs4 (1,337 restarts)
  └─ team-c/auth-service-7df57bb979-cksz8 (1,297 restarts)
  └─ team-b/notification-handler-55d744ccc4-xqzwd (1,269 restarts)

4 image-pull failures
  └─ platform-shared/cache-warmer (cannot pull image, 4 days)
  └─ team-c/billing-reconciler (cannot pull image, 4 days)
  └─ team-a/report-exporter (cannot pull image, 2 days)
  └─ team-b/audit-logger (cannot pull image, 2 days)

3 OOMKilled in last 24h
  └─ data-services/aggregator-job (killed 7 times)
  └─ team-c/order-processor (killed 3 times)
  └─ team-a/search-indexer (killed 2 times)

2 orphaned PersistentVolumeClaims
  └─ team-a/forgotten-data-backup (10 GiB, no consumer, 14 days)
  └─ team-b/temp-build-storage (50 GiB, no consumer, 21 days)

Warnings (3 total)

3 namespaces missing NetworkPolicy
  └─ gateway (8 pods, externally accessible workloads)
  └─ monitoring (12 pods, can reach all internal services)
  └─ auth-services (5 pods, contains sensitive workloads)

Security posture

CIS Pod Security score: 14/100
  └─ 31 containers running as root
  └─ 3 privileged containers
  └─ 14 containers without resource limits
  └─ 7 containers without read-only root filesystem

The scan took 9 seconds. None of these issues had triggered any alert in the monitoring stack.

4. Why the Dashboard Missed All of This

This is not a failure of any specific monitoring tool. It is a structural mismatch between what dashboards measure and what cluster operations require.

Consider the first finding: a pod that had restarted 1,824 times over four days.

To a metrics dashboard watching this deployment:

• The deployment’s desired replica count is satisfied — the kubelet keeps restarting the pod
• A pod exists in the Running state between crashes
• CPU and memory metrics look normal because the container barely lives long enough to consume anything significant
• The service endpoint exists (between crashes)

From the dashboard’s perspective, this deployment is healthy. The metrics it monitors are within thresholds.

The same dashboard cannot easily ask: “Has this pod restarted more than 100 times in the last day?” That requires looking at containerStatuses[].restartCount over time, which is per-pod state, not aggregate metric data. Most dashboards do not store this.

The orphaned PVCs are even more invisible. Storage capacity dashboards measure consumed storage. A 10 GiB PVC with no consuming pod is not “consumed” in any monitoring sense. It exists in the cluster, costs money every hour, and no metric tracks it.

The missing NetworkPolicies are completely invisible to metrics dashboards. NetworkPolicies are a configuration object, not a runtime metric. Dashboards do not audit cluster configuration.

These aren’t oversights in dashboard design. These are different questions that require different infrastructure.

5. The Operational Triage Pattern

The pattern I want to describe is structurally simple but operationally distinct from observability.

Operational triage answers four questions for a Kubernetes cluster:

1. What is broken? Pods in CrashLoopBackOff, ImagePullBackOff, OOMKilled
2. How bad is it? Severity classification — production vs development, system vs user workloads
3. Where is it? Namespace, resource name, owning team
4. What should I do next? A specific kubectl command, not a generic recommendation

The implementation has three properties that distinguish it from observability tooling.

Read-only by design

A triage layer must never modify cluster state. It is an observation tool, not a remediation tool. The Kubernetes ClusterRole it runs as should contain get and list verbs only — no create, update, patch, or delete. This makes it deployable in production environments without requiring approval from platform security teams.

Stateless

Each scan starts fresh. The triage layer does not maintain a database, does not require historical data, and does not need agents on cluster nodes. This is intentionally the opposite of how observability tools work. It is appropriate because triage answers a present-tense question (“what is wrong now?”) rather than a historical one (“how did we get here?”).

Aggregated by category

The triage layer does not list 4 individual crash-looping pods. It says “4 pods crash-looping” with a drill-down to each pod. This matches how a human triages an incident — first by severity and category, then by specific resource. A flat list of every problem is itself a form of noise.

6. Architecture of a Triage Layer

The architecture for an operational triage layer is straightforward:

                ┌─────────────────────────┐
                │   Kubernetes API Server │
                └────────────┬────────────┘
                             │ read-only
                ┌────────────▼────────────┐
                │  Triage Layer (in-mem)  │
                │                         │
                │   ┌─────────────────┐   │
                │   │ Crash Detector  │   │
                │   │ Image Detector  │   │
                │   │ OOM Detector    │   │
                │   │ PVC Detector    │   │
                │   │ NetPol Detector │   │
                │   │ Security Audit  │   │
                │   └────────┬────────┘   │
                │            │            │
                │   ┌────────▼────────┐   │
                │   │   Prioritizer   │   │
                │   │  (severity +    │   │
                │   │   group count)  │   │
                │   └────────┬────────┘   │
                └────────────┼────────────┘
                             │
                ┌────────────▼────────────┐
                │  Operator-facing view   │
                │  (CLI / Dashboard / API)│
                └─────────────────────────┘

Each detector is responsible for one category of operational risk. The prioritizer aggregates results by severity and group count. The output is a small, ordered list of what deserves attention.

Here is the core of a crash-loop detector in Go. The same pattern applies whether you build this yourself, use kubectl with shell scripting, or extend an existing tool:

type Issue struct {
    Severity   string
    Type       string
    Namespace  string
    Resource   string
    Message    string
    KubectlCmd string  // ready-to-run command
}

func DetectCrashLoops(client *kubernetes.Clientset) ([]Issue, error) {
    pods, err := client.CoreV1().Pods("").List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, err
    }

    var issues []Issue
    for _, pod := range pods.Items {
        for _, c := range pod.Status.ContainerStatuses {
            if c.State.Waiting == nil {
                continue
            }
            if c.State.Waiting.Reason != "CrashLoopBackOff" {
                continue
            }
            issues = append(issues, Issue{
                Severity:  "critical",
                Type:      "crash_loop",
                Namespace: pod.Namespace,
                Resource:  pod.Name,
                Message: fmt.Sprintf("CrashLoopBackOff - %d restarts",
                    c.RestartCount),
                KubectlCmd: fmt.Sprintf("kubectl logs %s -n %s --previous",
                    pod.Name, pod.Namespace),
            })
        }
    }
    return issues, nil
}

Each detector follows the same pattern: pull state from the API, filter for unhealthy conditions, return structured issues with kubectl commands. The full scan against a 243-pod cluster completes in approximately 9 seconds.

The prioritizer then groups identical issue types and orders by severity:

func PrioritizeIssues(issues []Issue) []IssueGroup {
    // Group by Type + Severity
    grouped := make(map[string][]Issue)
    for _, i := range issues {
        key := i.Severity + ":" + i.Type
        grouped[key] = append(grouped[key], i)
    }

    // Convert to ordered groups
    var groups []IssueGroup
    for _, items := range grouped {
        groups = append(groups, IssueGroup{
            Severity: items[0].Severity,
            Type:     items[0].Type,
            Count:    len(items),
            Examples: items[:min(3, len(items))],
        })
    }

    // Sort: critical first, then by group size
    sort.Slice(groups, func(i, j int) bool {
        if groups[i].Severity != groups[j].Severity {
            return severityRank(groups[i].Severity) <
                   severityRank(groups[j].Severity)
        }
        return groups[i].Count > groups[j].Count
    })

    return groups
}

The output is what an operator actually wants to see at 2 AM: a prioritized list of categories of trouble, with a specific resource example for each.

7. From 13 Issues to 5 Things to Fix

When the prioritizer ran against this cluster’s findings, it produced this:

Top 5 things to fix (prioritized by impact)

1. 4 pods crash-looping
   Examples: team-a/data-pipeline (1,824 restarts), 2 more
   Severity: CRITICAL
   Next step: kubectl logs <pod> -n <ns> --previous

2. 4 image-pull failures (4 days old)
   Examples: platform-shared/cache-warmer, 3 more
   Severity: CRITICAL
   Next step: kubectl describe pod <pod> -n <ns>

3. 3 namespaces missing NetworkPolicy
   Examples: gateway, monitoring, auth-services
   Severity: HIGH
   Next step: kubectl apply -f default-deny-networkpolicy.yaml

4. 2 orphaned PVCs (~$40/mo wasted)
   Examples: team-a/forgotten-data-backup, 1 more
   Severity: MEDIUM
   Next step: kubectl get pvc --all-namespaces

5. Security score below recommended threshold (14/100)
   6 of 7 CIS Pod Security checks failed
   Severity: MEDIUM
   Next step: review pod security context configurations

This is the output that matters. Not 243 pods. Not 28 namespaces. Five things to fix, in order.

For the on-call engineer at 2 AM, this is the difference between paralysis and progress.

8. What This Pattern Does Not Solve

Honest engineering means being clear about limits. The operational triage pattern has real boundaries.

It does not answer:

• “When did this issue start?” No historical data. Use Prometheus and your event store.
• “Is this issue getting worse?” No trend data. The triage layer is point-in-time.
• “Did our last deployment cause this?” No event correlation. Use your deployment audit log.
• “Why is the application slow?” No application-level visibility. Use APM tools.

For everything within the Kubernetes API surface — pods, deployments, statefulsets, PVCs, services, namespaces, network policies, configmaps — the triage pattern works well. For everything outside that surface, you still need observability tooling.

The triage layer is not a replacement for observability. It is the layer that tells you which questions to ask of your observability tooling.

9. One Implementation

The operational triage pattern is tool-agnostic. You can implement it with kubectl scripts, custom controllers, or extensions to existing platforms like Lens.

One implementation of the operational triage approach described in this article is OpsCart Watcher, a read-only open-source Kubernetes operational intelligence tool that aggregates reliability, security, cost, and waste findings into a prioritized view. The triage methodology itself is independent of any specific tool and can be applied using kubectl, Lens, custom scripts, or any Kubernetes platform with API access.

The findings in this article were collected using OpsCart Watcher v1.0.0. The scan output shown earlier (“Top 5 things to fix”) is rendered as a dashboard.

If you want to validate the pattern on your own cluster:

kubectl apply -f https://raw.githubusercontent.com/opscart/opscart-k8s-watcher/main/deploy/dashboard.yaml
kubectl port-forward -n opscart-system svc/opscart-dashboard 8080:80

The Docker image is built FROM scratch, runs as non-root, requires no cloud credentials, and ships with a read-only ClusterRole. A trivy image scan against it returns zero vulnerabilities. The full ClusterRole is auditable in deploy/dashboard.yaml.

You can also implement the pattern without any external tooling. The simplest possible version is one kubectl command:

kubectl get pods --all-namespaces \
  --field-selector=status.phase!=Running,status.phase!=Succeeded \
  -o wide

This will surface most CrashLoopBackOff and ImagePullBackOff issues. Run it on the largest production cluster you manage. Count how many pods come back in something other than Running or Succeeded.

That number is your operational triage debt. It is the gap between what your monitoring tells you and what your cluster is actually doing.

10. The Real Conclusion

The cluster I scanned for this investigation is healthy by every metric the platform team monitors. It is also accumulating 13 critical operational issues that no one has been alerted to. Both statements are true at the same time.

This is the gap that operational triage fills. It is not a replacement for observability. It is not a competitor to dashboards. It is the missing layer that turns 243 pods of state into 5 things to fix.

Visibility was never the bottleneck in Kubernetes operations. We see more cluster data than any single human can process. The bottleneck is attention — knowing which 5 things, out of thousands of resources, deserve a human’s time right now.

243 pods.

Five actually mattered.

The other 238 were running fine, in namespaces that did not need anyone’s attention that day. The job of an operational triage layer is to make that distinction obvious in under a minute.

Pick the largest production cluster you operate. Run the kubectl command from the previous section. Count the pods that come back.

The number you get is the gap your current tooling has not been measuring.