Top 30 Kubernetes Troubleshooting & CNI Questions: CrashLoopBackOff, CoreDNS & OOMKilled Guide

Senior Kubernetes engineering and Site Reliability Engineering (SRE) interviews have abandoned generic trivia questions like “What is the difference between a Deployment and a StatefulSet?”. Modern technical interview loops focus intensely on live operational post-mortems and cluster triage. Interviewers present broken clusters, failing pod states, and network packet dumps, asking: “A pod is stuck in CrashLoopBackOff with Exit Code 137, while DNS queries to other services are intermittently failing across two worker nodes. Walk me through your command-line triage process.”. In this master interview guide, we break down the Top 30 scenario-based Kubernetes troubleshooting and Container Network Interface (CNI) interview questions, detailing the Linux kernel mechanics, kubectl triage commands, and architectural remediations required to ace L2/L3 infrastructure interviews.

Kubernetes War Room Triage & Knowledge Domain Matrix

Scenario 1: Diagnosing OOMKilled (Exit Code 137) vs. Graceful Exit

The Interview Scenario: A critical Java microservice pod terminates repeatedly in production. kubectl get pods shows STATUS: CrashLoopBackOff with RESTARTS: 14. How do you definitively prove whether the application crashed due to an internal unhandled Java exception or was murdered by the Linux Kernel Out-Of-Memory (OOM) killer?

The Architectural Solution:

• Command-Line Inspection: Run kubectl describe pod <pod-name> and inspect the Last State container block. Look for Exit Code: 137 and Reason: OOMKilled. Exit Code 137 corresponds to 128 + 9 (SIGKILL), indicating the Linux kernel sent a non-catchable kill signal.
• Host-Level Verification: On the worker node hosting the pod, run dmesg -T | grep -E -i 'oom[-_]killer|killed process'. The kernel log will print the exact memory consumption, the cgroup path, and the victim process PID.
• Java JVM Trap: If running Java without JVM container awareness (-XX:+UseContainerSupport), the JVM calculates its heap allocation based on the host node’s physical RAM (e.g., 64GB) rather than the pod’s container limit (e.g., 4GB), immediately triggering an OOMKill upon memory spike.

# Command line triage workflow for Exit Code 137
kubectl get pod order-service-78f99d6-x8k2l -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

# Expected diagnostic output:
# {"exitCode":137,"finishedAt":"2026-11-05T08:30:12Z","reason":"OOMKilled","startedAt":"2026-11-05T08:15:00Z"}

Scenario 2: The 5-Second DNS Delay (Linux Kernel Conntrack Race Condition)

The Interview Scenario: High-traffic microservices experience random 5-second request timeouts when resolving other internal cluster services (e.g., database.prod.svc.cluster.local). The failure happens only on high-concurrency clusters. What is causing this, and what is the fix?

The Architectural Solution:

• Underlying Mechanism: The Linux glibc resolver sends A and AAAA DNS queries concurrently over UDP from the same socket. Under high concurrency, both UDP packets traverse the Linux kernel Netfilter connection tracking table (conntrack) simultaneously. Due to an internal kernel lock race condition in older kernels/iptables, one of the two conntrack entries drops, forcing glibc to wait for its default 5-second UDP retransmission timeout.
• Engineering Fix:
  1. Deploy NodeLocal DNSCache: Runs a DNS caching agent on every node as a DaemonSet, terminating UDP lookups locally and communicating with upstream CoreDNS via TCP.
  2. Add single-request-reopen or use-vc to pod DNS configuration options via dnsConfig to force sequential lookups or TCP mode.

# Pod Spec Fix to bypass the conntrack 5-second race condition
spec:
  dnsConfig:
    options:
      - name: single-request-reopen
      - name: ndots
        value: "2"

Scenario 3: CNI IP Address Exhaustion & Pod Sandbox Creation Failed

The Interview Scenario: When an Autoscaler triggers during a traffic surge, new pods are stuck in ContainerCreating. The event logs report: “Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container: CNI failed to allocate IP address”. How do you triage this on AWS VPC CNI or Calico?

The Architectural Solution:

• Root Cause: The node’s primary network interface or secondary IP address pool in the CNI IPAM daemon (e.g., aws-node or calico-node) has run out of allocatable IP addresses from the assigned subnet CIDR block.
• AWS VPC CNI Triage: Inspect the node’s ENI limits. An m5.large instance can attach a maximum of 3 ENIs with 10 IPs per ENI (max 30 pods). If the subnet has available IPs but the node cannot attach another ENI, verify WARM_IP_TARGET and MINIMUM_IP_TARGET settings in the aws-node daemonset.
• Permanent Fix: Enable Prefix Delegation (ENABLE_PREFIX_DELEGATION=true on AWS VPC CNI) to assign /28 IPv4 blocks instead of individual secondary IPs, increasing pod capacity per node to over 110 pods.

Scenario 4: MTU Size Mismatch Causing Silent Packet Drops

The Interview Scenario: Pods can communicate with small HTTP GET requests, but large payloads (e.g., file uploads or large JSON blobs > 1500 bytes) hang indefinitely until client timeout. Pings between pods succeed with zero loss. What is the root cause?

The Architectural Solution:

• Root Cause: MTU (Maximum Transmission Unit) mismatch in the CNI overlay network (VXLAN or Geneve). Overlay encapsulation adds 50 bytes of outer packet headers. If the host physical interface has an MTU of 1500, but the CNI virtual interface inside the pod is also set to 1500, large packets exceed the physical MTU (1550 bytes). If intermediate switches drop packets with the DF (Don’t Fragment) bit set, the packets vanish silently without an ICMP Fragmentation Needed notification.
• Remediation: Configure the CNI MTU to be at least 50 bytes smaller than the underlying host MTU (e.g., set CNI MTU to 1450 for standard 1500-byte physical networks, or 8950 for 9000-byte Jumbo Frame networks).

Comprehensive Kubernetes Diagnostic Error Matrix

Scenario 5: CoreDNS High CPU & Upstream Throttling

In large Kubernetes clusters running thousands of microservices, CoreDNS frequently becomes a bottleneck. When CoreDNS pods hit CPU limits, DNS latency spikes cluster-wide:

# Triage CoreDNS pod metrics and query latency via Prometheus metrics endpoint
kubectl top pods -n kube-system -l k8s-app=kube-dns

# Inspect CoreDNS live query logs to identify runaway misconfigured workloads
kubectl logs -n kube-system -l k8s-app=kube-dns -f --tail=100

# Scale CoreDNS deployment or deploy horizontal pod autoscaler
kubectl scale deployment coredns -n kube-system --replicas=6

Frequently Asked Questions (FAQ)