Kubernetes CPU Limits vs. Requests: The 'Invisible Throttling' Trap

One of the most confusing operational issues in Kubernetes is watching an application slow down or drop requests while your Grafana dashboard reports comfortable resource headroom—say, only 30% to 50% average CPU usage.

This issue frequently surfaces in stateful services, CI/CD runners, and test clusters (like Selenium Grid nodes running headless Chrome sessions). You scale down CPU limits thinking the workload is idle or over-provisioned, and suddenly the application becomes unstable.

The culprit is almost always CFS (Completely Fair Scheduler) quota throttling.

Here is what is happening under the hood and how to diagnose and prevent it.

1. How Kubernetes CPU Limits Actually Work

In Kubernetes specifications, you declare two distinct resource parameters:

resources:
  requests:
    cpu: "500m"
    memory: "1Gi"
  limits:
    cpu: "1000m"
    memory: "2Gi"

• requests are used by the Kubernetes kube-scheduler to find a worker node that has sufficient capacity. It guarantees baseline compute slice allocation.
• limits are enforced at the Linux cgroup level via the CFS quota system.

Unlike memory limits (which trigger an immediate Out-Of-Memory OOMKilled event when exceeded), CPU is a compressible resource. When a container exceeds its CPU limit, the Linux kernel does not kill the process; it throttles it.

2. The 100ms CFS Quota Window

Linux CFS divides time into periods, by default 100 milliseconds (100,000 microseconds).

When you specify a limit of cpu: 1000m (1 core), you are granting your container 100ms of CPU run time per 100ms period.

Now consider a multi-threaded process (such as a Java JVM, a headless browser session, or an asynchronous worker pool):

1. The process spawns 4 concurrent threads.
2. An incoming burst of work arrives. All 4 threads run simultaneously at 100% capacity.
3. In just 25 milliseconds of real time, the 4 threads consume 4 * 25ms = 100ms of CPU quota.
4. For the remaining 75 milliseconds of the period, the Linux kernel completely suspends the container's execution.
5. The container cannot respond to network packets, process health checks, or execute application code until the next period starts.

Why Your Dashboard Misses It

Standard Prometheus scraping typically happens every 15, 30, or 60 seconds. When your monitoring tool averages CPU consumption over 30-second intervals (rate(container_cpu_usage_seconds_total[1m])), the sharp 25ms burst and 75ms sleep smooths out into an innocuous, low average number (e.g., 25% or 30%).

The dashboard looks green, but end users and client calls experience latency spikes and timeout errors.

3. How to Detect Invisible Throttling

To check whether CPU throttling is degrading your workload, inspect the cgroup throttling metrics exported by cAdvisor:

# Percentage of periods throttled
sum(rate(container_cpu_cfs_throttled_periods_total{container="my-app"}[5m]))
/
sum(rate(container_cpu_cfs_periods_total{container="my-app"}[5m])) * 100

If this metric shows values above 5% to 10%, your application is spending meaningful slices of time frozen by the kernel.

You can also inspect the container directly within the node's cgroup filesystem:

# Inside the container or on the node (cgroup v1 path example)
cat /sys/fs/cgroup/cpu/cpu.stat
# Look for nr_throttled and throttled_time

4. Practical Fixes and Best Practices

A. Set Realistic Requests and Remove or Loosen Arbitrary Limits

Unless you are running an untrusted multi-tenant cluster where strict hard limits are legally or operationally required, setting tight CPU limits on multi-threaded workloads is often counter-productive.

A common pattern in high-performance production setups is:

• Set requests.cpu accurately: Set this to your actual expected peak or steady-state demand. Kubernetes uses requests for scheduling and prevents node starvation.
• Set limits.cpu significantly higher (or omit limits entirely): Allow the container to burst into unused node CPU capacity during short spikes without getting throttled.

B. Size for Concurrency

If your container runs an application with 4 or 8 worker threads, giving it a 1-core limit is a recipe for CFS throttling. Ensure CPU limits align with thread pool sizes.

C. Observe Behavioral Instability in Test Infrastructure

In test platforms like Selenium Grid or Jenkins executors, browsers and test runners generate intense, brief CPU spikes during DOM rendering and asset decoding. Trimming pod limits based purely on idle Grafana averages often causes test instability, flaky connection drops, and false-positive QA failures.

Summary

Do not rely solely on standard average CPU usage graphs to judge container health. Whenever an application experiences unexplained latency spikes or intermittent dropouts on Kubernetes, check container_cpu_cfs_throttled_periods_total first. Setting realistic requests and giving workloads headroom to burst is often the fastest path to stability.