Is Average CPU Utilization a Bad Metric for Shared CPU Decisions?

When cloud teams evaluate instance sizing and performance, CPU utilization metrics are often the starting point. But relying solely on average CPU utilization can lead to costly missteps—especially when dealing with https://bizzmarkblog.com/are-bots-and-internal-services-good-on-shared-cpu-if-concurrency-is-low/ shared CPU instances found in AWS, Azure, and Google Cloud offerings. These decisions can have immediate implications for latency, cost efficiency, and ultimately, application reliability.

In this post, I’ll dissect why average CPU utilization is a misleading metric, discuss how shared CPU is defined differently by cloud providers, and explain how to leverage percentile utilization and spike duration to make smarter capacity choices. I’ll reference AWS Compute Optimizer and Azure Advisor—two popular advisory tools—and explain what they get right and where their guidance falls short.

Why Average CPU Utilization Can Be Misleading

It’s tempting to judge your workload's CPU needs p95 cpu utilization sizing by glancing at the average CPU utilization from monitoring dashboards or recommendations from your cloud provider’s advisory tools. After all, if your CPU averages 20%, shouldn’t a smaller instance suffice?

Unfortunately, this average often hides critical information:

Short bursts of CPU demand: Many applications experience brief spikes that are 5-10x higher than their average CPU. Relying on average can cause you to size your instance too small to handle these bursts.
Steady but light usage: Services that are always on, like background daemons, internal APIs, or monitoring agents, maintain low constant CPU usage. These “always-on small services” mask the true peak workload and create false confidence in downsizing.
Impact on latency and throttling: For shared CPU instances, CPU bursting credits or shared scheduling policies can throttle your actual compute availability during spikes—even if your average CPU seems low.

In short, average utilization obscures peak behavior and burstiness, which are the real drivers of performance and cost.

Example: Misinterpreting Average CPU on T3 Instances

A common use case involves AWS T3 burstable instances. If a service’s average CPU runs at 25%, Compute Optimizer might recommend downsizing from a t3.large to t3.medium based on average CPU thresholds. But if the P95 or P99 CPU spikes reach 80-90%, the smaller instance will suffer from CPU credit depletion and throttling during bursts—causing increased latency or throttled request failures.

How Shared CPU Definitions Vary Across Cloud Providers

The term shared CPU gets thrown around in cloud conversations, but its meaning and implementation differ between providers.

Provider Instance Type Shared CPU Definition CPU Bursting Behavior AWS T3, T4g (burstable instances) CPU credits earned during idle time enable bursts above baseline usage Bursts until credits run out, then throttles Azure B-series (burstable VMs) Credits accumulate during low usage to allow CPU bursting Burst until credit exhaustion, throttling afterward Google Cloud e2-micro, e2-small Shared physical cores scheduled across guests; credits not explicitly exposed Performance depends on contention, no explicit bursting

Understanding these nuances is critical because the risk of degraded performance varies depending on the burst or sharing scheme. For AWS and Azure burstable instances, credit depletion leads to hard CPU throttling. On Google Cloud's shared core instances, CPU availability depends on hypervisor scheduling contention, which can cause variability without explicit throttling.

Always-On Small Services Hide Cloud Waste

The “always-on small services” pattern exacerbates the problem. These services maintain low-level activity 24/7, causing a non-trivial baseline CPU consumption just to stay alive and responsive.

Examples include health check daemons, logging agents, internal tooling, autoscaling controllers, and lightweight APIs.
Because these services rarely spike, their average CPU utilization looks low and stable, often misleading optimization tools to suggest smaller instances.
However, these services can have intermittent bursts triggered by job executions, background processing, or external traffic surges.

The outcome: advisors like AWS Compute Optimizer and Azure Advisor, which typically optimize based on average CPU over rolling windows, recommend instance sizes that don't accommodate real-world burst needs—leading to over-throttling or reduced performance.

Measure Peaks with the Right Observation Window

One common oversight is choosing an unsuitable time window for CPU metrics aggregation. Minute-averaged CPU data can still mask very short but impactful spikes.

Long aggregation windows (e.g., 1 hour): Smooth out spikes and can hide burst patterns.
Too short windows (<1 second): Produce noisy data that’s hard to interpret for autoscaling or sizing.

The sweet spot is typically around 1-minute granularity, coupled with percentile calculations over daily or weekly periods. This enables you to see how often and for how long CPU spikes occur, factoring in workload burstiness.

Use Percentiles and Spike Duration, Not Averages

To capture the performance realities of shared CPU workloads, focus on the following metrics:

1. Percentile CPU Utilization (P95, P99)

What it shows: The CPU utilization value below which 95% or 99% of all samples fall. This highlights tail-end usage rather than the norm.
Why it matters: If the P99 utilization consistently exceeds your chosen instance's baseline CPU capability, the service risks throttling or degraded performance.
Example guidance: AWS advises to size burstable instances to accommodate CPU spikes at P95 or above rather than average utilization.

2. Spike Duration and Frequency

What it shows: Length and count of CPU usage peaks above critical thresholds.
Why it matters: Short, infrequent spikes might be tolerable even on smaller instances; sustained or frequent peaks require larger buffers or guaranteed CPU.
Analysis technique: Identify how many bursts last longer than 30 seconds or 1 minute at >70% CPU usage to assess burst credit consumption risk.

3. CPU Credit Balances (For Burstable Instances)

Both AWS and Azure expose CPU credit metrics to track burst capacity remaining. Incorporating these time-series alongside percentile utilization informs whether your workload stays within the burstable envelope or regularly exhausts it.

How AWS Compute Optimizer and Azure Advisor Help (and Where They Fall Short)

Both tools provide valuable baseline insights but also illustrate common pitfalls of average-based recommendations.

AWS Compute Optimizer

Uses historical CPU and network utilization metrics, primarily average CPU over periods like 14 days.
Recommends instance sizes based on utilization levels and observed peak demands.
Does surface CPU credit balance information, but doesn't fully substitute for dedicated percentile or spike duration analysis.
Warning: Compute Optimizer’s recommendations might under-provision for bursty workloads if you don’t review P95/P99 spikes and credit usage.

Azure Advisor

Provides right-sizing recommendations based on average CPU and memory usage over a 7-day period.
Highlights potential cost savings but primarily relies on average metrics and heuristics.
Offers VM burst status but less granular credit bursting details compared to AWS.
Advisories may cut instance sizes when average CPU appears low but ignore burst requirements.

Practical Recommendations for Cloud Engineers

Always ask for P95 and P99 CPU utilization metrics before changing instance types. If your monitoring lacks percentile-based dashboards, export raw data to BigQuery, Athena, or Elasticsearch and compute percentiles yourself.
Investigate CPU credit utilization alongside CPU utilization on burstable instances. Low average CPU is meaningless if credits are depleting rapidly during burst periods.
Identify the duration and frequency of CPU spikes. Use a 1-minute granularity or finer logs for performance-critical services.
Beware of always-on small services with low average CPU but occasional demanding bursts. Consider grouping such small services onto shared hosts with guaranteed CPU.
Write rollback criteria into your pilot plans. Define acceptable latency, error rates, and CPU credit depletion thresholds before downsizing instances.
Use reserved or dedicated instances for steady workloads that require predictable CPU capacity. Avoid burstables for latency-sensitive, high-throughput services.

Conclusion

Average CPU utilization seriously underestimates the bursting needs of shared CPU instances. Blindly following average-based sizing recommendations leads to performance bottlenecks, inefficient autoscaling, and cloud waste.

By understanding the different shared CPU models, measuring CPU with percentile utilization and spike duration, and factoring in CPU credit consumption for burstable instances, you can make informed capacity decisions—balancing cost and performance optimally.

Next time you review cloud compute sizing or evaluate auto-scaling policies, dig into those P95 and P99 CPU spikes before hitting “Resize.” Your applications (and wallet) will thank you.