How Do I Prioritize a Migration Queue Without Boiling the Ocean?

Migrating workloads and optimizing cloud infrastructure can feel like trying to drink from a firehose. The natural temptation is computingforgeeks.com to fix everything at once, but that quickly leads to paralysis by analysis or costly delays. Instead, prioritizing a migration queue effectively requires a methodical, data-driven approach that balances impact with engineering effort.

From my 12 years of experience across AWS, Azure, and Google Cloud, there’s a handful of common pitfalls that trip up many teams. This post will walk through how to build a candidate shortlist, identify quick wins, and systematically approach migration prioritization using real data—plus practical guidance on leveraging tools like AWS Compute Optimizer and Azure Advisor to your advantage.

Why Prioritization Matters

Cloud cost and performance optimizations rarely happen overnight. Teams often wrestle with a growing migration backlog consisting of dozens or hundreds of workloads. Tackling the entire queue indiscriminately isn’t just inefficient — it's a recipe for burnout and wasted engineering hours.

Prioritizing migrations smartly allows you to:

Target services where optimization will pay off with tangible cost savings or stability improvements
Avoid wasted effort on workloads that are either well sized or where optimization benefits are marginal
Build momentum with quick wins that demonstrate value and bolster stakeholder confidence
Understand trade-offs between timing, engineering effort, and impact

Common Pitfalls That Inflate Migration Queues

Before diving into prioritization techniques, beware these cost and sizing myths that inflate your migration queue unnecessarily:

Always-on small services hide cloud waste. Small, low-utilization services can easily slip under the radar. But cumulatively, they often account for substantial waste, especially if they're continuously running without real demand spikes.
Shared CPU definitions differ by cloud provider. Don’t treat vCPU counts as strict performance guarantees. For instance, AWS’s “burstable” t-series and Azure’s B-series CPUs operate differently under load.
Decisions based solely on average CPU utilization are misleading. Average utilization masks peaks that cause throttling or latency issues — and these peaks often determine whether a migration or resizing is warranted.

Step 1: Define Your Observation Window and Metrics

Before you build your candidate shortlist, decide what your observation window will be. Metrics collected over too short or too long periods mislead prioritization.

Typical observation windows: 2–4 weeks capture workload seasonality without overwhelming with noise.
Key metrics to collect: CPU utilization, memory usage, network throughput, I/O rates, and importantly, the duration and frequency of utilization spikes.
Percentile metrics: Focus on P95 and P99 percentiles rather than averages. These metrics better reflect peak resource demands that trigger scaling or throttling.

For example, if a service averages 20% CPU but spikes to 90% CPU for 5 minutes 3 times a day, it’s a candidate for resizing differently than a service consistently at 20% CPU.

Why Percentiles and Spike Duration Matter

Cloud infrastructure providers charge and bill based on provisioned resources, not averages. But the user experience suffers when resource supply can't keep up with peak demand. Since the cloud bills continually, you want to know:

How often resource usage exceeds certain thresholds?
How long these excursions last?

This lets you understand whether you need bigger instances, autoscaling, or are paying for idle capacity you no longer need. For example, brief CPU spikes lasting seconds might be tolerable, but multi-minute sustained peaks require action.

Step 2: Use Cloud-Native Recommendations as a Starting Point, Not a Panacea

Both AWS Compute Optimizer and Azure Advisor provide excellent automated sizing recommendations, but:

They rely heavily on average utilization metrics and internal heuristics.
Their understanding of “shared CPU” or burstable instance types varies, affecting recommendations.
They may not incorporate application-level performance impact or interdependencies.

**Actionable advice:** Use these tools to generate your initial migration candidate list, but refine it using your own percentiles and spike durations.

Interpreting AWS Compute Optimizer Outputs

Recommendation Category Description Considerations Under-provisioned Instances with resource constraints during peak load Check spike duration and if autoscaling mitigates impact Over-provisioned Resources consistently underutilized Verify no sudden surge is missed; confirm no latency impact Idle Instances running with near-zero utilization Check dependency graphs before decommissioning

Azure Advisor Recommendations Nuances

Azure Advisor also flags unused disks and unattached IPs — these non-compute resources add hidden cost.
Recommendations differ if your service is part of an App Service Plan or a VM Scale Set, so consider group sizing effects.

Step 3: Build Your Candidate Shortlist Using Data-Driven Filters

Use the following logical approach to build your shortlist:

Filter out always-on small services with utilization well below thresholds and no latent growth signals — flag these for potential downtime or consolidation.
Flag instances with sustained P95 or above CPU/memory exceeding 75% over 5+ minutes repeatedly — these will likely need upsizing or autoscaling.
Exclude burstable/shared CPU types with sporadic spikes if the P99 spike duration is under 30 seconds and latency is not impacted.
Account for provider-specific definitions of shared CPU. For example, Azure B-series provides credits vs. AWS T-series burstable instances—treat differently.
Exclude instances flagged as idle without attached storage or services relying on them.

Example Filtering Table

Instance Avg CPU P95 CPU Spike Duration (P95) Recommendation Service A 15% 85% 6 minutes Upsize / Autoscale Service B 5% 23% 30 seconds Consider downsizing or consolidation Service C (burstable) 20% 95% 10 seconds Monitor; no immediate action

Step 4: Estimate Engineering Effort vs. Value for Quick Wins

Engineering effort to migrate or resize varies greatly depending on the workload complexity, dependencies, and platform. To avoid boiling the ocean, categorize candidate workloads by estimated effort and potential impact:

Candidate Type Engineering Effort Cost/Performance Impact Example Prioritization Low Effort, High Impact Days Cost reduction > 20% Right-sizing idle VMs, automating shutdown Highest priority (quick wins) Medium Effort, Medium Impact Weeks Cost reduction 10-20% Migrating monolith to autoscaling groups Next priority High Effort, Low Impact Months Cost reduction <10% Rebuilding custom infrastructure components Consider deprioritizing

Make sure you also set rollback criteria before running any pilot migration—including validation of P95/P99 metrics post-migration and service-level objective (SLO) adherence.

Step 5: Iterate, Measure, and Adjust Your Planning

Prioritization is not a one-and-done exercise. After completing a migration or resizing, update your observation windows and metrics to measure the impact and reprioritize remaining workloads accordingly.

Leverage your telemetry to detect regressions or unexpected costs.
Re-run AWS Compute Optimizer / Azure Advisor recommendations on updated environments.
Adjust your prioritization rationale based on observed durations and resource usage.

Summary: Best Practices to Prioritize Without Boiling the Ocean

Use the right observation window: Capture 2-4 weeks of data granularity with P95/P99 and spike durations.
Don’t be fooled by averages: Focus on peak utilizations and how long spikes last.
Leverage cloud-native optimizer tools as guidance: AWS Compute Optimizer and Azure Advisor are excellent for candidate identification but need your contextual validation.
Understand provider nuances: Shared CPU burstable instances behave differently—treat their sizing accordingly.
Build a candidate shortlist by filtering data-driven flags: Exclude workloads unlikely to deliver value.
Map engineering effort to potential impact: Prioritize quick wins to build momentum.
Define rollback and validation criteria upfront: This saves time and risk during migration pilots.

By adopting this disciplined approach, you can transform an overwhelming migration backlog into an achievable, impactful roadmap—saving your organization money and improving reliability without boiling the ocean.

If you’re ready to dive in, start by pulling your P95 and P99 utilization metrics now—and ask yourself: which workloads are truly worth your engineering effort?