How to Benchmark AWS Performance After Migration
Published on · Updated on
Reworked the guide around repeatable before-and-after tests, current AWS telemetry, bottleneck diagnosis, and cost comparison.
A migration can change response times, capacity, reliability, and operating cost even when the application passes its smoke tests. To understand the difference, compare the old and AWS environments under the same representative workload, record how each test was run, and connect user-facing results to service metrics.
This guide explains how to build that comparison. A load generator creates traffic; CloudWatch metrics, application logs, and traces measure how the workload responds. Those tools have different jobs.
Set goals for the comparison
Measure user outcomes and the capacity behind them
| Measure | What to record |
|---|---|
| Latency | End-to-end time for important requests or transactions. Compare percentiles such as p50, p95, and p99 for the same operations; an average alone can hide slow requests. |
| Throughput and errors | Offered load and successfully completed requests or transactions per second, along with errors, timeouts, and retries. A test that sends more requests but completes fewer is not faster. |
| Resource behavior | Relevant service metrics such as CPU, memory pressure, storage latency and queue depth, network use, database load, throttles, and scaling events. Choose metrics for the services in the workload. |
| Cost | Cost for the same workload scope and a comparable operating period, including relevant compute, storage, data transfer, and monitoring. Attribute shared costs consistently; where attribution is sound, compare cost per successful transaction. Label modeled estimates separately from billed cost. |
Set success criteria from the workload
Choose the user journeys and service objectives that matter before testing. For example, a checkout flow may have a response-time objective, a minimum successful transaction rate, and a maximum error rate. Derive success and safe stop criteria from your service commitments and observed business needs; there is no universal AWS latency or CPU threshold that fits every workload.
Prepare a comparable test
Capture the pre-migration baseline
Before cutover, record the application version, workload and data shape, traffic mix, client location, dependency configuration, and resource setup alongside the measurements. Include representative normal and peak periods. If you already migrated, use historical monitoring, application logs, or saved test results where available.
If no reliable pre-migration measurements exist, establish a baseline on AWS and compare it with your service objectives and production behavior. You can still find bottlenecks, but you cannot make a precise before-and-after performance claim from missing data.
Keep the workload and conditions consistent
Use the same application version, representative dataset, request mix, and load pattern in both environments where possible. Record any differences that cannot be held constant, such as instance capacity, database engine, network path, cache state, or dependent service. Those differences are part of the result and may explain a change in performance.
A smaller staging system can catch functional regressions, but its capacity and scaling results do not directly represent production unless you have validated the relationship. AWS recommends realistic workloads and a production-like environment for load tests; use synthetic or sanitized data. See the AWS Well-Architected guidance on load testing a workload.
Configure telemetry for the services involved
Use CloudWatch to collect and compare supported service metrics, logs, and alarms. Confirm the metric coverage, dimensions, and collection interval for each resource before the test. EC2 guest memory is not included in the default instance metrics; collect it with the CloudWatch agent or another host monitoring system.
Instrument the application path so you can see where a request spends time across services. AWS recommends migrating X-Ray instrumentation to OpenTelemetry; the X-Ray SDKs and daemon entered maintenance mode on February 25, 2026. See the support timeline and AWS's migration guidance. For Amazon RDS and Aurora, CloudWatch Database Insights shows database load by waits and SQL contributors; compare those signals with application latency. See the AWS Database Insights documentation and our RDS guide.
Run the same workload on both sides
Use a load generator for traffic
Choose a load-testing tool that can reproduce the relevant protocol, request mix, arrival rate or concurrency, and user journeys. For example, Apache JMeter test plans and Locust user flows can generate scripted load; choose based on your workload. CloudWatch metrics and tracing tools observe a test; they do not generate user traffic. Keep test data synthetic or sanitized, and make sure test requests cannot trigger unintended customer actions such as real payments or email.
Run controlled scenarios and save the conditions
- Start from a stable deployment and make sure the test generator can produce the required load without becoming the bottleneck.
- Run a short smoke test, then warm the application as your scenario requires. State whether caches and serverless initialization are warm or cold; do not mix those conditions across runs.
- Apply a repeatable traffic pattern for a fixed period. Cover representative normal and peak demand, and test bursts or sustained load when those patterns matter to the workload.
- Record the offered load, successful throughput, latency percentiles, errors, resource metrics, test duration, deployment and configuration versions, and any relevant background activity.
- Repeat each scenario more than once in both environments and inspect run-to-run variability before concluding that one is faster. Save the generator settings and timestamps so another engineer can reproduce the comparison.
Use a dedicated production-like environment for high-load tests. If you use production telemetry to confirm real user experience, distinguish that evidence from a controlled load test and keep any synthetic traffic within an approved, safe limit.
Example benchmark record
This illustrative record shows what to hold constant and what to compare; it contains no measured results or performance thresholds.
| Field | Record for both environments | Compare across runs |
|---|---|---|
| Scenario | checkout-v1 user journey | p50, p95, and p99 end-to-end latency |
| Application and data | Same release identifier and sanitized dataset snapshot | Successful transactions per second and errors or timeouts |
| Load profile | Same saved script, offered traffic pattern, client region, warm-up policy, and run duration | Resource metrics, trace spans, and scaling behavior during the run |
| Environment | Record service sizes, versions, and any differences | Cost for a comparable workload scope and period, with shared costs attributed consistently |
Compare results and find the constraint
Compare like conditions
For each scenario, compare latency percentiles at the same offered load, successful throughput, and error or timeout rates. Align the application timeline with resource metrics and trace spans. Note differences in capacity, data, cache state, and network path instead of attributing every change to the cloud platform.
Compare resource signals near the time performance degrades. High latency with saturated CPU suggests a different investigation from high latency with CPU headroom and a growing storage queue, database wait, or downstream delay. A metric is evidence about a constraint, not proof of its root cause.
Trace the slow part before changing capacity
Use traces to locate slow service calls, then use service-specific telemetry to test the likely cause. For an RDS or Aurora bottleneck, Database Insights shows database load and can break it down by waits and SQL statements. Those signals help direct investigation; correlate them with application latency and resource metrics before changing a query, storage, or instance size.
Make one well-scoped change at a time where practical. Resize or scale a resource only when the measurements point to a capacity constraint, and preserve the service's availability and recovery requirements.
Rerun the baseline after a change
After tuning, rerun the same scenario with the same load pattern and test conditions. Compare the result to the original baseline, check that error rates and recovery behavior remain acceptable, and record both the configuration change and its effect. This shows whether the change improved the workload rather than only changing an isolated metric.
For migration planning, data verification, and a tested cutover path, see our AWS content migration guide.
Keep the comparison useful over time
Choose a test cadence that fits the workload
Run fast, stable performance checks in CI when they can catch regressions without noisy results. Reserve larger capacity, burst, and endurance tests for changes to application behavior, infrastructure, traffic patterns, or migration architecture. Keep the workload definition and results with the deployment version, and base any release gate on an agreed service objective and repeatable evidence.
Conclusion
Make the comparison repeatable
A useful post-migration benchmark is a repeatable comparison, not a single score. Preserve the workload definition, record differences between environments, and tie any tuning decision to user-facing results and supporting service metrics.
FAQs
What metrics should I compare after migration?
Compare end-to-end latency percentiles, successful throughput at a stated offered load, errors and timeouts, relevant service resource metrics, and cost over a comparable period. Choose metrics and limits that match your workload and service objectives.
Does a smaller test environment represent production performance?
Not by itself. A smaller environment can validate behavior, but capacity and scaling results are directly comparable only when the relationship to production has been validated. Record any environment differences and use a production-like setup for capacity conclusions.
How do I find a post-migration bottleneck?
Align application latency with traces and service metrics for the same test interval. Investigate the service call, database wait, queue, or resource that changed with the user-facing symptom, then test one targeted fix with the same workload.