How to Profile HPC Workloads on AWS: A Practical Guide
Published on · Updated on
Rebuilt the guide around repeatable profiling; corrected CloudWatch, EFA, FSx, and profiler guidance; removed unsupported performance claims.
Profiling shows where an application spends time during a run. Monitoring shows how its hosts and AWS resources behave over time. For an HPC job, use both: CloudWatch can reveal instance pressure, selected device activity when configured, and service or file-system metrics, while an application profiler can help explain CPU hotspots, GPU activity, MPI wait time, or rank imbalance. An instance metric by itself does not identify a slow function or explain why one MPI rank is holding up the rest.
A useful investigation starts with one representative workload, records a comparable baseline, and changes one suspected bottleneck at a time. Keep the job's queue delay separate from its execution time so cluster startup and scheduling do not obscure application performance.
Start with the workload and the question
HPC describes several kinds of work: a shared-memory CPU simulation, a multi-node MPI program, a GPU-accelerated model, and a data-heavy pipeline can all have different limits. Begin with a question that a measurement can answer. Examples include whether a single-node solver is CPU-bound, whether adding MPI ranks improves throughput, whether ranks spend substantial time waiting in communication, or whether file access delays a checkpoint.
Record the input and amount of work completed, application version, compiler and build options, number of nodes and ranks, rank placement, instance type, operating-system image, scheduler settings, network configuration, and storage layout. For scaling tests, note whether total work stays fixed (strong scaling) or work per rank stays fixed (weak scaling). Track queue wait, node startup, execution wall time, and completed work separately. Confirm that every comparison produces correct output.
Build a repeatable baseline
- Choose a representative case. Use an input and scale that reflect the behavior you need to improve. Very small tests may hide communication, memory, or I/O costs that matter in a production run.
- Stabilize the comparison. Keep the software, input, node count, launch options, and relevant AWS configuration constant while you collect a baseline. Note whether data is already cached and whether the run includes initialization or checkpoint work.
- Measure an unprofiled run. Record job execution time, useful work completed, and the infrastructure signals related to your question. Repeat runs when natural variation could change the conclusion.
- Collect a focused profile. Profile only the time window or ranks needed to answer the question. Profiling changes runtime and can add data-collection overhead; that overhead depends on the tool, collection mode, and application. Compare profiled and unprofiled runs rather than assuming a fixed slowdown.
- Keep the evidence together. Save the job settings, tool and version, profile files, logs, metric time range, and result correctness with the comparison. Profile files can expose process names, paths, call stacks, or other execution details, so limit access to them.
Tools such as Linux perf may need access to hardware performance counters or process data. Their permissions are controlled by the operating system and can differ by image, kernel, and instance. Use the cluster's approved profiling setup and least-privilege access; do not apply blanket changes to ptrace or performance-counter protections just to make one command run. The Linux kernel's perf security guidance explains the access controls and the sensitivity of collected data.
Choose a tool for the question
No single profiler explains a whole cluster. Use node metrics to locate a period or resource worth investigating, then collect the application-level evidence that explains what the code was doing.
| Question | Useful starting point | What it does not establish by itself |
|---|---|---|
| Are hosts or AWS resources saturated during a job? | Amazon CloudWatch service metrics. The CloudWatch agent can add selected guest metrics and per-process measurements with its procstat plugin. | EC2 instance metrics are not source-level profiles, MPI-rank timelines, or function call stacks. Agent process metrics are measurements over time, not sampled call-path analysis. |
| Which CPU code paths or hardware events dominate on a node? | Linux perf stat for supported counters and perf record for sampled call paths; Intel VTune Profiler is another option on compatible targets. | Tool support, counters, symbols, and collection permissions depend on the processor, operating system, kernel, and build. A node-level CPU sample alone does not show cluster-wide rank imbalance. |
| Are MPI communication or synchronization limiting scaling? | An MPI-aware profiler or trace tool that supports the application's MPI implementation and launcher. NVIDIA Nsight Systems supports MPI API tracing for Open MPI and MPICH; its multi-node collection has launcher-specific setup. | A network-byte counter cannot tell how long the application waited in a collective or which rank delayed progress. Verify the tool's current MPI and launcher support before collecting. |
| Is an NVIDIA GPU workload spending time in kernels or waiting on CPU and data? | NVIDIA Nsight Systems for a host/device timeline and Nsight Compute for kernel-level analysis. | These are NVIDIA GPU tools and require compatible hardware, drivers, software, and collection settings. GPU activity alone does not prove that useful work or end-to-end job throughput improved. |
| Is shared storage or the network involved? | Correlate application or operating-system I/O evidence with EC2/EFA and, when used, FSx for Lustre metrics. | Service and device metrics describe resources or traffic, not the application call site or the job responsible for a shared resource's activity. |
For example, a CPU-focused run on a Linux node can use perf to compare hardware counters and sampled call paths:
perf stat -- ./solver input.dat
perf record -g -- ./solver input.dat
perf report
Use the profiler and event set supported by the node's architecture. Preserve symbols or debug information if you need to connect samples to source code. For remote collection, many profilers let you collect on the compute node and inspect transferred results on an analysis host; avoid running a graphical interface on a production compute node unless that is part of your approved setup.
CloudWatch has a complementary role. Default EC2 metrics such as CPUUtilization describe an instance over a time period; they do not identify which process or rank consumed that CPU. See the current EC2 metric definitions. The CloudWatch agent's procstat plugin can publish configured process measurements, but it does not produce sampled source-level profiles. See the site's HPC monitoring guide for how service, host, scheduler, GPU, EFA, and file-system signals fit together.
Profile the AWS configuration that matters
Select instance types from evidence
Shortlist instance types from the workload's measured CPU, memory, accelerator, local-storage, EBS, and network needs. Check processor architecture and software compatibility, and compare the exact features available in the Region where the job will run. AWS's instance catalog changes; its instance-type discovery guidance recommends evaluating candidates with the workload because published specifications do not replace an application test. For a broader process covering resource history, candidate sizing, and rollout, see our EC2 right-sizing guide.
Use EFA when the application can use it
Elastic Fabric Adapter (EFA) is intended for supported inter-instance communication patterns in HPC and AI/ML applications. It integrates with Libfabric and supports documented MPI and NVIDIA communication-library stacks on supported instance types. EFA is useful only when the application and software stack use the EFA path and network communication is a meaningful part of the runtime; attaching an EFA does not make every network workload faster.
Check the supported instance types, operating system, MPI version, EFA software, security-group rules, and placement before comparing runs. For OS-bypass communication, AWS requires the EFA security group to allow traffic to and from itself; this rule does not require broad public ingress. AWS recommends a cluster placement group for low-latency placement, and EFA-enabled ParallelCluster queues must stay within a single Availability Zone; EFA does not support communication across Availability Zones. See AWS's EFA overview, EFA and MPI setup guide, and ParallelCluster EFA guidance for supported stacks and setup details. Where installed, fi_info -p efa -t FI_EP_RDM can confirm that the Libfabric EFA provider is available; also verify that the application's MPI library selects it for the run.
Treat FSx for Lustre as a workload choice
Amazon FSx for Lustre is a parallel file system for supported Linux clients that need concurrent access to shared data, such as simulation inputs or checkpoints. It is not a default requirement for every HPC job. Whether it helps depends on the file system's storage class and configured capacity or throughput, client network, file sizes, access pattern, and concurrency. AWS's FSx for Lustre performance guidance describes these dependencies.
Use the AWS/FSx metrics to compare file-system and storage-target activity with the job's read and write phases; AWS lists their scope and dimensions in the FSx for Lustre metrics reference. Pair that evidence with an application or host profile before concluding that storage is the bottleneck. Choose scratch or persistent storage according to data durability and recovery needs, and test with representative files and concurrent clients. See our FSx for Windows File Server versus Lustre comparison for protocol, workload, and storage-class trade-offs.
Separate scheduler time from application time
AWS ParallelCluster, Slurm, or AWS Batch can manage queued jobs and compute capacity, but scheduler metrics answer different questions from an application profile. Record queue delay and node provisioning separately from time spent executing the solver. Scaling a compute fleet to run pending jobs does not resolve a CPU, MPI, GPU, or I/O bottleneck inside a job that is already running.
Read profiles alongside workload evidence
Compare like with like
Compare the same input, amount of work, software build, launch configuration, and measurement boundaries. For each run, retain elapsed execution time and useful work completed; use normalized throughput only when the completed work is comparable. A shorter profile sample, a CPU counter, and a CloudWatch average describe different things, so keep their units, scope, and time ranges visible.
Follow a symptom to its cause
- Adding ranks stops reducing runtime: inspect per-rank work and MPI wait or collective time. Check for uneven work before changing network settings; use EFA only if communication evidence and the deployed software path support that conclusion.
- CPU utilization is high but useful work is low: inspect sampled call paths and thread behavior for spin loops, contention, or time spent waiting. An instance-wide average can hide idle ranks alongside busy ones.
- A GPU is idle during parts of the run: use a compatible GPU timeline to see whether CPU work, data movement, synchronization, or application gaps precede kernel execution.
- Job time rises during reads or checkpoints: correlate host I/O waits with FSx or attached-storage metrics and the relevant application phase. High aggregate file-system traffic alone does not prove that this job is blocked on storage.
Test one change and measure the result
Use the profile to form a specific hypothesis, such as uneven rank placement, excessive communication, a CPU hotspot, or a serialized checkpoint. Change one relevant factor, keep the workload and measurement boundary consistent, and rerun the baseline comparison. Confirm output correctness, job runtime, useful work completed, and any resource or infrastructure costs that matter to the workload.
Do not treat a faster kernel, lower CPU percentage, or higher network rate as a successful optimization by itself. The outcome that matters is the job's correct end-to-end work at the required scale and an acceptable resource cost. Re-profile after a material change to the application, input size, MPI stack, instance architecture, or data path.
A practical profiling checklist
- State the performance question and select one representative, correct workload.
- Record software, inputs, ranks, nodes, instance and storage configuration, and separate queue delay from execution time.
- Capture an unprofiled baseline and only the CloudWatch or host metrics relevant to the question.
- Choose a CPU, MPI, GPU, or I/O tool that supports the deployed hardware and software stack; measure collection overhead and protect the resulting profile data.
- Correlate application results with AWS resource evidence, change one factor, then repeat the comparable run and check correctness, runtime, work completed, and cost.