Published on · Updated on
Reworked failure handling, queue and event retry semantics, regional recovery limits, and resilience testing guidance.
A request that crosses several AWS services can fail even when no service is completely down. A dependency may respond slowly, accept a write but lose the response, throttle only part of its traffic, or recover while queued work is still building up. Resilience depends on what each boundary does with that failure: wait, retry, queue, reject, fail over, or record work for recovery.
Start by defining the failure you need to survive and what successful recovery means for each user operation. The recovery time objective (RTO) is the target time to restore service. The recovery point objective (RPO) sets how much recent data loss the workload can tolerate, usually measured in time. Then make each dependency's retry and data behavior explicit. AWS services provide useful building blocks, but a queue, replica, health check, or retry policy does not by itself make the whole workflow recoverable.
Availability Zones (AZs) are separate locations within a Region, designed to limit the impact of some local infrastructure failures. Distributing compute and data across AZs can help a workload continue through an AZ impairment, but only when the affected service and your application are configured to use the remaining capacity. Managed services differ: some replicate across AZs by default, while others require a multi-AZ option or application-level replication. Check the behavior of each service in the path rather than assuming that every resource is redundant. The AWS Well-Architected guidance on multiple locations describes these boundaries.
Multi-AZ placement also does not protect against every failure. A bad deployment, incorrect policy, exhausted quota, corrupted data, or shared dependency can affect all AZs. Design failure isolation around the actual fault: separate workloads or tenants where a single noisy neighbor could exhaust capacity, keep independent dependencies from sharing the same constrained resource, and make a degraded mode explicit where the business can tolerate one.
A gray failure is a partial or ambiguous fault: requests might time out intermittently, latency may rise for one operation, or only one downstream path may fail. A basic “healthy/unhealthy” check can miss this. Watch user-facing success and latency alongside dependency errors, throttling, timeouts, retry counts, and resource saturation. Trace a request across service boundaries so an apparently healthy front end does not hide a failing dependency.
Set a deadline for the complete operation and give each downstream call only the time remaining in that budget. Include connection establishment and response reading in the timeout. A client-side timeout should free the caller's waiting resources and give it an explicit timeout or degraded result; the remote service may still be processing the request or may have committed a state change, so completion can remain unknown. Allowing a caller, connection, or worker to wait indefinitely can tie up capacity and spread the failure upstream. AWS's client timeout guidance recommends configuring and testing connection and request timeouts for the workload.
Retry only errors that are plausibly temporary, such as throttling or a transient network error. Do not retry permanent failures such as invalid input or missing permissions. Bound retries by both attempt count and elapsed time, use capped exponential backoff with jitter, and choose one layer to own retries for a given call. Retries at several layers multiply load during an outage. AWS's retry guidance recommends testing the chosen policy and checking whether an SDK already retries before adding another retry loop.
A timeout does not tell the caller whether a state-changing request completed. The dependency may have committed a write before its response was lost. Make retried mutations idempotent: assign a stable key to the logical operation, persist its outcome, and return that outcome for duplicate requests. Pass the same key to downstream services that support it. This prevents common duplicate effects, but it does not create a general exactly-once guarantee across independent systems. See AWS guidance on idempotent mutating operations.
Retries are for brief, recoverable faults. If a dependency remains unhealthy, an application-level circuit breaker can stop calls temporarily, fail quickly or use an intentional fallback, and allow limited probes to check for recovery. A bulkhead can cap the resources or concurrency devoted to one dependency, tenant, or workload so its failure does not consume every worker. These are design patterns that your application or chosen library must implement and operate; AWS does not install a universal circuit breaker or bulkhead across service calls. AWS describes the circuit breaker pattern and bulkhead architectures as workload design choices.
A queue can let a producer continue while a consumer is temporarily unavailable, but it converts immediate failure into delayed work. Size and scale consumers for recovery as well as normal traffic, define how long queued work may wait, and make processing safe when a message is delivered more than once. A queue is not a backup: messages can expire, be moved to a dead-letter queue, or be deleted after processing according to its configuration.
For an Amazon SQS event source mapping, Lambda polls the queue and invokes the function with a batch. By default, if the function reports an error, the whole batch becomes visible again after the visibility timeout, including records the function already handled. Enable the event source mapping's ReportBatchItemFailures response type and return the identifiers of failed records to retry only those records. The handler must report failures accurately; successful side effects still need idempotency because messages can be delivered again. See the Lambda SQS error-handling guide.
A dead-letter queue (DLQ) is a separate queue that you create and configure in the source queue's redrive policy. SQS moves a message there when it reaches the configured maximum receive count without being deleted; it does not diagnose or repair the message. Alarm on failed processing and DLQ depth, inspect and fix the cause, then decide whether and how to redrive the message. Choose a receive count, visibility timeout, and retention period that fit the processing time and recovery policy. For FIFO queues, stop a batch after the first failure and report the failed and unprocessed records when preserving order matters. A DLQ can break FIFO ordering, so decide whether isolating a poison message is acceptable for that workflow. See the SQS dead-letter queue guidance.
Asynchronous Lambda invocation has a different boundary. Lambda accepts an event into its managed queue before running the function, so an accepted invocation response does not mean that the handler completed successfully. Lambda retries function errors according to its asynchronous invocation settings; events can eventually expire or be discarded. Configure the event age and retry limit for the workload, and add a Lambda dead-letter queue or on-failure destination if failed events must be retained. These controls are separate from an SQS event source mapping's queue redrive policy. See Lambda's asynchronous error behavior and its configurable failure handling.
An EventBridge rule has its own retry policy for delivery to a target and can send undelivered events to a configured SQS DLQ. When the target is a Lambda function invoked asynchronously, EventBridge's delivery retry ends once Lambda accepts the event; a later function error is handled by Lambda's asynchronous retry and failure destination. These DLQs therefore capture different failures. Configure and monitor the one appropriate to each boundary, and do not assume an event was processed merely because delivery succeeded. See AWS's guides to EventBridge target retries and EventBridge target DLQs.
Availability Zones within one Region can mitigate some AZ-level failures. A separate Region is a recovery boundary for a workload that must withstand a regional outage, but it adds deployment, data, security, routing, and operational work.
Choose a second Region from the workload's RTO and RPO, data residency rules, and recovery cost—not as a blanket requirement. The AWS Well-Architected guidance on using multiple locations describes how these fault boundaries differ.
Replication is not the same as backup. A replica may receive an accidental deletion or corrupted write; asynchronous replication may also leave the recovery Region behind the latest acknowledged write. Understand each data service's replication and promotion behavior, and keep recoverable backups or point-in-time recovery in the recovery plan. AWS's disaster recovery guidance explicitly calls for backups as well as replication where appropriate.
Recovery requires more than a database replica. Prepare and test application artifacts, configuration, IAM and resource policies, secrets, encryption keys, certificates, networking, service quotas, and monitoring in the recovery Region. Route 53 health checks and failover routing can direct new DNS queries to a healthy endpoint, but resolver caches honor the record TTL, so clients may continue using an old answer. A DNS change is not an instantaneous switch or proof that the application and its data are ready.
Backup and restore, pilot light, warm standby, and active-active designs keep different resources ready and have different recovery costs and times. Their actual RTO and RPO depend on the whole workload and its runbooks; service-level examples are not guarantees for your application. For a focused comparison, see AWS DR: Pilot Light vs Warm Standby.
Global data services can make selected records available in more than one Region, but they do not remove application-level choices about concurrent writes, conflict handling, replication delay, or recovery from bad data. Decide which Region can accept writes during failover, how the application detects replication lag, and how it reconciles state before failback. Use each database's current documentation for its precise consistency and promotion guarantees; do not infer them from the phrase “global table” or “global database.”
Write a runbook for the full user operation, not just the traffic switch. Define who or what declares a failure, which health signals trigger action, how operators prevent two Regions from accepting conflicting writes, how in-flight requests and queued work are handled, and how the recovery site is validated before it receives production traffic. Automated failover can reduce detection and response time, but a false positive can cause avoidable downtime or data loss; require health checks that reflect application readiness and test the decision path.
After failover, verify how the selected data service handles replication and promotion. Do not direct writes to both Regions unless the application and data service are designed to reconcile concurrent changes.
Confirm that dependencies are healthy, required data is current, and representative application transactions succeed before returning production traffic.
Document the authority to begin failback, data checks, traffic steps, and a way to stop or reverse the procedure if health degrades. A recovery script alone cannot decide which copy is correct.
Change routing only after the recovery Region is ready. With DNS-based routing, consider resolver caching and record TTL; clients may keep using a cached answer while it remains valid.
During failover and failback, monitor customer-visible success and latency, data replication state, errors, and queue backlogs. Keep operators informed until the application is stable and in-flight work has drained or been reconciled.
A resilient Direct Connect design protects the hybrid network path; it does not by itself protect an application or provide regional disaster recovery. Where the workload needs it, use connections with independent devices and paths, and consider separate Direct Connect locations for protection against a location failure. The Direct Connect Resiliency Toolkit helps design and test connection redundancy.
A Site-to-Site VPN can be a backup path only if routing, capacity, firewall behavior, and failover are configured and tested. A secondary path that has never carried production-like traffic may fail because of a route, capacity, or policy mismatch.
Payments make the timeout problem concrete: a processor may authorize a charge even though the caller never receives its response. Retry the same logical payment with the same idempotency key when the processor supports it; do not create a fresh charge request just because the first response was lost. Persist the order and payment states so the application can reconcile an unknown outcome. If a later workflow step fails, a refund or void is a compensating business action, not a rollback that erases the original transaction. A saga can coordinate such steps, but every action and compensation still needs defined retry and duplicate-handling behavior.
Measure the outcome customers care about, then inspect the service signals that explain it. Correlate request and event identifiers across logs and traces. Track end-to-end latency and success rate, plus dependency timeouts and retries, Lambda errors and throttles, queue age and backlog, DLQ depth, EventBridge delivery failures, and replication lag where available. Page on signals tied to the service objective and give the on-call engineer enough context to find the failing boundary; a healthy component metric alone cannot prove that the complete operation works.
Exercise recovery paths before an incident. Test duplicate delivery, poison messages, downstream timeouts, throttling, AZ impairment, Region failover and failback, and restoration from backup. For AWS resources supported by it, AWS Fault Injection Service (FIS) can run controlled experiments with CloudWatch alarm stop conditions. FIS acts on real resources, so start in a contained environment, define steady-state and stop conditions, and verify both user-visible behavior and recovery objectives. The internal AWS FIS testing guide provides a further introduction.