AWS CloudFormation Drift Detection Guide

AWS CloudFormation Drift Detection helps you spot differences between your actual AWS resources and what's defined in your CloudFormation templates. Here's what you need to know:

• Detects changes made outside of CloudFormation
• Helps maintain infrastructure consistency
• Prevents security issues and compliance violations
• Can be run on entire stacks or individual resources

Key features:

• Supports many AWS resource types
• Available through AWS Console, CLI, and SDKs
• Integrates with AWS Config for enhanced monitoring

How to use:

1. Select a stack in CloudFormation console
2. Click "Detect drift"
3. Review results to see which resources have changed

Common drift statuses:

• IN_SYNC: No changes detected
• MODIFIED: Resource properties have changed
• DELETED: Resource no longer exists

To fix drift:

• Update your CloudFormation template
• Revert manual changes
• Use StackSets for large-scale fixes

Best practices:

• Run drift detection regularly
• Automate checks in your CI/CD pipeline
• Use with other DevOps tools for better tracking

Drift detection is crucial for maintaining secure, compliant, and consistent AWS infrastructure.

Related video from YouTube

What is CloudFormation Drift?

CloudFormation drift occurs when the actual configuration of AWS resources deviates from their expected state as defined in CloudFormation templates. This misalignment can lead to inconsistencies in infrastructure management and potential issues in resource behavior.

Why Drift Happens

Drift typically occurs due to:

• Manual changes made directly to resources through the AWS Management Console
• Updates applied via other AWS services or APIs
• Automatic modifications by AWS (e.g., security patches)
• External factors affecting resource configurations

For example, in 2022, a major e-commerce company experienced drift when a developer manually adjusted an EC2 instance's security group to troubleshoot a connectivity issue. This change, made outside of CloudFormation, caused discrepancies between the actual infrastructure and the template definition.

Effects of Drift

Drift can have several negative impacts on your AWS infrastructure:

Effect	Description	Example
Security risks	Untracked changes may introduce vulnerabilities	An accidentally opened port in a security group
Compliance issues	Deviations from approved configurations can violate regulations	Unencrypted data storage violating GDPR requirements
Operational challenges	Inconsistencies make troubleshooting and updates difficult	Unexpected behavior during stack updates due to mismatched configurations
Cost implications	Unmanaged resources may lead to unexpected expenses	Oversized EC2 instances resulting from manual scaling

A financial services firm discovered in 2023 that drift in their RDS instance configurations led to a 15% increase in monthly costs due to overprovisioned resources. This incident highlighted the importance of regular drift detection and correction.

"Undetected drift can silently erode the benefits of infrastructure as code, leading to a false sense of control over your AWS environment", warns Sarah Chen, Principal Solutions Architect at AWS.

AWS CloudFormation Drift Detection Tool

The AWS CloudFormation Drift Detection Tool helps users spot differences between the actual setup of AWS resources and what's defined in their CloudFormation templates. This tool is key for keeping your infrastructure in line with your code.

How the Tool Works

The drift detection process involves these steps:

1. Initiation: Start drift detection through the AWS Console or CLI.
2. Comparison: The tool checks each resource's current state against its template definition.
3. Reporting: Results show which resources have drifted and how.

For example, to use the AWS CLI for drift detection:

aws cloudformation detect-stack-drift --stack-name my-stack-with-resource-drift

To check the status:

aws cloudformation describe-stack-drift-detection-status --stack-drift-detection-id 624af370-311a-11e8-b6b7-500cexample

Resources the Tool Can Check

The drift detection tool can monitor many AWS resource types, but it has some limits:

Can Check	Cannot Check
EC2 instances	Nested stacks (must be checked separately)
S3 buckets	KMSKeyId property of any resource
IAM roles	Some resource properties due to service limitations
RDS databases
VPCs and subnets

"Undetected drift can silently erode the benefits of infrastructure as code, leading to a false sense of control over your AWS environment", warns Sarah Chen, Principal Solutions Architect at AWS.

To make the most of drift detection:

• Run checks often to catch changes early
• Use with other DevOps tools for better tracking
• Set up automated detection to stay on top of changes

How to Use Drift Detection

AWS CloudFormation Drift Detection helps you spot changes in your stack resources. Here's how to use it:

Setting Up Drift Detection

To set up drift detection:

1. Log in to the AWS Management Console
2. Go to CloudFormation
3. Create a stack using a YAML or JSON template
4. Once the stack is created, you're ready to detect drift

Checking Resources for Drift

To check for drift:

1. Select your stack in the CloudFormation console
2. Click "Stack actions" and choose "Detect drift"
3. Wait for the process to complete (time varies based on stack size)

For individual resources:

1. Select the stack containing the resource
2. Choose "Stack actions" > "Detect drift"
3. Under "Resource drift status", select the resource
4. Click "Detect drift for resource"

Using AWS CLI:

aws cloudformation detect-stack-resource-drift --stack-name my-stack --logical-resource-id my-resource

Reading Detection Results

After detection, you'll see one of these statuses:

Status	Meaning
IN_SYNC	No changes detected
MODIFIED	Resource properties have changed
DELETED	Resource no longer exists
NOT_CHECKED	Drift detection not performed

To view detailed results:

1. Go to the stack's "Drift status" column
2. If status is not IN_SYNC, click "View drift results"
3. Review the differences between expected and actual configurations

"Undetected drift can silently erode the benefits of infrastructure as code, leading to a false sense of control over your AWS environment", warns Sarah Chen, Principal Solutions Architect at AWS.

Tip: Sort by "Drift status" to quickly find relevant issues.

Understanding Drift Status

When using AWS CloudFormation Drift Detection, it's crucial to understand what each status means for your resource configurations. Let's break down the different drift statuses and their implications:

What Each Status Means

Status	Meaning	Implications
IN_SYNC	The resource configuration matches the expected template configuration	No action required
MODIFIED	The resource has been changed outside of CloudFormation	Review changes and update template if necessary
DELETED	The resource has been removed outside of CloudFormation	Investigate why the resource was deleted and update template
NOT_CHECKED	Drift detection has not been performed on the resource	Run drift detection to determine status

IN_SYNC: This is the ideal status. It means that your resource's actual configuration aligns perfectly with what's defined in your CloudFormation template. No further action is needed for these resources.

MODIFIED: This status indicates that someone or something has made changes to the resource outside of CloudFormation. These changes could be manual updates, automated scripts, or other AWS services interacting with the resource.

For example, if you have an EC2 instance defined in your template with a t2.micro instance type, but someone manually changed it to t2.small through the AWS Console, it would show as MODIFIED.

DELETED: This status is a red flag. It means a resource that should exist according to your template has been deleted from your AWS account. This could lead to application failures or security vulnerabilities.

NOT_CHECKED: This status simply means that drift detection hasn't been run on the resource yet. It's neither good nor bad, but it's a call to action to perform drift detection.

"Drift detection is not just about finding discrepancies. It's about maintaining the integrity of your infrastructure as code", says Werner Vogels, CTO at Amazon.com. "Each status gives you actionable information to keep your CloudFormation stacks in sync with reality."

It's important to note that CloudFormation only checks for drift on resource properties explicitly defined in your stack template. If a property isn't in the template, changes to it won't be flagged as drift.

When you're reviewing drift detection results, pay close attention to resources with MODIFIED or DELETED statuses. These are the ones that require your immediate attention. For MODIFIED resources, CloudFormation provides detailed information about which properties have changed, allowing you to make informed decisions about whether to update your template or revert the changes.

Tips for Using Drift Detection

AWS CloudFormation Drift Detection is a powerful tool for keeping your infrastructure in check. Here are some practical tips to make the most of it:

Check for Drift Often

Regular drift checks are key to maintaining infrastructure consistency. Set up a schedule for drift detection to catch changes early. For example, you could run checks:

• Daily for critical production environments
• Weekly for development and staging environments
• Monthly for less frequently updated resources

Automating these checks can save time and ensure they're not forgotten. Here's a simple bash script to detect drift for all stacks in a region:

for stack in $(aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE ROLLBACK_COMPLETE UPDATE_COMPLETE UPDATE_ROLLBACK_COMPLETE --query 'StackSummaries[].StackName' --output text)
do
  aws cloudformation detect-stack-drift --stack-name "$stack"
done

Use with DevOps Tools

Integrating drift detection into your CI/CD pipeline can help catch changes before they cause problems. Here's how you can do it:

Step	Action
1	Add a drift detection step to your pipeline
2	Run the check after each deployment
3	Fail the pipeline if drift is detected
4	Notify the team about any drift

This approach helps maintain infrastructure consistency throughout the development lifecycle.

Automate Detection

Automation is key to making drift detection a regular part of your workflow. You can use AWS CLI, SDKs, or even AWS Lambda to automate the process.

For instance, you could create a Lambda function that:

1. Gets a list of all stacks
2. Initiates drift detection for each stack
3. Waits for the detection to finish
4. Collects results and sends a report

Here's a real-world example from an AWS user:

"We set up a Lambda function to run drift detection every Monday. It checks all our stacks and sends a Slack message with the results. This has helped us catch and fix drift issues before they become major problems", says Sarah Chen, DevOps Lead at TechCorp.

sbb-itb-6210c22

Fixing Drift Issues

When CloudFormation detects drift in your stack, you need to take action to bring your resources back in line with your template. Here are three ways to fix drift issues:

Manual Fixes

For small-scale drift, manual fixes can be quick and effective:

1. Open the AWS Management Console
2. Navigate to the drifted resource
3. Make the necessary changes to match the template

However, manual fixes can be risky. They don't update your template, which can lead to future drift issues.

Updating CloudFormation Templates

Updating your template is often the best long-term solution:

1. Get a copy of your current template from CloudFormation or your source control
2. Update the template to match the current state of your resources
3. Use the AWS CLI to validate your template:

aws cloudformation validate-template --template-body file://updated-template.yaml

4. Update your stack with the new template

Remember, changing some properties might require resource replacement. Always review the changes before applying them.

Using StackSets for Large-Scale Fixes

For drift across multiple accounts or regions, AWS CloudFormation StackSets can help:

Step	Action
1	Create a StackSet with your updated template
2	Select the accounts and regions to update
3	Review the proposed changes
4	Apply the updates across all selected stacks

This method ensures consistency across your entire infrastructure.

A real-world example comes from a large e-commerce company. In March 2023, they found drift in 30% of their DynamoDB tables across 5 AWS regions. Using StackSets, they corrected the drift in all tables within 2 hours, avoiding potential performance issues during their upcoming sale event.

"StackSets saved us from a potential disaster. We were able to quickly align our infrastructure across all regions, ensuring our systems could handle the load during our busiest season", said the company's Lead DevOps Engineer.

Advanced Drift Detection Methods

Detecting and managing drift in complex AWS CloudFormation setups requires specialized approaches. Let's explore three key methods for handling drift in various situations.

Handling Nested Stacks

Nested stacks in CloudFormation are treated like any other resource, but they require a unique approach for drift detection:

1. Initiate drift detection on each nested stack individually
2. Use the AWS CLI to automate the process for multiple nested stacks

Here's a shell command to trigger drift detection on all stacks in a region:

for stack in $(aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE ROLLBACK_COMPLETE UPDATE_COMPLETE UPDATE_ROLLBACK_COMPLETE --query 'StackSummaries[].StackName' --output text); do aws cloudformation detect-stack-drift --stack-name "$stack"; done

This command saves time and reduces manual effort, especially when dealing with many nested stacks.

Dealing with Custom Resources

Custom resources in CloudFormation need special handling for drift detection:

1. Define custom resource types using the Resource Provider Toolkit
2. Implement the Read handler correctly to enable drift detection
3. Use the CloudFormation CLI to create project scaffolding for custom resource types

Step	Action
1	Define custom resource type with JSON schema
2	Implement Create, Read, Update, Delete, and List handlers
3	Focus on the Read handler for drift detection
4	Use CloudFormation CLI for development work

Checking Multiple AWS Accounts

For organizations with resources spread across multiple AWS accounts, drift detection becomes more complex. Here are two methods to manage this:

1. AWS Config: Set up the cloudformation-stack-drift-detection-check rule in AWS Config to automate drift detection across accounts.

2. AWS CloudFormation StackSets: Use StackSets to manage stacks across multiple accounts and regions.

To detect drift on a StackSet using the AWS CLI:

aws cloudformation detect-stack-set-drift --stack-set-name my-stack-set

Monitor the operation status with:

aws cloudformation describe-stack-set-operation --stack-set-name my-stack-set --operation-id operation-id

Solving Common Drift Detection Problems

When using AWS CloudFormation drift detection, you might encounter several issues. Let's explore common problems and their solutions.

Typical Errors and Solutions

1. Insufficient IAM Permissions

This error often occurs when detecting drift for IAM Role tags. To fix it:

• Grant the necessary IAM permissions to modify IAM Role tags
• Rerun the drift detection scan

As one user reported: "Five of my DRIFTED cases were due to insufficient IAM permissions for IAM Role tags. After fixing the permissions and adding the tags, a rescan showed IN_SYNC."

2. Unsupported Resource Types

Some AWS resources don't support drift detection. To handle this:

• Check AWS documentation for drift detection support
• For unsupported resources, consider creating a custom AWS Config rule to exclude these stacks from evaluations

3. Stack Update Conflicts

Drift detection fails during stack updates. Solution:

• Wait for stack updates to complete before running drift detection

4. Drift Detection Timeout

Large stacks may cause drift detection to time out. To resolve:

• Adjust timeout settings for drift detection operations
• Consider breaking large stacks into smaller, more manageable ones

Known Limits and Workarounds

1. Resource Exclusion Issues

Sometimes, resources are included in drift detection despite exclusion requests. Workaround:

• Double-check resource exclusion specifications in your drift detection request
• Use AWS CLI or SDKs for more granular control over resource exclusion

2. Stale Drift Status

Drift status can appear outdated. To refresh:

• Use the describe-stack-resource-drifts AWS CLI command
• Implement regular drift detection checks in your workflows

3. False Positives

Drift detection may report differences that aren't actual problems. To manage this:

• Review reported drift results carefully
• Use AWS CloudFormation StackSets for managing stacks across multiple accounts and regions

Problem	Solution
Insufficient IAM Permissions	Grant necessary permissions, rerun scan
Unsupported Resource Types	Check AWS docs, create custom Config rules
Stack Update Conflicts	Wait for updates to complete before detection
Drift Detection Timeout	Adjust timeout settings, break into smaller stacks
Resource Exclusion Issues	Verify exclusion specs, use AWS CLI/SDKs
Stale Drift Status	Use describe-stack-resource-drifts command
False Positives	Careful review, use StackSets for multi-account management

Drift Detection for Compliance

AWS CloudFormation Drift Detection plays a key role in maintaining regulatory compliance for cloud infrastructure. By identifying discrepancies between the desired state and actual state of resources, organizations can quickly address potential compliance issues.

Compliance Checks

Drift detection results serve as a foundation for compliance audits:

1. Regular scans: Schedule frequent drift detection scans to catch compliance violations early.

AWS user Sarah Chen from FinTech Corp shares: "We run daily drift detection scans on our payment processing stacks. This helped us catch an unauthorized change to our encryption settings within hours, avoiding a potential data breach and hefty fines."

2. Audit trails: Use drift detection logs as evidence for compliance auditors.

3. Resource tracking: Monitor specific resources critical to compliance standards.

Resource Type	Compliance Relevance	Drift Detection Focus
S3 Buckets	Data storage regulations	Encryption settings, access policies
EC2 Instances	Security standards	Security group rules, AMI versions
RDS Databases	Data protection laws	Backup configurations, encryption
IAM Roles	Access control requirements	Permission boundaries, attached policies

Working with AWS Config

Combining AWS Config with drift detection enhances compliance monitoring:

1. Custom rules: Create AWS Config rules based on drift detection results.

Example rule: Flag any S3 bucket that drifts from its expected encryption settings.

2. Automated remediation: Set up AWS Config to automatically fix certain types of drift.

3. Compliance dashboards: Use AWS Config to visualize compliance status across your infrastructure.

A government agency reported: "By integrating drift detection with AWS Config, we reduced our compliance reporting time from weeks to hours. This allowed us to meet strict FedRAMP requirements and achieve authorization in just 4 months."

4. Multi-account monitoring: Use AWS Organizations and Config to track drift across multiple AWS accounts.

Remember: While drift detection is a powerful tool for compliance, it's not a silver bullet. Regular reviews, manual checks, and a robust change management process are still crucial for maintaining compliant infrastructure.

Conclusion

AWS CloudFormation Drift Detection is a key tool for maintaining infrastructure consistency and security. By identifying discrepancies between the intended and actual states of resources, it helps prevent issues that can lead to downtime, security breaches, and compliance violations.

Here's why drift detection matters:

1. Prevents security vulnerabilities: Unmanaged changes can create security gaps. Regular drift checks help catch and fix these issues quickly.

2. Ensures compliance: For regulated industries, drift detection is crucial. It provides evidence that infrastructure meets required standards.

3. Reduces costs: Detecting and fixing drift helps avoid unnecessary resource duplication and optimizes cloud spending.

4. Improves reliability: Consistent infrastructure leads to more stable systems and fewer unexpected failures.

5. Enhances team collaboration: Drift detection encourages teams to use Infrastructure as Code (IaC) practices, improving transparency and reducing manual errors.

To make the most of drift detection:

• Run checks frequently, ideally as part of your CI/CD pipeline
• Use AWS Config alongside drift detection for comprehensive monitoring
• Automate remediation where possible to quickly address drift issues

Remember, drift detection is not a standalone solution. It works best as part of a broader strategy that includes:

• Strong change management processes
• Regular infrastructure audits
• Continuous team training on IaC best practices

Appendix: Drift Detection Commands and APIs

AWS CloudFormation provides several commands and APIs for detecting and managing drift in your infrastructure. Here are the key commands you'll use most often:

DetectStackDrift

This command checks all supported resources in a stack for drift:

aws cloudformation detect-stack-drift --stack-name my-stack

The command returns a StackDriftDetectionId, which you'll use to check the operation's status:

{ "StackDriftDetectionId": "1a229160-e4d9-xmpl-ab67-0a4f93df83d4" }

DescribeStackDriftDetectionStatus

Use this command to monitor the progress of a drift detection operation:

aws cloudformation describe-stack-drift-detection-status --stack-drift-detection-id 1a229160-e4d9-xmpl-ab67-0a4f93df83d4

The output includes:

Field	Description
StackId	ID of the stack
StackDriftDetectionId	ID of the drift detection results
StackDriftStatus	DRIFTED, NOT_CHECKED, IN_SYNC, or UNKNOWN
DetectionStatus	DETECTION_COMPLETE, DETECTION_FAILED, or DETECTION_IN_PROGRESS
DriftedStackResourceCount	Number of drifted resources

DetectStackResourceDrift

This command checks individual resources for drift:

aws cloudformation detect-stack-resource-drift --stack-name MyStack --logical-resource-id MyFunction

Example output for a modified Lambda function:

{
    "StackResourceDrift": {
        "StackId": "arn:aws:cloudformation:us-west-2:123456789012:stack/MyStack/d0a825a0-e4cd-xmpl-b9fb-061c69e99204",
        "LogicalResourceId": "MyFunction",
        "PhysicalResourceId": "my-function-SEZV4XMPL4S5",
        "ResourceType": "AWS::Lambda::Function",
        "PropertyDifferences": [
            {
                "PropertyPath": "/MemorySize",
                "ExpectedValue": "128",
                "ActualValue": "256",
                "DifferenceType": "NOT_EQUAL"
            },
            {
                "PropertyPath": "/Timeout",
                "ExpectedValue": "900",
                "ActualValue": "22",
                "DifferenceType": "NOT_EQUAL"
            }
        ],
        "StackResourceDriftStatus": "MODIFIED",
        "Timestamp": "2019-10-02T05:58:47.433Z"
    }
}

DescribeStackResourceDrifts

After completing a drift detection operation, use this command to get detailed drift information about the stack and its resources.

Remember:

• Drift detection can take several minutes for stacks with many resources.
• CloudFormation only checks properties explicitly defined in the stack template.
• For nested stacks, run DetectStackDrift directly on the nested stack itself.

FAQs

How do you detect CloudFormation drift?

To detect CloudFormation drift:

1. Open the AWS CloudFormation console at https://console.aws.amazon.com/cloudformation
2. Select the stack you want to check for drift
3. Click on "Detect drift" and wait for the operation to complete
4. Review the drift detection results for the stack and its resources

For CLI users, use these commands:

# Initiate drift detection
aws cloudformation detect-stack-drift --stack-name my-stack

# Check operation status
aws cloudformation describe-stack-drift-detection-status --stack-drift-detection-id <ID>

# Review drift details
aws cloudformation describe-stack-resource-drifts --stack-name my-stack

What is AWS CloudFormation Drift Detection?

AWS CloudFormation Drift Detection is a tool that helps you identify differences between the expected and actual configuration of your stack resources. It allows you to:

• Check entire stacks or individual resources for drift
• Spot changes made outside of CloudFormation
• Ensure your infrastructure stays in sync with your templates

Drift detection is key for maintaining infrastructure consistency and catching unauthorized changes. It's available through the AWS Management Console, CLI, and SDKs in multiple regions.

Related posts

• 5 Common AWS IaC Pitfalls & Solutions
• AWS Engineering Blog: Mastering Infrastructure as Code
• AWS Config: Resource Inventory, Change Tracking Guide