Running CloudFormation Drift Detection on All Your Stacks

How to Drift

In typical AWS fashion, the feature exists but it’s up to us to integrate it ourselves. We can click manually on each stack and hit “Detect Drift” to run it:

Then once it’s done, you have to click through each stack to see the results again. This gets old as soon as you have more than one stack.

In an ideal world, I’d have it run automatically on every ready stack daily and get alerts to any new drift. For example, this could be done with a Lambda function and CloudWatch Events.

But for now, let’s look at a way of running drift detection on every stack without getting RSI from all that clicking.

Fire up that AWS CLI

Update (2019-07-12) As pointed out by jeshan on reddit, it's also possible to automate this with AWS Config with the cloudformation-stack-drift-detection-check rule.

First, trigger drift detection on each stack in the current region with this shell one-liner:

for stack in $(aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE ROLLBACK_COMPLETE UPDATE_COMPLETE UPDATE_ROLLBACK_COMPLETE --query 'StackSummaries[].StackName' --output text); do aws cloudformation detect-stack-drift --stack-name "$stack"; done

To unpack it:

• We start with a shell for loop. This uses the keywords for, in, do, and done.
• The loop goes over the output of the aws cloudformation list-stacks command. We filter them with --stack-status-filter to only the stacks in “green” states (yes we have to list all these states explicitly…). We then reduce the output to a single line of space separated stack names using --query, which filters the data with JMESPath, and --output text, which uses plain text rather than JSON output.
• The shell’s for loop splits the list-stacks output by whitespace and runs the loop body for each item set as the $stack variable.
• We run aws cloudformation detect-stack-drift in the loop body, templated with the $stack variable substitution.

I’ve tested this on ZSH and Bash with two AWS accounts, but this might break with non-alphanumeric stack names, some shells, or some shell configurations.

When you run it, you’ll see output with a result from each call to detect-stack-drift, for example:

{
    "StackDriftDetectionId": "00000000-0000-0000-0000-000000000000"
}
{
    "StackDriftDetectionId": "00000000-0000-0000-0000-000000000001"
}

Second, twiddle your thumbs a bit. Maybe make a cup of tea, or coffee. Drift detection takes a little while, and may be internally limited if you’re doing it on a lot of stacks.

Third and finally, let’s list the stack names and results in a table with the CLI. This is much more convenient than the web console, which doesn’t show the drift status on the list view, so you have to click through each damn one. Also we can sort them by name, so much easier!

Run this single command:

aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE ROLLBACK_COMPLETE UPDATE_COMPLETE UPDATE_ROLLBACK_COMPLETE --query 'sort_by(StackSummaries, &StackName)[*].[StackName, StackStatus, DriftInformation.StackDriftStatus, DriftInformation.LastCheckTimestamp]' --output table

This will give you output that looks like this:

----------------------------------------------------------------------------------
|                                   ListStacks                                   |
+---------------------+------------------+----------+----------------------------+
|  example-com        |  UPDATE_COMPLETE |  IN_SYNC |  2019-05-30T19:01:26.528Z  |
|  example-org        |  UPDATE_COMPLETE |  IN_SYNC |  2019-05-30T19:01:22.004Z  |
|  do-not-delete      |  UPDATE_COMPLETE |  IN_SYNC |  2019-05-30T19:01:23.157Z  |
|  illuminati-funding |  UPDATE_COMPLETE |  IN_SYNC |  2019-05-30T19:01:13.454Z  |
+---------------------+------------------+----------+----------------------------+

If some of the stacks haven’t completed drift detection yet as per the third and fourth columns, you can just rerun this listing until they have.

False positives

I’ve found a couple cases with my current modestly-sized client where false positive stack drift is detected:

• Tags added automatically by AWS being reported as drift, e.g. when using a launch template, EC2 adds the tags aws:ec2launchtemplate:id and aws:ec2launchtemplate:version
• Load balancer listener rules not pulling every detail back in ‘Actual’ e.g. a redirect rule being picked up by drift detection, but not its RedirectConfig

It seems there’s a little way to go on the feature. That said, false positives are better than false negatives here.

If you want to build alerting around drift, you might want to add in a way to ignore false positives.

Fin

Enjoy stopping drift,

—Adam