Contributors:

• Sasha Tulchinskiy, Senior Solutions Architect, Deloitte;
• Ajith Joseph, Manager, Deloitte;
• Tony Witherspoon, Principal, Deloitte;

Introduction

We have noticed that there is a lot of information on how to provision resources on AWS quickly and efficiently. However, there are times when you need to terminate or remove them us as efficiently and quickly as you created them?

Our team just wrapped up a successful proof of concept (POC) leveraging AWS, which resulted in creation of many AWS resources across multiple regions. The monthly invoices started getting quite large and we wanted to reduce our expenses quickly. As you can see in the screenshot below, we had over 400 resources created this short POC. Now imagine if this was a large enterprise-scale environment!

In researching our options to reduce the costs, we identified two distinct strategies for terminating unneeded billable resources:

1. Delete AWS resources and keep using the same AWS account
2. Re-provision the AWS account and start clean

Disclaimer: all experiments below were conducted in a non-production environment with a full anticipation for it to be broken/unusable as a result. Please proceed with caution!

Deleting expensive AWS resources by rolling back automation steps

We have checked per-service reports in AWS Billing console and created a prioritized list of the most expensive resources to remove first. These resources were launched through AWS ServiceCatalog and CloudFormation, so deletion process would involve termination of Catalog products and CFN stacks.

We started looking for a tool that may make the process more efficient. There are a couple of open source products that are designed to help you with cleanup tasks and, perhaps coincidentally, have “nuke” in the name – cloud-nuke and aws-nuke (both with MIT License). This document is not intended to provide comparison between these tools, but rather an observation of cloud-nuke capabilities for our scenario.

Dropping a Nuke

Cloud Nuke can be easily installed and executed directly from the Cloud (for example, AWS Workspace or AWS CloudShell as we did).

After launching Cloud-Nuke in a default configuration and waiting about 30 minutes for completion, we observed a few features and challenges with the tool:

• It will try to terminate resources of all supported types in all regions (except GovCloud), including those that you may want to keep. Leverage the config file option to exclude any resources you want to preserve:

• The tool does not support some AWS resource types, including some of the most expensive:

• Managed Active Directory
• Workspaces
• CloudHSM (very expensive!)
• Route53, etc.

• It is fortunate that AWS CloudShell is not disabled either, which is helpful to continue working in the account.

• Resources with Termination Protection set to "true" are skipped – unless you change the setting before running the tool

• Same for those that cannot be deleted due to "resources in use" (for example VPCs with existing VPC peering and VPN Connections). However, Cloud-Nuke is smart enough to wait for some resources to be deleted before proceeding further.

• It fails when a resource is protected from deletion by AWS Organizations Service Control Policies (for example, for IAM roles/CloudWatch Log Groups/Lambda functions created in member accounts by AWS ControlTower). We had to add AWS Lambda exclusion to the next Cloud-Nuke run, knowing that there is no cost of just having functions without running them!

$ cloud-nuke aws --exclude-resource-type cloudwatch-loggroup --exclude-resource-type iam --exclude-resource-type lambda

It produced the following output:

• As a result, we had to painstakingly disable Termination Protection for impacted resources, re-run Cloud-Nuke several times through trial and error, and still going back to AWS Service Catalog and CloudFormation for Stack termination. Note that it takes about 20-30 min to AWS Config to reflect deletion of resources.

Re-provisioning AWS accounts

At last, you can consider AWS account termination through AWS Organization and, according to AWS Documentation, on-demand billing will stop immediately. However, if you change your mind later and decide to recover the account, use of all resources that were not terminated will be billed even for the time the account was "suspended"!

Summary

From this experiment, we learned that proper AWS resource termination process should include a combination of:

1. Deletion of the most expensive resources through reversing automation steps to include disabling Termination Protection,
2. Inventory review to confirm deletion of the most expensive resources and dependencies
3. Account termination through AWS ControlTower.

While we did not have a chance to try AWS-Nuke tool, reading through GitHub documentation did not indicate any material difference in dealing with AWS resource dependencies and termination protection.

Happy Cloud Sailing!