blog/source/_posts/cloud-control-config.md

title	excerpt	date	tags
Managing inventory: AWS Cloud Control vs Config	List AWS resources account-level and organisation-level	2021-10-08	aws

AWS announced a new API called Cloud Control that provides a standard sets of APIs to manage AWS resources. Imagine running aws cloudcontrol create-resource to launch EC2 and Lambda, instead of using aws ec2 run-instances and aws lambda create-function.

Aside from CRUD operations, it also supports List operation to discover all deployed resources filtered by a specific resource type (e.g. AWS::ECS::Cluster). When I first read the announcement, I wonder how it compares to {% post_link aws-config 'AWS Config' %}, a feature I'm actively using mainly for security audit, but it could also perform inventory task.

Since Cloud Control is a recent feature, the latest library is required. For Python library, I ran pip install boto3 --upgrade to update it to version xxx. Then, I created a minimal Python script to test out Cloud Control's ListResources.

#!/usr/bin/env python

# ./cloud-control.py --profile profile-name --region region-name

from argparse import ArgumentParser
import boto3
from botocore.config import Config
from itertools import count
from json import dump, loads

parser = ArgumentParser(description = 'Find the latest AMIs.')
parser.add_argument('--profile', '-p',
  help = 'AWS profile name. Parsed from ~/.aws/config (SSO) or credentials (API key).',
  required = True)
parser.add_argument('--region', '-r',
  help = 'AWS Region, e.g. us-east-1',
  required = True)
args = parser.parse_args()
profile = args.profile
region = args.region

session = boto3.session.Session(profile_name = profile)
my_config = Config(region_name = region)
client = session.client('cloudcontrol', config = my_config)

results = []
response = {}

for i in count():
  # https://docs.aws.amazon.com/cloudcontrolapi/latest/APIReference/API_ListResources.html
  params = {
    # https://docs.aws.amazon.com/cloudcontrolapi/latest/userguide/supported-resources.html
    'TypeName': 'AWS::EC2::FlowLog'
  }
  if i == 0 or 'NextToken' in response:
    if 'NextToken' in response:
      params['NextToken'] = response['NextToken']
    response = client.list_resources(**params)
    results.extend(response['ResourceDescriptions'])
  else:
    break

prop_list = []

# Extract properties only
for ele in results:
  prop_list.append(loads(ele['Properties']))

if len(prop_list) >= 1:
  with open('cloud-control.json', 'w') as w:
    # Save the first dictionary only
    dump(dict(sorted(prop_list[0].items())), w, indent = 2)

In the first draft of the script, I noticed that the API doesn't support AWS::EC2::Instance yet. It took me a while to troubleshoot until I found this list of supported resources. The error wasn't very helpful, e.g. "Resource type AWS::EC2::Instance does not support LIST action". It's more straightforward to just say "Resource type xxx does not support Cloud Control yet".

The announcement did mention not all resources are supported, but I didn't expect AWS' bread and butter are unsupported, including AWS::S3::Bucket. I'm sure these resources will be supported eventually, it's just that support of new products are prioritised at the moment as implied from the announcement, "It will support new AWS resources typically on the day of launch".

I tested on AWS::EC2::PrefixList, instead of the currently unsupported AWS::EC2::Instance. It worked fine, the output syntax is exactly what the documentation outlines. To compare it to Config, I created another equivalent script.

#!/usr/bin/env python

# ./aws-config.py --profile profile-name --account-id {account-id} --region region-name

from argparse import ArgumentParser
import boto3
from botocore.config import Config
from itertools import count
from json import dump, loads

parser = ArgumentParser(description = 'Find the latest AMIs.')
parser.add_argument('--profile', '-p',
  help = 'AWS profile name. Parsed from ~/.aws/config (SSO) or credentials (API key).',
  required = True)
parser.add_argument('--account-id', '-a',
  help = 'AWS account ID. See ~/.aws/config if SSO is used.',
  required = True,
  type = str)
parser.add_argument('--region', '-r',
  help = 'AWS Region, e.g. us-east-1',
  required = True)
args = parser.parse_args()
profile = args.profile
account_id = args.account_id
region = args.region

session = boto3.session.Session(profile_name = profile)
my_config = Config(region_name = region)
client = session.client('config', config = my_config)

results = []
response = {}

for i in count():
  params = {
    'Expression': "SELECT configuration WHERE resourceType = 'AWS::EC2::FlowLog'" \
      f" AND accountId = '{account_id}'" \
      f" AND awsRegion = '{region}'",
    'ConfigurationAggregatorName': 'ConfigAggregator' # may need to update
  }
  if i == 0 or 'NextToken' in response:
    if 'NextToken' in response:
      params['NextToken'] = response['NextToken']
    response = client.select_aggregate_resource_config(**params)
    results.extend(response['Results'])
  else:
    break


conf_list = []

# Extract configuration only
for ele in results:
  conf_list.append(loads(ele).get('configuration', {}))

if len(conf_list) >= 1:
  with open('aws-config.json', 'w') as w:
    # Save the first dictionary only
    dump(dict(sorted(conf_list[0].items())), w, indent = 2)

Before I get to the output comparison, notice the accountId and awsRegion filters I used in the SQL statement. It's necessary because I'm using an aggregator that collects data from all accounts and regions in an AWS Organization (which have AWS Config enabled). Like most other AWS APIs, Cloud Control only works on a combination of account and region. If you want discover resources in 5 combinations of account and region, that'll requires 5 API calls, in contrast to just one API call via Config's aggregator.

Here is the output of Cloud Control:

{
  "DeliverLogsPermissionArn": String,
  "Id": String,
  "LogDestination": String,
  "LogDestinationType": String,
  "LogFormat": String,
  "LogGroupName": String,
  "MaxAggregationInterval": Integer,
  "ResourceId": String,
  "ResourceType": String,
  "Tags": [ Tag, ... ],
  "TrafficType": String
}

Config:

{
  "creationTime": Float,
  "deliverLogsPermissionArn": String,
  "deliverLogsStatus": String,
  "flowLogId": String,
  "flowLogStatus": String,
  "logDestination": String,
  "logDestinationType": String,
  "logFormat": String,
  "logGroupName": String,
  "maxAggregationInterval": Float,
  "resourceId": String,
  "tags": [ Tag, ... ],
  "trafficType": String
}

Syntax used by CloudFormation template:

{
  "DeliverLogsPermissionArn" : String,
  "LogDestination" : String,
  "LogDestinationType" : String,
  "LogFormat" : String,
  "LogGroupName" : String,
  "MaxAggregationInterval" : Integer,
  "ResourceId" : String,
  "ResourceType" : String,
  "Tags" : [ Tag, ... ],
  "TrafficType" : String
}