post: cloud control

source/_posts/cloud-control-config.md

@@ -0,0 +1,193 @@
+---
+title: 'Managing inventory: AWS Cloud Control vs Config'
+excerpt: List AWS resources account-level and organisation-level
+date: 2021-10-08
+tags:
+- aws
+---
+
+AWS announced a new API called [Cloud Control](https://aws.amazon.com/blogs/aws/announcing-aws-cloud-control-api/) that provides a standard sets of APIs to manage AWS resources. Imagine running `aws cloudcontrol create-resource` to launch EC2 and Lambda, instead of using `aws ec2 run-instances` and `aws lambda create-function`.
+
+Aside from CRUD operations, it also supports List operation to discover all deployed resources filtered by a specific resource type (e.g. `AWS::ECS::Cluster`). When I first read the announcement, I wonder how it compares to {% post_link aws-config 'AWS Config' %}, a feature I'm actively using mainly for security audit, but it could also perform inventory task.
+
+Since Cloud Control is a recent feature, the latest library is required. For Python library, I ran `pip install boto3 --upgrade` to update it to version xxx. Then, I created a minimal Python script to test out Cloud Control's [ListResources](https://docs.aws.amazon.com/cloudcontrolapi/latest/APIReference/API_ListResources.html).
+
+``` py
+#!/usr/bin/env python
+
+# ./cloud-control.py --profile profile-name --region region-name
+
+from argparse import ArgumentParser
+import boto3
+from botocore.config import Config
+from itertools import count
+from json import dump, loads
+
+parser = ArgumentParser(description = 'Find the latest AMIs.')
+parser.add_argument('--profile', '-p',
+  help = 'AWS profile name. Parsed from ~/.aws/config (SSO) or credentials (API key).',
+  required = True)
+parser.add_argument('--region', '-r',
+  help = 'AWS Region, e.g. us-east-1',
+  required = True)
+args = parser.parse_args()
+profile = args.profile
+region = args.region
+
+session = boto3.session.Session(profile_name = profile)
+my_config = Config(region_name = region)
+client = session.client('cloudcontrol', config = my_config)
+
+results = []
+response = {}
+
+for i in count():
+  # https://docs.aws.amazon.com/cloudcontrolapi/latest/APIReference/API_ListResources.html
+  params = {
+    # https://docs.aws.amazon.com/cloudcontrolapi/latest/userguide/supported-resources.html
+    'TypeName': 'AWS::EC2::FlowLog'
+  }
+  if i == 0 or 'NextToken' in response:
+    if 'NextToken' in response:
+      params['NextToken'] = response['NextToken']
+    response = client.list_resources(**params)
+    results.extend(response['ResourceDescriptions'])
+  else:
+    break
+
+prop_list = []
+
+# Extract properties only
+for ele in results:
+  prop_list.append(loads(ele['Properties']))
+
+if len(prop_list) >= 1:
+  with open('cloud-control.json', 'w') as w:
+    # Save the first dictionary only
+    dump(dict(sorted(prop_list[0].items())), w, indent = 2)
+```
+
+In the first draft of the script, I noticed that the API doesn't support `AWS::EC2::Instance` yet. It took me a while to troubleshoot until I found [this list](https://docs.aws.amazon.com/cloudcontrolapi/latest/userguide/supported-resources.html) of supported resources. The error wasn't very helpful, e.g. "Resource type AWS::EC2::Instance does not support LIST action". It's more straightforward to just say "Resource type xxx does not support Cloud Control yet".
+
+The announcement did mention not all resources are supported, but I didn't expect AWS' _bread and butter_ are unsupported, including `AWS::S3::Bucket`. I'm sure these resources will be supported eventually, it's just that support of new products are prioritised at the moment as implied from the announcement, "It will support new AWS resources typically on the day of launch".
+
+I tested on `AWS::EC2::PrefixList`, instead of the currently unsupported `AWS::EC2::Instance`. It worked fine, the output syntax is exactly what the documentation outlines. To compare it to Config, I created another equivalent script.
+
+``` py
+#!/usr/bin/env python
+
+# ./aws-config.py --profile profile-name --account-id {account-id} --region region-name
+
+from argparse import ArgumentParser
+import boto3
+from botocore.config import Config
+from itertools import count
+from json import dump, loads
+
+parser = ArgumentParser(description = 'Find the latest AMIs.')
+parser.add_argument('--profile', '-p',
+  help = 'AWS profile name. Parsed from ~/.aws/config (SSO) or credentials (API key).',
+  required = True)
+parser.add_argument('--account-id', '-a',
+  help = 'AWS account ID. See ~/.aws/config if SSO is used.',
+  required = True,
+  type = str)
+parser.add_argument('--region', '-r',
+  help = 'AWS Region, e.g. us-east-1',
+  required = True)
+args = parser.parse_args()
+profile = args.profile
+account_id = args.account_id
+region = args.region
+
+session = boto3.session.Session(profile_name = profile)
+my_config = Config(region_name = region)
+client = session.client('config', config = my_config)
+
+results = []
+response = {}
+
+for i in count():
+  params = {
+    'Expression': "SELECT configuration WHERE resourceType = 'AWS::EC2::FlowLog'" \
+      f" AND accountId = '{account_id}'" \
+      f" AND awsRegion = '{region}'",
+    'ConfigurationAggregatorName': 'ConfigAggregator' # may need to update
+  }
+  if i == 0 or 'NextToken' in response:
+    if 'NextToken' in response:
+      params['NextToken'] = response['NextToken']
+    response = client.select_aggregate_resource_config(**params)
+    results.extend(response['Results'])
+  else:
+    break
+
+
+conf_list = []
+
+# Extract configuration only
+for ele in results:
+  conf_list.append(loads(ele).get('configuration', {}))
+
+if len(conf_list) >= 1:
+  with open('aws-config.json', 'w') as w:
+    # Save the first dictionary only
+    dump(dict(sorted(conf_list[0].items())), w, indent = 2)
+```
+
+Before I get to the output comparison, notice the `accountId` and `awsRegion` filters I used in the SQL statement. It's necessary because I'm using an [aggregator](https://docs.aws.amazon.com/config/latest/developerguide/aggregate-data.html) that collects data from **all accounts and regions** in an AWS Organization (which have AWS Config enabled). Like most other AWS APIs, Cloud Control only works on a combination of account and region. If you want discover resources in 5 combinations of account and region, that'll requires 5 API calls, in contrast to just one API call via Config's aggregator.
+
+Here is the output of Cloud Control:
+
+``` json
+{
+  "DeliverLogsPermissionArn": String,
+  "Id": String,
+  "LogDestination": String,
+  "LogDestinationType": String,
+  "LogFormat": String,
+  "LogGroupName": String,
+  "MaxAggregationInterval": Integer,
+  "ResourceId": String,
+  "ResourceType": String,
+  "Tags": [ Tag, ... ],
+  "TrafficType": String
+}
+```
+
+Config:
+
+``` json
+{
+  "creationTime": Float,
+  "deliverLogsPermissionArn": String,
+  "deliverLogsStatus": String,
+  "flowLogId": String,
+  "flowLogStatus": String,
+  "logDestination": String,
+  "logDestinationType": String,
+  "logFormat": String,
+  "logGroupName": String,
+  "maxAggregationInterval": Float,
+  "resourceId": String,
+  "tags": [ Tag, ... ],
+  "trafficType": String
+}
+```
+
+Syntax used by [CloudFormation template](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-flowlog.html):
+
+``` json
+{
+  "DeliverLogsPermissionArn" : String,
+  "LogDestination" : String,
+  "LogDestinationType" : String,
+  "LogFormat" : String,
+  "LogGroupName" : String,
+  "MaxAggregationInterval" : Integer,
+  "ResourceId" : String,
+  "ResourceType" : String,
+  "Tags" : [ Tag, ... ],
+  "TrafficType" : String
+}
+```