hidekazu-konishi.com
Deploy AWS Cloudformation Stack Cross-Region with AWS Lambda Custom Resources
First Published:
Last Updated:
The first reason I decided to write this article is because there are limitations that Amazon CloudFront's AWS Certificate Manager certificates and Lambda@Edge can only be used in the Virginia region (us-east-1).
For example, if you deploy a stack that creates an Amazon S3 and Amazon CloudFront static hosting environment in the Tokyo region (ap-northeast-1), you need to create resources in the Virginia region (us-east-1) to add AWS Certificate Manager certificates and Lambda@Edge.
In fact, this requirement can be achieved by using the AWS Cloud Development Kit (AWS CDK). However, the second reason is that I wanted to try and verify a method to deploy using only AWS CloudFormation templates from the AWS Management Console without installing and setting up a software development framework like AWS CDK.
When using only AWS CloudFormation templates to deploy AWS resources, considering Cross-Region, the following main options can be considered:
1. Create all resources in the us-east-1 AWS CloudFormation stack
2. Create only the necessary resources in us-east-1 by calling a Lambda custom resource from the AWS CloudFormation stack deployed in ap-northeast-1
3. Create only the necessary resources in us-east-1 with AWS CloudFormation StackSets
4. Manually create the relevant part without automating it using the AWS Management Console, etc.
With option 1, it is completed within the us-east-1 region, but it cannot be selected when you want to create resources in other regions.
Options 2 and 3 are approaches that actively automate, but you need to pass the ARNs, etc. of resources created in us-east-1 to the AWS CloudFormation stack deployed in ap-northeast-1 and associate Amazon CloudFront with AWS Certificate Manager certificates and Lambda@Edge.
Option 4 can be said to be a realistic method for saving development time when you don't need to create the same environment multiple times or don't need to fully automate the process after parameter input.
This time, I will try using option 2 among these, calling a Lambda custom resource in the original AWS CloudFormation stack, creating resources by AWS CloudFormation stack in us-east-1, receiving the return value in the original AWS CloudFormation stack, and continuing the process.
* The source code published in this article and other articles by this author was developed as part of independent research and is provided 'as is' without any warranty of operability or fitness for a particular purpose. Please use it at your own risk. The code may be modified without prior notice.
What is an AWS CloudFormation custom resource?
An AWS CloudFormation custom resource that can be used as an option is for executing resources that cannot be used as an AWS CloudFormation resource type with an AWS Lambda function or triggering them with an Amazon SNS topic.For example, cases such as using external services or on-premises systems can be mentioned, but there are various ways to use it, such as processing across regions within AWS services and calling microservices built within AWS, as in this case.
However, when implementing a custom resource with AWS Lambda, it is necessary to be aware that limitations occur due to the specifications of AWS Lambda, such as the need to keep the execution time within a maximum of 15 minutes.
Also, when implementing a custom resource with AWS Lambda, it is necessary to return a response to the executing AWS CloudFormation stack in a format specific to it.
To implement this response return function, you need to either use the provided CloudFormation-specific method or create your own response method.
Details will be described later.
AWS Lambda custom resource that deploys AWS CloudFormation stack across regions
First, I will show the usage pattern of this AWS Lambda custom resource in a Architecture Diagram.
The main purpose is to deploy a separate AWS CloudFormation stack across regions with the parameters provided from the calling AWS CloudFormation stack, return the result to the caller, and use the result parameters to associate resources in the caller's processing.
This AWS CloudFormation stack deployed across regions needs to be completed within the maximum 15 minutes of AWS Lambda execution time. If deploying a CloudFormation stack that takes more than 15 minutes using a custom resource, it would be better to split the CloudFormation template into multiple ones and execute them, or execute them using another method without time limit.
Also, the IAM policy permissions of the IAM role attached to this Lambda custom resource need to grant necessary permissions for resources accessed by the Lambda function, CloudFormation execution, resources deployed by the CloudFormation stack, etc.
The AWS Lambda custom resource that deploys an AWS CloudFormation stack across regions was implemented in Python as follows. The runtime used is Python 3.9. There are several points I devised, so I will list the features.
- Created a custom method to return a response according to the custom resource specification
When called from a CloudFormation stack, cfnresponse.send can be used, but it cannot be used for standalone Lambda execution or testing, so I created a custom method called cfn_response_send.
Reference: cfn-response module - AWS CloudFormation
- Allowed selecting the template to deploy from S3 or hard-coding
The method of retrieving the template from S3 allows executing various templates. On the other hand, if you want to specialize in executing a specific template, you can select hard-coding to prevent other templates from being executed.
- Simplified the specification of arguments for the CloudFormation template deployed from the custom resource
By specifying keys that are not used as arguments for the CloudFormation template to deploy and passing the rest, I made the argument passing generic.
- Aligned with the policy of creating resources even in situations different from the requested process
Instead of using the requested Create and Update processes directly for CloudFormation deployment, I switched to the Create process when the stack does not exist in the Update process, etc., to align with the policy of creating resources as much as possible.
- Allowed selecting to skip processing
I made it possible to select skipping processing so that you can maintain dependencies in the caller while not using it in the Create process and using it from the Update process.
# ## Features of this custom resource
# * Created a custom method to return a response according to the custom resource specification.
# When called from a CloudFormation stack, cfnresponse.send can be used, but it cannot be used for standalone Lambda execution or testing, and the code content cannot be customized, so I created a custom method called cfn_response_send.
# Reference: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-lambda-function-code-cfnresponsemodule.html
# * Allowed selecting the template to deploy from S3 or hard-coding.
# The method of retrieving the template from S3 allows executing various templates. On the other hand, if you want to specialize in executing a specific template, you can select hard-coding to prevent other templates from being executed.
# * Simplified the specification of arguments for the CloudFormation template deployed from the custom resource.
# By specifying keys that are not used as arguments for the CloudFormation template to deploy and passing the rest, I made the argument passing generic.
# * Aligned with the policy of creating resources even in situations different from the requested process.
# Instead of using the requested Create and Update processes directly for CloudFormation deployment, I switched to the Create process when the stack does not exist in the Update process, etc., to align with the policy of creating resources as much as possible.
# * Allowed selecting to skip processing.
# I made it possible to select skipping processing so that you can maintain dependencies in the caller while not using it in the Create process and using it from the Update process.
import urllib3
import json
import boto3
import botocore
import datetime
# import cfnresponse
SUCCESS = 'SUCCESS'
FAILED = 'FAILED'
http = urllib3.PoolManager()
# Keys not used as arguments for the CloudFormation stack to deploy out of the arguments received from the calling CloudFormation stack
config = {
"ExcludeKeys": ["ServiceToken", "StackName", "CfnTplS3Bucket", "CfnTplS3Key", "IsSkip"]
}
# If you want to directly describe the CloudFormation template to deploy in the code, describe it here
fixed_cfn_template = '''
'''
# Client that deploys CloudFormation stack in us-east-1 region
cfn_client = boto3.client('cloudformation', region_name='us-east-1')
# S3 resource that retrieves CloudFormation template
s3_resource = boto3.resource('s3')
# Custom method that returns a response to CloudFormation according to the custom resource specification
# When called from a CloudFormation stack, cfnresponse.send can be used, but it cannot be used for standalone Lambda execution, so a custom method is used
def cfn_response_send(event, context, response_status, response_data, physical_resource_id=None, no_echo=False, reason=None):
response_url = event['ResponseURL']
print(response_url)
response_body = {
'Status': response_status,
'Reason': reason or 'See the details in CloudWatch Log Stream: {}'.format(context.log_stream_name),
'PhysicalResourceId': physical_resource_id or context.log_stream_name,
'StackId': event['StackId'],
'RequestId': event['RequestId'],
'LogicalResourceId': event['LogicalResourceId'],
'NoEcho': no_echo,
'Data': response_data
}
json_response_body = json.dumps(response_body)
print('Response body:')
print(json_response_body)
headers = {
'content-type': '',
'content-length': str(len(json_response_body))
}
try:
response = http.request('PUT', response_url, headers=headers, body=json_response_body)
print('Status code:', response.status)
except Exception as e:
print('cfn_response_send(..) failed executing http.request(..):', e)
def lambda_handler(event, context):
response_data = {}
try:
print(('event:' + json.dumps(event, indent=2)))
resource_properties = event['ResourceProperties']
is_skip = resource_properties.get('IsSkip')
# If "true" is passed to IsSkip, return an empty response without processing anything
# Used when the caller wants to skip custom resource processing
if is_skip is not None and is_skip.strip().lower() == 'true':
print('Skip All Process.')
cfn_response_send(event, context, SUCCESS, response_data)
# cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)
return
stack_name = resource_properties.get('StackName')
exclude_keys = config['ExcludeKeys']
cfn_params = []
for key, val in resource_properties.items():
if key not in exclude_keys:
cfn_params.append({
'ParameterKey': key,
'ParameterValue': val
})
print('----------[Start]cfn_params:----------')
print(cfn_params)
print('----------[End]cfn_params:------------')
cfn_template = fixed_cfn_template
try:
# If the template is not directly described in the fixed_cfn_template variable, retrieve the template from the specified S3 bucket object
if cfn_template.strip() == '':
cfn_tpl_s3_bucket = resource_properties['CfnTplS3Bucket']
cfn_tpl_s3_key = resource_properties['CfnTplS3Key']
bucket = s3_resource.Bucket(cfn_tpl_s3_bucket)
obj = bucket.Object(cfn_tpl_s3_key).get()
cfn_template = obj['Body'].read().decode('utf-8')
print('----------[Start]cfn_template:----------')
print(cfn_template)
print('----------[End]cfn_template:------------')
except Exception as e:
print('lambda_handler(..) failed executing read s3 obj(..):', e)
raise
if event['RequestType'] == 'Delete':
try:
print('Delete Stacks')
# In the case of the Delete process, simply delete
print('Run cfn_client.delete_stack.')
response = cfn_client.delete_stack(
StackName=stack_name
)
print('cfn_client.delete_stack response:')
print(response)
# Wait for the stack's Delete process to complete
waiter = cfn_client.get_waiter('stack_delete_complete')
waiter.wait(StackName=stack_name)
except (botocore.exceptions.ValidationError, botocore.exceptions.ClientError) as err:
err_msg = str(err)
print('err_msg:')
print(err_msg)
# A deletion error loop occurs between associated resources, so exceptions are not raised.
if event['RequestType'] == 'Create' or event['RequestType'] == 'Update':
# Execution policy of Create process and Update process
# Create process request and stack does not exist ⇒ Create process
# Create process request and stack exists ⇒ Do nothing
# Update process request and stack does not exist ⇒ Create process
# Update process request and stack exists and changes exist ⇒ Update process
# Update process request and stack exists and no changes exist ⇒ Do nothing
# Flag indicating whether a stack with the same name exists
exist_stack = False
try:
print('Run cfn_client.describe_stacks.')
existing_stacks = cfn_client.describe_stacks(StackName=stack_name)
print('Existing Stacks:')
print(existing_stacks)
# If the stack exists and describe_stacks is processed normally, exist_stack is True
exist_stack = True
except (botocore.exceptions.ValidationError, botocore.exceptions.ClientError) as err:
err_msg = str(err)
print('err_msg:')
print(err_msg)
# If the stack does not exist and describe_stacks results in an error, exist_stack is False
if 'DescribeStack' in err_msg and 'does not exist' in err_msg:
print(('describe_stacks error: ' + err_msg))
exist_stack = False
else:
raise
# Flag indicating whether the Create process is necessary
need_create = True
if event['RequestType'] == 'Update':
print('Update Stacks')
if exist_stack == False:
# In the case of Update process and stack does not exist, create the stack with the Create process
need_create = True
else:
# Flag indicating whether the Create process is necessary
need_create = False
# If a stack matching the stack name already exists, process with Update
try:
print('Run cfn_client.update_stack.')
response = cfn_client.update_stack(
StackName=stack_name,
TemplateBody=cfn_template,
Parameters=cfn_params,
Capabilities=[
'CAPABILITY_NAMED_IAM'
]
)
print('cfn_client.update_stack response:')
print(response)
# Wait for the stack's Update process to complete
waiter = cfn_client.get_waiter('stack_update_complete')
waiter.wait(StackName=stack_name)
except (botocore.exceptions.ValidationError, botocore.exceptions.ClientError) as err:
err_msg = str(err)
print('err_msg:')
print(err_msg)
# If Update process is requested but there are no changes, do nothing
if 'UpdateStack' in err_msg and 'No updates are to be performed' in err_msg:
print(('update_stacks error: ' + err_msg))
else:
raise
if event['RequestType'] == 'Create' or need_create == True:
print('Create Stacks')
# Create process the stack in the case of Create process request and stack does not exist, or Update process request and stack does not exist
if exist_stack == False:
print('Run cfn_client.create_stack.')
response = cfn_client.create_stack(
StackName=stack_name,
TemplateBody=cfn_template,
Parameters=cfn_params,
Capabilities=[
'CAPABILITY_NAMED_IAM'
]
)
print('cfn_client.create_stack response:')
print(response)
# Wait for the stack's Create process to complete
waiter = cfn_client.get_waiter('stack_create_complete')
waiter.wait(StackName=stack_name)
# Retrieve the Created or Updated stack
stacks = cfn_client.describe_stacks(StackName=stack_name)
# Retrieve the contents of Outputs and format them as return values
outputs = stacks['Stacks'][0]['Outputs']
print(('Outputs:' + json.dumps(outputs, indent=2)))
for output in outputs:
response_data[output['OutputKey']] = output['OutputValue']
print(('Outputs:' + json.dumps(response_data, indent=2)))
# Return a response to the calling stack according to the CloudFormation custom resource specification
cfn_response_send(event, context, SUCCESS, response_data)
# cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)
return
except Exception as e:
print('Exception:')
print(e)
# Return a response to the calling stack according to the CloudFormation custom resource specification
cfn_response_send(event, context, FAILED, response_data)
# cfnresponse.send(event, context, cfnresponse.FAILED, response_data)
return
About the arguments from the calling CloudFormation stack passed to the AWS Lambda custom resource
The arguments from the calling CloudFormation stack passed to the above custom resource are expected to be in the following format.
{
"RequestType": "Create",
"ResponseURL": "http://pre-signed-S3-url-for-response",
"StackId": "arn:aws:cloudformation:ap-northeast-1:123456789012:stack/stack-name/guid",
"RequestId": "unique id for this create request",
"ResourceType": "Custom::TestResource",
"LogicalResourceId": "MyTestResource",
"ResourceProperties": {
"ServiceToken": "arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:function:CustomResourceToDeployCloudformationStack",
"StackName": "CustomResourceToDeployCloudformationStackDemo",
"CfnTplS3Bucket": "[S3 bucket name that stores the CloudFormation template to deploy]",
"CfnTplS3Key": "[Object key after the S3 bucket name for the CloudFormation template to deploy]",
"IsSkip": "[Flag indicating whether to skip the CloudFormation template deployment process]",
"[Key 1 to pass to the stack to deploy]": "[Value 1 to pass to the stack to deploy]",
"[Key 2 to pass to the stack to deploy]": "[Value 2 to pass to the stack to deploy]",
--omitted--
}
}
Out of this format, the arguments that need to be set in the calling stack to use the custom resource are as follows.
- StackName
The name of the CloudFormation stack that the custom resource deploys across regions.
- CfnTplS3Bucket
The bucket name when storing the CloudFormation template that the custom resource deploys across regions in S3.
- CfnTplS3Key
The key after the bucket name when storing the CloudFormation template that the custom resource deploys across regions in S3.
- IsSkip
Used when skipping the CloudFormation template deployment process from the custom resource to across regions. If "true", the process is skipped without execution.
- [Key X to pass to the stack to deploy]
Arguments passed to the CloudFormation template that the custom resource deploys across regions. In the custom resource implemented this time, arguments other than
"ServiceToken","StackName","CfnTplS3Bucket","CfnTplS3Key","IsSkip"
passed are all passed as arguments to the CloudFormation template to deploy. Therefore, the caller needs to be careful not to pass unnecessary arguments.
References:
Tech Blog with curated related content
Custom resources - AWS CloudFormation
What is AWS CloudFormation? - AWS CloudFormation
Summary
In this article, I mainly created a Lambda custom resource that is called from a CloudFormation stack in a region other than us-east-1 and deploys a CloudFormation stack in us-east-1.In future articles, I would like to introduce examples of actually deploying resources using this custom resource.
Written by Hidekazu Konishi
Copyright © Hidekazu Konishi ( hidekazu-konishi.com ) All Rights Reserved.