Deploy CloudFormation Template

In this step we deploy a Cloudformation template to configure various AWS services required for this lab.

  1. Log into your AWS account and ensure you are in the Northern Virginia region (us-east-1). If you are running this lab at an AWS event, your host may have created an Event Engine account for you to use. If so, click this link for details on how to access the account.

  2. Right-click “Launch Stack” below and open up the link in a new window. This will take you to the CloudFormation service in your account.

    Image

  3. The template location in Amazon S3 will have been filled out for you, so click Next

  4. Provide an email address for receiving AWS SNS notifications, provide a name for the AWS Glue Crawler that will be created, and provide a database name that will be registered in the AWS Glue catalog. In the screenshot below “glueautomation” was used for crawler name and database name, but feel free to provide any name you prefer.
    Click Next Cloudformation-create-stack-step2

  5. On the next screen, accept all defaults and click Next Cloudformation-create-stack-step3

  6. On the final screen, scroll all the way to end and acknowledge the warning that “AWS CloudFormation might create IAM resources.”. Then click Create stack to start the Cloudformation deployment. Cloudformation-create-stack-step4

    The CloudFormation deployment should take less than 5 minutes to deploy. You can watch the progress of the deployment under Events as shown: Cloudformation-create-stack-step5

  7. Once the Cloudformation status shows Create_complete click on the Outputs tab and capture the details of the following resources:

    • 2 Lambda functions : Copy S3 data, Glue Trigger
    • 3 S3 Buckets : Holding, Raw, Processed

    Cloudformation-create-stack-output

    Important - To receive SNS notification for Glue ETL jobs.

  8. To receive email notifications when an ETL job completes, check for a new email that would have been sent to the address you provided in the CloudFormation deployment screen. Click Confirm Subscription: Cloudformation-sns-confirmation

  9. On the AWS SNS console choose Topics. An SNS topic named <stackname>-SNSProcessedEvent will be shown in the display.

    Select topic and confirm that the status for subscription shows as Confirmed. Cloudformation-sns-confirmation-status

    If for some reason you did not get an email, choose Request confirmation and then check your inbox. Also be sure to check your email junk folder for the request confirmation link. Cloudformation-sns-resend-confirmation