SCALING YOUR MIGRATION OPERATIONS

Can you scale your migration operations?
Your company is going all-in on AWS, and the sysops on the migration team just called in sick on a big migration day. (probably too much work over the last few days, connecting to each instance to reconfigure them)
You are now the primary sysops on the migration, but can you migrate hundreds servers a day and still make it in time at home ?

Is your nickname really OpsAutomator ? Let’s find out.

Task 1: Scripting is everything
Background
Congratulations on your new job ! The migration lead is concerned about your ability to handle all these servers at once. Remember what happend to the previous engineer ? While searching about AWS, you found out that AWS Systems Manager has a feature to automate common administrative tasks across groups of instances. This service looks like exactly what you need!

Your task
Your task is to find a way to execute the following script on every migrated instance, without having to connect to any of them. The document should be named ConfigureProxy and should run only on linux platform.

#!/bin/bash
set -e
PROXY=“http://proxy.acme.aws:3128”
NOPROXY=“169.254.169.254,ssm.eu-west-1.amazonaws.com,ec2.eu-west-1.amazonaws.com,ec2messages.eu-west-1.amazonaws.com”
mkdir -p /etc/systemd/system/snap.amazon-ssm-agent.amazon-ssm-agent.service.d
cat << EOF > /etc/systemd/system/snap.amazon-ssm-agent.amazon-ssm-agent.service.d/override.conf
[Service]
Environment=“http_proxy= P R O X Y " E n v i r o n m e n t = " h t t p s p r o x y = PROXY" Environment="https_proxy= PROXY"Environment="httpsp​roxy=PROXY”
Environment=“no_proxy=KaTeX parse error: Expected '}', got 'EOF' at end of input: …{ HTTP::proxy "PROXY”;
HTTPS::proxy “$PROXY”;
}
EOF
echo “Proxy configuration complete.”
Getting started
Jump to the AWS Console and start looking at how AWS Systems Manager handles remote script execution.

Inventory
Your AWS Account is provisioned with an Amazon EC2 instance named MigratedInstance that you can use for testing and troubleshooting.

Services you should use
AWS Systems Manager, Amazon EC2 for troubleshooting

Task validation
Your task is validated automatically. You can also click on the check my progress button to have some guidance.

Clue 2:Solution
In order to solve this task, you must perform complete the following steps:

Go to AWS Systems Manager console page.
Click on Documents
Click Create command or session
Enter “ConfigureProxy” for Name
Select YAML for content and paste the following

schemaVersion: “2.2”
description: “Command Document Example YAML Template”
mainSteps:

• action: “aws:runShellScript”
  name: “configureLinuxProxy”
  precondition:
  StringEquals:
  - platformType
  - Linux
  inputs:
  runCommand:
  • ‘#!/bin/bash’
  • ‘set -e’
  • ‘PROXY=“http://proxy.acme.aws:3128”’
  • ‘NOPROXY=“169.254.169.254,ssm.eu-west-1.amazonaws.com,ec2.eu-west-1.amazonaws.com,ec2messages.eu-west-1.amazonaws.com”’
  • ‘mkdir -p /etc/systemd/system/snap.amazon-ssm-agent.amazon-ssm-agent.service.d’
  • ‘cat << EOF > /etc/systemd/system/snap.amazon-ssm-agent.amazon-ssm-agent.service.d/override.conf’
  • ‘[Service]’
  • ‘Environment=“http_proxy=$PROXY”’
  • ‘Environment=“https_proxy=$PROXY”’
  • ‘Environment=“no_proxy=$NOPROXY”’
  • ‘EOF’
  • ‘systemctl daemon-reload’
  • ‘systemctl restart snap.amazon-ssm-agent.amazon-ssm-agent.service’
  • ‘cat << EOF > /etc/apt/apt.conf.d/02proxy’
  • ‘Acquire {’
  • ‘HTTP::proxy “$PROXY”;’
  • ‘HTTPS::proxy “$PROXY”;’
  • ‘}’
  • ‘EOF’
  • ‘echo “Proxy configuration complete.”’
    The document created will have one step named configureLinuxProxy. This step has a precondition to run only on Linux hosts. It takes then as input a runCommand object containing a list of shell commands to execute.

Click Create document

Task 2: If you do it twice, it’s one time too many.
Background
You made it through the day, all servers were migrated in time thanks to you and your manager is very pleased! As he hang up the call, he sends you a mail with a list of actions to be performed during migration. Those actions were dropped from the initial migration steps because the sysops didn’t have time to complete them.

FROM : manager@acme.com

Thanks again for your help today. Your script and RunCommand document really helped reduce the migration duration. Now that you have some spare time, maybe you could take care of the following list ?

• Tag the instance to track which one are being reconfigured. We had some issues with instances being missed.
• Apply the proxy settings
• Apply the patch baseline
• Remove the discovery agent
• Tag the instance to track the completion so the next team knows when to reconfigure the application.

Regards,
Your manager.
Your task
In this second task, you are expected to create an automation document named PostMigrationAutomation (case sensitive) which will perform the tasks above:

Tag the instance with the key : migration_status and the value : postmigration_started
Run the ConfigureProxy document to set the proxy settings
Run AWS-RunPatchBaseline document to check the instance for updates
Run the RemoveDiscoveryAgent document to clean the server from the discovery agent
Tag the instance with the key : migration_status and the value : postmigration_completed
Getting started
Open AWS Systems Manager console and look for some samples.

Inventory
One IAM Role to use to delegate the automation in the Output properties

Services you should use
AWS Systems Manager, Amazon EC2 for troubleshooting

Task validation
Your task is validated automatically. You can also click on the check my progress button to have some guidance.

NB:
You are not expected to run the automation document for successful completion of the challenge.

Clue 2:Solution
To solve this tasks, you must perform the following actions.

Go to AWS Systems Manager console.
Go to Documents.
Click on Create automation.
Enter “PostMigrationAutomation” as Name.
Enter the ARN of SSMAutomationRole for Assume role.
For Input parameters, Parameter name enter “instanceId”.
For Step 1:
Enter “tag_start” for Step Name.
Select Create tags for AWS resources for Action type.
Enter - ‘{{instanceId}}’ for Resource IDs under Inputs.
Enter [{“Key”: “migration_status”, “Value”: “postmigration_started”}] for Tags.
Click on Add step.
For Step 2:
Enter “configure_proxy” for Step Name.
Select Run a command on a managed instance for Action type.
Enter “ConfigureProxy” for Document name under Inputs
Enter - ‘{{instanceId}}’ for InstanceIds under Inputs.
Click on Add step.
For Step 3:
Enter “patch_instance” for Step Name.
Select Run a command on a managed instance for Action type.
Enter “AWS-RunPatchBaseline” for Document name under Inputs
Enter - ‘{{instanceId}}’ for InstanceIds under Inputs.
Click on Add step.
For Step 4:
Enter “remove_discovery_agent” for Step Name.
Select Run a command on a managed instance for Action type.
Enter “RemoveDiscoveryAgent” for Document name under Inputs
Enter - ‘{{instanceId}}’ for InstanceIds under Inputs.
Click on Add step.
For Step 5:
Enter “tag_complete” for Step Name.
Select Create tags for AWS resources for Action type.
Enter - ‘{{instanceId}}’ for Resource IDs under Inputs.
Enter [{“Key”: “migration_status”, “Value”: “postmigration_completed”}] for Tags.
Click on Create automation.
This document will perform the 5 steps of tagging, configuring proxy, patching, removing the agent and tagging again. All this steps will be launched throught the SSM Automation Role. The parameter instanceId will determine which instance must be targeted.

Task 3: All hands off ! Time to relax
Keeping track of migrated servers
Background
You automated all the post migrations steps and servers are being migrated day in, day out. You made sure to never forget a server by tagging them when the post migration starts and when the post migration ends. As the migration project is a bit late on schedule, your manager is calling in to ask if you could postpone your holidays to keep the migration flowing. As you have no intentions to do so, you are looking for a way to trigger the automation automatically even when on holidays.

Your task
In this third tasks, you should look at triggering the automation when the tag migration_status = ready_for_postmigration is set. You can do so by creating a rule for associated event in CloudTrail. The rule should be named AutomationTrigger.

sample event:

{
“version”: “0”,
“id”: “67df5f3e-57ea-fac5-4c6e-fc83b9abcdef”,
“detail-type”: “AWS API Call via CloudTrail”,
“source”: “aws.ec2”,
“account”: “123456789012”,
“time”: “2020-10-19T06:22:21Z”,
“region”: “eu-west-1”,
“resources”: [],
“detail”: {
“eventVersion”: “1.05”,
“userIdentity”: {
“type”: “AssumedRole”,
“principalId”: “AROAI4UAWJ76D55TKEVY6:Example”,
“arn”: “arn:aws:sts::123456789012:assumed-role/Admin/ExampleSession”,
“accountId”: “123456789012”,
“accessKeyId”: “ASIAUTDEHZRDLEXAMPLE”,
“sessionContext”: {
“sessionIssuer”: {
“type”: “Role”,
“principalId”: “AROAI4UAWJ76D5EXAMPLE”,
“arn”: “arn:aws:iam::123456789012:role/Admin”,
“accountId”: “123456789012”,
“userName”: “Admin”
},
“webIdFederationData”: {},
“attributes”: {
“mfaAuthenticated”: “false”,
“creationDate”: “2020-10-19T06:13:22Z”
}
}
},
“eventTime”: “2020-10-19T06:22:21Z”,
“eventSource”: “ec2.amazonaws.com”,
“eventName”: “CreateTags”,
“awsRegion”: “eu-west-1”,
“sourceIPAddress”: “10.0.0.1”,
“userAgent”: “console.ec2.amazonaws.com”,
“requestParameters”: {
“resourcesSet”: {
“items”: [
{
“resourceId”: “i-00c9eab2d04abcdef”
}
]
},
“tagSet”: {
“items”: [
{
“key”: “migration_status”,
“value”: “ready_for_postmigration”
}
]
}
},
“responseElements”: {
“requestId”: “a939f941-6a83-4048-9717-a9ba43485aad”,
“_return”: true
},
“requestID”: “a939f941-6a83-4048-9717-a9ba43485aad”,
“eventID”: “80e3b894-d070-4ad5-b122-e0712da23d79”,
“eventType”: “AwsApiCall”
}
}
Getting started
Open Amazon EventBridge and create your own rule.
You will be dealing with JSON events, so I would recommend to use a JSONPath evaluator to ease the troublehsooting.

Inventory
One IAM Role to start the automation in the Output properties.

Services you should use
Amazon EventBridge, AWS Systems Manager, Amazon EC2 for troubleshooting.

Task validation
Your task is validated automatically. You can also click on the check my progress button to have some guidance.

Clue 3:Solution
In order to solve this task:

Go to Amazon EventBridge console.
Click on Create rule.
Enter “AutomationTrigger” for Name.
Select Event pattern.
Select custom pattern.
Enter the following pattern in Event pattern.
{
“source”: [
“aws.ec2”
],
“detail-type”: [
“AWS API Call via CloudTrail”
],
“detail”: {
“eventSource”: [
“ec2.amazonaws.com”
],
“eventName”: [
“CreateTags”
],
“requestParameters”: {
“tagSet”: {
“items”: {
“key”: [
“migration_status”
],
“value”: [
“ready_for_postmigration”
]
}
}
}
}
}
This pattern means that we are looking for aws.ec2 events, coming from CloudTrail, where the source is ec2.amazonaws.com, the event name is CreateTags and the Key and Value are migration_status and ready_for_postmigration.

Click Save
For Select targets, choose SSM Automation.
Select PostMigrationAutomation for Document.
Choose Input Transformer for Configure automation parameter(s)
Paste the following json in the first cell
{“instances”: “$.detail.requestParameters.resourcesSet.items[*].resourceId”}
Paste the following json in the second cell (replace instanceId with the parameter name of your PostMigrationAutomation SSM document)
{“instanceId”: []}
Choose Use existing role
Select Amazon_EventBridge_Start_Automation
Click Create

MY PIPELINE IS STUCK!
Background
AWS CodePipeline includes a number of actions that help you configure build, test, and deploy resources for your automated release process. AWS CodePipeline enables custom actions that can be leveraged to achieve a wide range of integrations. In the current release process, the team is interested in creating a custom action that triggers a Lambda Function.

Your Task
One of your team members started working on implementing it and has since left the organization. A new team member joined your team and deployed the infrastructure. However, the deployed infrastructure is not working. It turns out that the previous Engineer did not commit the latest code to the source code repository (all changes were locally stored). The CodePipeline execution keeps running for 1 hour before timing out. Your team needs you to help troubleshoot the issue and fix it.

Getting started
Check the Output Properties tab of the challenge to get the name of the CodePipeline which is having issues. CodePipeline would already be running (in-progress).

Things to note
You can stop the pipeline execution by following the below steps
Click the “Stop Execution” button in CodePipeline.
Select the execution that is currently in progress
Select the “Stop and abandon” option.
Click the “Stop” button.
To trigger the pipeline after making changes, click on the “Release Changes” button in CodePipeline.
Inventory
CodePipeline
Lambda function: Added as a Custom Action step in CodePipeline - BrokenPipelineLambdaFunction
IAM Roles
CodePipeline
Lambda Function
S3 buckets for CodePipeline
Task Validation
The CodePipeline will execute successfully and the task will automatically be marked as completed in a few minutes. In addition you can always check your progress by pressing the ‘Check my progress’ button in challenge detail section.

Services used
CodePipeline, S3 Buckets, Lambda Function, API Gateway

Clue 1:Getting started
IAM role associated with the Lambda function needs permissions to post results to CodePipeline. Lambda function is named BrokenPipelineLambdaFunction. The IAM role name can be obtained from the “Permissions” tab in the Lambda console. This role is missing the necessary permissions.

Clue 2:One more hint
Lambda function code needs to be updated to be able to put success or failure result to CodePipeline.

The APIs that need to be added in the code are listed below:

Put success result
import boto3

cp_client = boto3.client(‘codepipeline’)
cp_client.put_job_success_result(jobId=event[‘CodePipeline.job’][‘id’])
Put failure result
import boto3

cp_client.put_job_failure_result(
jobId=event[‘CodePipeline.job’][‘id’], failureDetails={‘message’: str(e), ‘type’: ‘JobFailed’})

LOST IN NATRANSLATION
You are the senior network administrator of Impeccable Network. Your company wants to enable internet connectivity for their private servers. Your security team is extremely opinionated and has stipulated that you need to enable outbound traffic via NAT instances instead of NAT gateway to provide more granular controls. Your security team also insisted on using NACLs in addition to security groups.

Before leaving for vacation you had implemented this in dev and verified the configuration for a NAT Instance. While you were gone, the rest of the networking team has been testing in the same VPC and have made some changes after which the NATing has stopped working.

You have to demo the solution to the security team next week and you need to fix the VPC configurations before that. Are you ready for the challenge?

Task 1: First things first - Internet Access to and from the NAT Instance
Background
Review the architecture diagram. The private subnets will access the Internet via the NAT Instance. Currently the connection from the NAT Instance to the Webserver Instance is timing out.

Therefore, we should first verify Internet Connectivity from the NAT Instance. What VPC configurations are needed for Internet Connectivity from the NAT Instance?

Your Task
Log into the NAT instance to troubleshoot outbound network connectivity. Make appropriate changes to AWS configurations until outbound traffic from the NAT instances is working.

To verify the Internet connectivity from the NAT Instance, you need to first log into it. Use the Bastion host as the jump host to log into the NAT Instance.

To successfully complete the task, you should accomplish the following:

Access the Webserver Instance from the NAT Instance

You can verify the connectivity by running the following cURL command:

curl -Iv http://webserver-public-ip/
You can use the VPC Flow logs to see if the HTTP traffic from the NAT Instance is in ACCEPT state. VPC flow logs is a great feature which helps in troubleshooting connectivity issues. To view the VPC flow logs, go to the logs section on the CloudWatch console and look for the log group FlowLogsGroup. Look for the NAT Instance ENI to view the logs for the NAT Instance. If you don’t see the logs from the NAT Instance Private IP to the Webserver Public IP, in the flow logs, it means either the logs haven’t been published yet or you don’t have Internet connectivity to the instance (no route to the Internet). If the traffic is in REJECT state, it means either the Security Group or the NACL is not allowing TCP Port 80 traffic from the NAT Instance to the Webserver public IP.

The format for the VPC flow log is:

[version] [account-id] [interface-id] [srcaddr] [dstaddr] [srcport] [dstport] [protocol] [packets] [bytes] [start] [end] [action] [log-status]
Refer to the VPC Flow log documentation for more details on flow logs

Things you should avoid doing:

Allowing “All traffic” from or to “0.0.0.0/0” or “Anywhere” in the security group and Network ACL rules (mainly inbound).
The HTTP application is running on port 80 on the Webserver Instance. Avoid allowing “ALL TCP” ports.
Getting Started
Download the private key pair from the top right corner of the challenge “SSH Key Pair”.

You will use this key to SSH into the Bastion Instance. From the bastion host, you can SSH into the NAT Instance. Your key must not be publicly viewable for SSH. Use the command chmod 400 your-key-name.pem if needed. To SSH into the NAT Instance from the bastion Instance, you will need the private key on the bastion host. This is not secured and you can use SSH forwarding to avoid saving keys on bastion Instance.

For Linux/Mac/cygwin users, we suggest using ssh-agent. The steps needed to use SSH forwarding are -

Add private key to keychain by using the command -

ssh-add -K your-key-name.pem
Use the followingcommand to SSH into the Bastion Instance -

ssh -A ec2-user@Bastion-Instance-Public-IPAddress
For windows users and for more details SSH forwarding and how to configure ssh-agent on Windows machine, refer to the link.

Once you are logged into the bastion host, use the following command to SSH from the bastion host into the NAT Instance

ssh ec2-user@10.0.1.4
Validate that the web server is running and publicly accessbile

The Webserver instance is accessible over the Internet. Use the public IP of this Instance found in the Output Properties of your challenge as WebServerPublicIp to verify Internet connectivity from your local machine.

Inventory
VPC:
Name - VPC-JAM
CIDR - 10.0.0.0/16
Bastion Instance:
Public IP : You can get it from the Outputs Properties of this challenge BastionInstanceEIP
NAT Instance:
Private IP: 10.0.1.4
WebServer Instance:
Public IP : You can get it from the Outputs Properties of this challenge WebServerPublicIp
Services you should use
VPC
EC2
Task Validation
At the end of this task, you should be able to get a 200 O.K response for the cURL command. This will verify Internet Connectivity from the NAT Instance. The task will automatically complete once you have the right VPC configurations.

Note: The task will NOT complete if you have allowed traffic from 0.0.0.0/0 in the inbound rules.