Free previewLab7 hours

Launch to Production

Launch EC2 instances, attach storage, bake AMIs, run Auto Scaling behind an ALB, access hosts with SSM, and use bastion hosts to reach private resources.

Mental model

EC2 is a virtual machine in AWS's data center. Everything else (storage, networking, access control) is a service you attach to it. This lab builds that picture layer by layer.

Prerequisites

0 of 5 checked

Launch, Access, Security Groups, IMDS

Goal

Launch an EC2 instance. Connect to it via SSH. Understand security groups as the network gate. Query the instance metadata service to understand how an instance knows about itself.

Estimated time: 1 hour

Create a key pair

What's happening here

SSH uses asymmetric cryptography. You hold the private key locally. AWS stores the public key and injects it into the instance at launch via cloud-init. When you SSH in, your local key proves identity without a password ever crossing the wire. AWS never sees your private key, you generate the key pair and only the public half goes to AWS. If you lose the private key, you lose SSH access to that instance permanently (unless you have SSM Session Manager set up, covered in Modern Access: SSM Session Manager + EC2 Instance Connect, and the bastion pattern in Bastion Hosts & Private Resource Access).

bash

# Import your local public key to AWS
aws ec2 import-key-pair \
  --key-name ec2-lab-key \
  --public-key-material fileb://~/.ssh/ec2-lab.pub \
  --region $AWS_REGION

# Verify
aws ec2 describe-key-pairs \
  --key-names ec2-lab-key \
  --region $AWS_REGION \
  --query 'KeyPairs[0].{Name:KeyName,Fingerprint:KeyFingerprint}'

Checkpoint: key pair visible in AWS with a fingerprint.

Create a security group

What's happening here

A security group is a stateful firewall attached to an instance (or ENI). Stateful means: if you allow inbound SSH on port 22, the response traffic is automatically allowed outbound, you don't need an explicit outbound rule for the reply. Two rules we need:

Port 22 (SSH), so you can connect
Port 80 (HTTP), so you can hit the web server we'll install

We scope SSH to your IP only ($(curl -s ifconfig.me)/32). Opening port 22 to 0.0.0.0/0 is the single most common EC2 security mistake.

bash

export VPC_ID=$(aws ec2 describe-vpcs \
  --filters Name=isDefault,Values=true \
  --query 'Vpcs[0].VpcId' --output text --region $AWS_REGION)

export MY_IP=$(curl -s ifconfig.me)

export SG_ID=$(aws ec2 create-security-group \
  --group-name ec2-lab-sg \
  --description "EC2 lab security group" \
  --vpc-id $VPC_ID \
  --region $AWS_REGION \
  --query 'GroupId' --output text)

# SSH from your IP only
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp --port 22 \
  --cidr $MY_IP/32 \
  --region $AWS_REGION

# HTTP from anywhere
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp --port 80 \
  --cidr 0.0.0.0/0 \
  --region $AWS_REGION

echo "Security Group: $SG_ID"

Checkpoint: SG created with two inbound rules. No inbound rule for any other port.

Launch an EC2 instance

What's happening here

Key decisions at launch time:

AMI (Amazon Machine Image. The template the instance boots from. We use Amazon Linux 2023 (AL2023)) AWS's maintained Linux distro, free tier eligible, has SSM agent pre-installed.
Instance type: t3.micro = 2 vCPU, 1GB RAM. t3 = burstable (accumulates CPU credits when idle, spends them on spikes). Fine for a lab.
--associate-public-ip-address: assigns a public IP so you can SSH in from your machine. Without this, the instance is only reachable inside the VPC.
--block-device-mappings (we explicitly set DeleteOnTermination=false on the root volume. By default it's true) terminate the instance and the disk is gone. We'll test this in Storage: EBS + EFS.

bash

# Get latest Amazon Linux 2023 AMI
export AMI_ID=$(aws ec2 describe-images \
  --owners amazon \
  --filters \
    Name=name,Values="al2023-ami-2023*-x86_64" \
    Name=state,Values=available \
  --query 'sort_by(Images,&CreationDate)[-1].ImageId' \
  --output text --region $AWS_REGION)

echo "AMI: $AMI_ID"

export INSTANCE_ID=$(aws ec2 run-instances \
  --image-id $AMI_ID \
  --instance-type t3.micro \
  --key-name ec2-lab-key \
  --security-group-ids $SG_ID \
  --associate-public-ip-address \
  --block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"VolumeSize":8,"DeleteOnTermination":false}}]' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ec2-lab}]' \
  --region $AWS_REGION \
  --query 'Instances[0].InstanceId' --output text)

echo "Instance ID: $INSTANCE_ID"

# Wait for it to be running
aws ec2 wait instance-running --instance-ids $INSTANCE_ID --region $AWS_REGION

export PUBLIC_IP=$(aws ec2 describe-instances \
  --instance-ids $INSTANCE_ID \
  --query 'Reservations[0].Instances[0].PublicIpAddress' \
  --output text --region $AWS_REGION)

echo "Public IP: $PUBLIC_IP"

Checkpoint: instance state running. Public IP printed.

SSH into the instance

What's happening here

The default username for Amazon Linux 2023 is ec2-user. This is AMI-specific. Ubuntu uses ubuntu, RHEL uses ec2-user or cloud-user, Debian uses admin. Using the wrong username is the most common SSH failure. -i specifies the private key. -o StrictHostKeyChecking=no skips the host fingerprint prompt on first connection (acceptable for a lab, not for production).

bash

ssh -i ~/.ssh/ec2-lab \
  -o StrictHostKeyChecking=no \
  ec2-user@$PUBLIC_IP

Once inside, run:

bash

# Who am I?
whoami

# What machine is this?
uname -a
hostname

# What's the instance's view of itself?
curl -s http://169.254.169.254/latest/meta-data/instance-id
curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone
curl -s http://169.254.169.254/latest/meta-data/local-ipv4
curl -s http://169.254.169.254/latest/meta-data/public-ipv4

# Install nginx to use in the next check
sudo dnf install -y nginx
sudo systemctl start nginx
sudo systemctl enable nginx

exit

From your local machine:

bash

curl http://$PUBLIC_IP

Checkpoint: nginx default page returned. SSH works. IMDS returns instance metadata.

Query the Instance Metadata Service (IMDS)

What's happening here

IMDS (169.254.169.254) is a special link-local IP reachable only from inside the instance. It's how the instance knows its own identity (AZ, region, instance ID, instance type) without you hardcoding any of this. IMDSv2 (the current version) requires a session token, a PUT request first to get a token, then GET requests using that token. This prevents SSRF attacks where a compromised app tricks the server into leaking its own credentials via IMDS. This is also how the instance profile delivers temporary IAM credentials to code running on the instance, we'll see this in User Data + IAM Instance Profile + S3 Access.

bash

ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP

bash

# IMDSv2 -- get a session token first
TOKEN=$(curl -s -X PUT \
  "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Now query with the token
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/

# Useful fields
for FIELD in instance-id instance-type placement/availability-zone local-ipv4 public-ipv4 ami-id; do
  echo "$FIELD: $(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/$FIELD)"
done

exit

Checkpoint: all metadata fields return correct values matching your instance.

Break It: Launch, Access, Security Groups, IMDS

Break 1: SSH with wrong username

bash

ssh -i ~/.ssh/ec2-lab ubuntu@$PUBLIC_IP

Observe: Permission denied (publickey). The key is correct but ubuntu user doesn't exist on Amazon Linux. Error looks like a key problem but it's a username problem, a common source of confusion.

Break 2: Remove SSH rule, try to connect

bash

# Get the SSH rule ID
SGR_ID=$(aws ec2 describe-security-group-rules \
  --filters Name=group-id,Values=$SG_ID \
  --region $AWS_REGION \
  --query 'SecurityGroupRules[?FromPort==`22`].SecurityGroupRuleId' \
  --output text)

# Revoke the SSH rule
aws ec2 revoke-security-group-ingress \
  --group-id $SG_ID \
  --security-group-rule-ids $SGR_ID \
  --region $AWS_REGION

# Try to SSH
ssh -i ~/.ssh/ec2-lab -o ConnectTimeout=10 ec2-user@$PUBLIC_IP

Observe: connection times out. The instance is running and healthy, the security group is silently dropping packets. No RST, no error message, just silence. This is what a blocked port looks like vs a closed port (which returns connection refused).

bash

# Restore SSH access
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp --port 22 \
  --cidr $MY_IP/32 \
  --region $AWS_REGION

Break 3: IMDSv1 without token (observe the difference)

bash

ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP

# Try querying IMDS without a token (IMDSv1 style)
curl -s http://169.254.169.254/latest/meta-data/instance-id

exit

Observe: returns 401 if IMDSv2-only is enforced, or returns data if IMDSv1 is still allowed (default on older instances). This is a security posture question. IMDSv2-only prevents SSRF credential theft.

User Data + IAM Instance Profile + S3 Access

Goal

Bootstrap an instance automatically on first boot using user data. Attach an IAM role so the instance can call AWS APIs. Prove it works by calling S3, no hardcoded credentials anywhere.

Estimated time: 1 hour

Write a user data script

What's happening here

User data is a script that runs once on first boot, as root, via cloud-init. It runs before the instance is reachable via SSH. This is how you automate instance configuration without logging in manually. Key behaviors:

Runs only on first boot: not on stop/start, not on reboot. If you want it to run on every boot, use cloud-init's bootcmd directive instead.
Runs as root: no sudo needed inside the script.
Output logs go to /var/log/cloud-init-output.log: this is where you look when user data silently fails.

We'll write a script that installs nginx and serves a page showing the instance's own metadata, so when we hit the web server, we can see which instance responded.

bash

cat > userdata.sh << 'EOF'
#!/bin/bash
set -ex

# Log everything to cloud-init output
exec > >(tee /var/log/userdata.log) 2>&1

# Install nginx
dnf install -y nginx

# Query IMDS for instance identity
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/instance-id)
AZ=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/placement/availability-zone)
INSTANCE_TYPE=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/instance-type)

# Write index page with instance identity
cat > /usr/share/nginx/html/index.html << HTML
<html>
<body>
  <h1>EC2 Lab Instance</h1>
  <p>Instance ID: $INSTANCE_ID</p>
  <p>AZ: $AZ</p>
  <p>Type: $INSTANCE_TYPE</p>
  <p>Launched: $(date)</p>
</body>
</html>
HTML

# Start nginx
systemctl enable nginx
systemctl start nginx

echo "User data complete"
EOF

Create IAM instance profile

What's happening here

An IAM instance profile is a container that holds one IAM role and attaches it to an EC2 instance. When code runs on the instance and calls an AWS SDK or CLI, the credential chain automatically fetches temporary credentials from IMDS (/latest/meta-data/iam/security-credentials/{role-name}). No hardcoded keys, no environment variables, no credential files. Three components:

Trust policy: allows EC2 service to assume this role
Permission policy: what the role can do (S3 read in this case)
Instance profile: the wrapper that attaches the role to an EC2 instance

This is the same concept as the ECS task role from the deployment lab, just the EC2 equivalent.

bash

# Trust policy -- EC2 can assume this role
cat > ec2-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "ec2.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}
EOF

# Create the role
export ROLE_ARN=$(aws iam create-role \
  --role-name ec2-lab-role \
  --assume-role-policy-document file://ec2-trust-policy.json \
  --query 'Role.Arn' --output text)

echo "Role ARN: $ROLE_ARN"

# Attach S3 read access
aws iam attach-role-policy \
  --role-name ec2-lab-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

# Create instance profile
aws iam create-instance-profile \
  --instance-profile-name ec2-lab-profile

# Add role to instance profile
aws iam add-role-to-instance-profile \
  --instance-profile-name ec2-lab-profile \
  --role-name ec2-lab-role

echo "Instance profile created"

Launch instance with user data + instance profile

bash

export INSTANCE_ID_2=$(aws ec2 run-instances \
  --image-id $AMI_ID \
  --instance-type t3.micro \
  --key-name ec2-lab-key \
  --security-group-ids $SG_ID \
  --associate-public-ip-address \
  --iam-instance-profile Name=ec2-lab-profile \
  --user-data file://userdata.sh \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ec2-lab-userdata}]' \
  --region $AWS_REGION \
  --query 'Instances[0].InstanceId' --output text)

echo "Instance ID: $INSTANCE_ID_2"

aws ec2 wait instance-running \
  --instance-ids $INSTANCE_ID_2 --region $AWS_REGION

export PUBLIC_IP_2=$(aws ec2 describe-instances \
  --instance-ids $INSTANCE_ID_2 \
  --query 'Reservations[0].Instances[0].PublicIpAddress' \
  --output text --region $AWS_REGION)

echo "Public IP: $PUBLIC_IP_2"

# Wait ~60s for user data to finish, then hit the web server
sleep 60
curl http://$PUBLIC_IP_2

Checkpoint: web page shows instance ID, AZ, and instance type, populated automatically by user data on first boot. You never logged in.

Call S3 from inside the instance

What's happening here

The AWS CLI inside the instance uses the credential provider chain: environment variables → ~/.aws/credentials → instance profile via IMDS. Since we didn't set env vars or a credentials file, it falls through to IMDS and fetches temporary credentials from the instance profile automatically. The temporary credentials from IMDS rotate every ~6 hours. The SDK/CLI handles refresh transparently. You never deal with expiry in your code. This is the correct way to give EC2 instances AWS permissions. Hardcoding access keys in code or on the instance is a security anti-pattern, keys leak via git, logs, error messages.

bash

ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP_2

bash

# Confirm no credentials configured manually
cat ~/.aws/credentials 2>/dev/null || echo "No credentials file -- good"
echo $AWS_ACCESS_KEY_ID  # Should be empty

# The credential chain resolves to IMDS
aws sts get-caller-identity

# Look at what IMDS is serving
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Get the role name
ROLE=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/)
echo "Role: $ROLE"

# Inspect the temporary credentials
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE | python3 -m json.tool

# List S3 buckets -- uses instance profile credentials automatically
aws s3 ls

# List objects in a specific bucket (use any bucket in your account)
aws s3 ls s3://$(aws s3api list-buckets --query 'Buckets[0].Name' --output text) 2>/dev/null || echo "No buckets or no objects"

exit

Checkpoint: aws sts get-caller-identity shows the ec2-lab-role. S3 commands work. No credentials file exists on the instance.

Inspect user data logs

bash

ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP_2

bash

# Full cloud-init output
sudo cat /var/log/cloud-init-output.log

# Your custom log
sudo cat /var/log/userdata.log

# cloud-init status
cloud-init status

exit

When user data silently fails in production (nginx not running, app not installed) this log is always the first place to look.

Checkpoint: logs show each step of the user data script executing. cloud-init status shows done.

Break It: User Data + IAM Instance Profile + S3 Access

Break 1: Detach instance profile, try S3

bash

# Detach the instance profile
aws ec2 disassociate-iam-instance-profile \
  --association-id $(aws ec2 describe-iam-instance-profile-associations \
    --filters Name=instance-id,Values=$INSTANCE_ID_2 \
    --query 'IamInstanceProfileAssociations[0].AssociationId' \
    --output text --region $AWS_REGION) \
  --region $AWS_REGION

# SSH in and try S3
ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP_2
aws s3 ls
exit

Observe: Unable to locate credentials. The credential chain exhausted all sources, no env vars, no credentials file, no IMDS role. S3 call fails.

bash

# Reattach the profile
aws ec2 associate-iam-instance-profile \
  --instance-id $INSTANCE_ID_2 \
  --iam-instance-profile Name=ec2-lab-profile \
  --region $AWS_REGION

sleep 10

# Confirm S3 works again
ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP_2
aws s3 ls
exit

Break 2: User data doesn't re-run on stop/start

bash

# Stop the instance
aws ec2 stop-instances --instance-ids $INSTANCE_ID_2 --region $AWS_REGION
aws ec2 wait instance-stopped --instance-ids $INSTANCE_ID_2 --region $AWS_REGION

# Start it
aws ec2 start-instances --instance-ids $INSTANCE_ID_2 --region $AWS_REGION
aws ec2 wait instance-running --instance-ids $INSTANCE_ID_2 --region $AWS_REGION

export PUBLIC_IP_2=$(aws ec2 describe-instances \
  --instance-ids $INSTANCE_ID_2 \
  --query 'Reservations[0].Instances[0].PublicIpAddress' \
  --output text --region $AWS_REGION)

sleep 30
curl http://$PUBLIC_IP_2

Observe: nginx still running, page still shows original launch timestamp. User data ran once on first boot (stop/start doesn't re-run it. Note also that the public IP changed) stop/start assigns a new public IP unless you use an Elastic IP.

Storage: EBS + EFS

Goal

Understand the two persistent storage options for EC2. EBS (a single-instance block device. EFS) a shared network file system across multiple instances. Understand when each applies.

Estimated time: 1 – 1.5 hours

EBS

Attach a second EBS volume

What's happening here

Every EC2 instance already has a root EBS volume (the /dev/xvda we saw at launch). Now we're adding a second volume, like adding a second hard drive to a laptop. Key constraints:

EBS volumes live in a single AZ. You can only attach an EBS volume to an instance in the same AZ.
One volume, one instance at a time (standard EBS, Multi-Attach is a niche exception for io2 volumes with clustered filesystems).
Volume persists independently of the instance. You can detach it and reattach to another instance, data survives.

gp3 is the current default volume type: general purpose SSD, 3000 IOPS baseline, cheaper than gp2. Use io2 only for databases needing guaranteed high IOPS.

bash

# Get the AZ of our instance
export INSTANCE_AZ=$(aws ec2 describe-instances \
  --instance-ids $INSTANCE_ID_2 \
  --query 'Reservations[0].Instances[0].Placement.AvailabilityZone' \
  --output text --region $AWS_REGION)

echo "Instance AZ: $INSTANCE_AZ"

# Create a 10GB gp3 volume in the same AZ
export VOL_ID=$(aws ec2 create-volume \
  --availability-zone $INSTANCE_AZ \
  --size 10 \
  --volume-type gp3 \
  --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=ec2-lab-data}]' \
  --region $AWS_REGION \
  --query 'VolumeId' --output text)

echo "Volume ID: $VOL_ID"

aws ec2 wait volume-available --volume-ids $VOL_ID --region $AWS_REGION

# Attach to instance
aws ec2 attach-volume \
  --volume-id $VOL_ID \
  --instance-id $INSTANCE_ID_2 \
  --device /dev/sdf \
  --region $AWS_REGION

sleep 5

aws ec2 describe-volumes \
  --volume-ids $VOL_ID --region $AWS_REGION \
  --query 'Volumes[0].{State:State,Attachments:Attachments[0].State}'

Checkpoint: volume state in-use: attachment state attached.

Format, mount, write data

bash

ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP_2

bash

# Confirm the new device is visible
lsblk

# It shows as nvme1n1 (or xvdf on older instance types)
# AWS maps /dev/sdf -> /dev/nvme1n1 on Nitro instances
export DEVICE=$(lsblk -o NAME,TYPE | grep disk | grep -v nvme0 | awk '{print $1}' | head -1)
echo "Device: /dev/$DEVICE"

# Check -- no filesystem yet
sudo file -s /dev/$DEVICE

# Create ext4 filesystem
sudo mkfs -t ext4 /dev/$DEVICE

# Confirm filesystem created
sudo file -s /dev/$DEVICE

# Mount it
sudo mkdir -p /data
sudo mount /dev/$DEVICE /data

# Verify
df -h /data

# Write some data
echo "Written from instance $HOSTNAME at $(date)" | sudo tee /data/test.txt
sudo ls -la /data/
sudo cat /data/test.txt

exit

Checkpoint: /data mounted, test.txt written.

Persist data across stop/start

bash

# Stop the instance
aws ec2 stop-instances --instance-ids $INSTANCE_ID_2 --region $AWS_REGION
aws ec2 wait instance-stopped --instance-ids $INSTANCE_ID_2 --region $AWS_REGION

# Start it
aws ec2 start-instances --instance-ids $INSTANCE_ID_2 --region $AWS_REGION
aws ec2 wait instance-running --instance-ids $INSTANCE_ID_2 --region $AWS_REGION

export PUBLIC_IP_2=$(aws ec2 describe-instances \
  --instance-ids $INSTANCE_ID_2 \
  --query 'Reservations[0].Instances[0].PublicIpAddress' \
  --output text --region $AWS_REGION)

sleep 20

ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP_2

bash

# Is the volume still attached?
lsblk

# Is it still mounted? (it won't be -- mount doesn't survive reboot without /etc/fstab)
df -h | grep data || echo "Not mounted -- need to remount"

# Remount
export DEVICE=$(lsblk -o NAME,TYPE | grep disk | grep -v nvme0 | awk '{print $1}' | head -1)
sudo mount /dev/$DEVICE /data

# Data still there?
cat /data/test.txt

exit

Key insight

the volume survived stop/start but the mount didn't. To make mounts persist across reboots, add to /etc/fstab. We're skipping that here but it's essential in production.

Checkpoint: test.txt content intact after stop/start.

Detach and reattach to a different instance

What's happening here

This demonstrates EBS as a portable block device. You can detach a volume from one instance and attach it to another, the data moves with the volume. This is how you migrate data between instances, or recover data from a terminated instance's volume. Real use case: your instance is failing and you need to recover data. Detach its root volume, attach to a healthy instance, mount it, copy the data out.

bash

# Unmount first (always unmount before detaching)
ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP_2 'sudo umount /data'

# Detach
aws ec2 detach-volume --volume-id $VOL_ID --region $AWS_REGION
aws ec2 wait volume-available --volume-ids $VOL_ID --region $AWS_REGION

# Attach to the first instance
aws ec2 attach-volume \
  --volume-id $VOL_ID \
  --instance-id $INSTANCE_ID \
  --device /dev/sdf \
  --region $AWS_REGION

sleep 10

# SSH into the first instance and read the data
ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP

bash

export DEVICE=$(lsblk -o NAME,TYPE | grep disk | grep -v nvme0 | awk '{print $1}' | head -1)
sudo mkdir -p /data
sudo mount /dev/$DEVICE /data
cat /data/test.txt

exit

Checkpoint: data written on instance 2 is readable from instance 1 after detach/reattach.

Break It: EBS

Break: Try to attach a volume across AZs

bash

# Get the AZ of the first instance
INSTANCE_1_AZ=$(aws ec2 describe-instances \
  --instance-ids $INSTANCE_ID \
  --query 'Reservations[0].Instances[0].Placement.AvailabilityZone' \
  --output text --region $AWS_REGION)

# Create a volume in a DIFFERENT AZ
OTHER_AZ=$(aws ec2 describe-availability-zones \
  --region $AWS_REGION \
  --query "AvailabilityZones[?ZoneName!='$INSTANCE_1_AZ'].ZoneName" \
  --output text | awk '{print $1}')

echo "Instance AZ: $INSTANCE_1_AZ | Other AZ: $OTHER_AZ"

WRONG_AZ_VOL=$(aws ec2 create-volume \
  --availability-zone $OTHER_AZ \
  --size 5 --volume-type gp3 \
  --region $AWS_REGION \
  --query 'VolumeId' --output text)

aws ec2 wait volume-available --volume-ids $WRONG_AZ_VOL --region $AWS_REGION

# Try to attach -- this will fail
aws ec2 attach-volume \
  --volume-id $WRONG_AZ_VOL \
  --instance-id $INSTANCE_ID \
  --device /dev/sdg \
  --region $AWS_REGION 2>&1 || echo "Failed as expected"

# Cleanup
aws ec2 delete-volume --volume-id $WRONG_AZ_VOL --region $AWS_REGION

Observe: us-east-1a volume cannot attach to a us-east-1b instance. EBS is AZ-scoped. This is the single biggest EBS gotcha in production migrations.

EFS

Create an EFS file system

What's happening here

EFS is a managed NFS (Network File System). Unlike EBS:

Multi-AZ: data is replicated across AZs automatically. No AZ pinning.
Multi-instance: multiple EC2 instances can mount and read/write the same file system simultaneously.
Elastic: no capacity to provision. It grows and shrinks automatically. You pay per GB stored.

EFS uses mount targets: one per AZ, each with its own IP inside the VPC. Instances connect to the mount target in their AZ via NFS (port 2049). So EFS needs a security group that allows NFS from your instance security group. Real use case: an ASG of web servers all need to read/write uploaded files. EBS can't do this (one instance only). S3 can't be mounted natively. EFS is the right tool.

bash

# Security group for EFS -- allow NFS from EC2 instances
export EFS_SG=$(aws ec2 create-security-group \
  --group-name ec2-lab-efs-sg \
  --description "EFS mount target security group" \
  --vpc-id $VPC_ID \
  --region $AWS_REGION \
  --query 'GroupId' --output text)

# Allow NFS (port 2049) from the EC2 instance security group
aws ec2 authorize-security-group-ingress \
  --group-id $EFS_SG \
  --protocol tcp --port 2049 \
  --source-group $SG_ID \
  --region $AWS_REGION

# Create EFS file system
export EFS_ID=$(aws efs create-file-system \
  --performance-mode generalPurpose \
  --throughput-mode bursting \
  --encrypted \
  --tags Key=Name,Value=ec2-lab-efs \
  --region $AWS_REGION \
  --query 'FileSystemId' --output text)

echo "EFS ID: $EFS_ID"

# Wait for it to be available
aws efs describe-file-systems \
  --file-system-id $EFS_ID --region $AWS_REGION \
  --query 'FileSystems[0].{LifeCycleState:LifeCycleState,SizeBytes:SizeInBytes.Value}'

Create mount targets

bash

# Create a mount target in each subnet (one per AZ)
for SUBNET in $(aws ec2 describe-subnets \
  --filters Name=vpc-id,Values=$VPC_ID \
  --query 'Subnets[*].SubnetId' \
  --output text --region $AWS_REGION); do
  echo "Creating mount target in subnet: $SUBNET"
  aws efs create-mount-target \
    --file-system-id $EFS_ID \
    --subnet-id $SUBNET \
    --security-groups $EFS_SG \
    --region $AWS_REGION \
    --query 'MountTargetId' --output text
done

# Wait for mount targets to be available (~30s)
sleep 30

aws efs describe-mount-targets \
  --file-system-id $EFS_ID --region $AWS_REGION \
  --query 'MountTargets[*].{AZ:AvailabilityZoneName,State:LifeCycleState,IP:IpAddress}'

Checkpoint: mount targets in each AZ, all available.

Mount EFS on both instances and test shared access

What's happening here

We mount the same EFS file system on both instances. Whatever one instance writes, the other sees immediately. This is the key differentiator from EBS, true shared storage across instances. We use the EFS DNS name ($EFS_ID.efs.$AWS_REGION.amazonaws.com) which resolves to the mount target IP in the instance's AZ automatically. Same DNS name works for every instance regardless of which AZ it's in.

On instance 1:

bash

ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP

bash

# Install EFS utilities
sudo dnf install -y amazon-efs-utils

# Mount EFS (replace EFS_ID and AWS_REGION with your values)
sudo mkdir -p /shared
sudo mount -t efs -o tls $EFS_ID:/ /shared

# Verify
df -h /shared

# Write from instance 1
echo "Written by instance 1: $(hostname) at $(date)" | sudo tee /shared/from-instance-1.txt
sudo ls /shared/

exit

On instance 2 (new terminal):

bash

ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP_2

bash

sudo dnf install -y amazon-efs-utils
sudo mkdir -p /shared
sudo mount -t efs -o tls $EFS_ID:/ /shared

# Read what instance 1 wrote
cat /shared/from-instance-1.txt

# Write from instance 2
echo "Written by instance 2: $(hostname) at $(date)" | sudo tee /shared/from-instance-2.txt

exit

Back on instance 1:

bash

ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP
cat /shared/from-instance-2.txt
ls /shared/
exit

Checkpoint: both files visible from both instances. EFS is the shared source of truth.

EFS survives instance termination

bash

# Terminate instance 1
aws ec2 terminate-instances --instance-ids $INSTANCE_ID --region $AWS_REGION
aws ec2 wait instance-terminated --instance-ids $INSTANCE_ID --region $AWS_REGION

echo "Instance 1 terminated"

# SSH into instance 2 -- data from instance 1 still there
ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP_2

bash

ls /shared/
cat /shared/from-instance-1.txt
exit

Observe: instance 1 is gone but its data persists in EFS. This is the fundamental promise of a network file system, the data's lifecycle is independent of any individual instance.

Checkpoint: data written by terminated instance 1 is still readable from instance 2.

Break It: EFS

Break: Try to mount EFS without the NFS security group rule

bash

# Temporarily revoke NFS access from instance SG
aws ec2 revoke-security-group-ingress \
  --group-id $EFS_SG \
  --protocol tcp --port 2049 \
  --source-group $SG_ID \
  --region $AWS_REGION

# Try to mount from instance 2
ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP_2

bash

sudo umount /shared 2>/dev/null
sudo mount -t efs -o tls,mounttimeout=10 $EFS_ID:/ /shared 2>&1 || echo "Mount failed as expected"
exit

Observe: mount hangs then times out. NFS traffic on port 2049 is silently dropped by the security group. Same silent-drop behavior as blocking SSH in Launch, Access, Security Groups, IMDS, security groups don't send rejection messages.

bash

# Restore the rule
aws ec2 authorize-security-group-ingress \
  --group-id $EFS_SG \
  --protocol tcp --port 2049 \
  --source-group $SG_ID \
  --region $AWS_REGION

AMI Creation + Immutable Infrastructure

Goal

Snapshot a configured instance into an AMI. Launch a fresh instance from it that comes up pre-configured. Understand why this is the EC2 equivalent of a Docker image.

Estimated time: 45 min

Configure the instance

What's happening here

An AMI is a point-in-time snapshot of an instance's root volume (and any other volumes you choose to include). It captures the OS, all installed packages, configuration files, and application code at the moment of creation. Once you have an AMI, launching 10 instances from it gives you 10 identical, pre-configured instances, no user data script needed for complex setups, no configuration drift between instances. This is the immutable infrastructure pattern: never modify running instances. Instead, bake a new AMI and replace instances. Same philosophy as Docker images, you don't exec into a container and make changes; you update the Dockerfile and rebuild.

bash

ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP_2

bash

# Install additional software to bake into the AMI
sudo dnf install -y nginx htop curl jq

# Write a custom application config
sudo bash -c 'cat > /etc/nginx/conf.d/app.conf << EOF
server {
    listen 8080;
    location /status {
        return 200 "AMI-baked instance running\n";
        add_header Content-Type text/plain;
    }
}
EOF'

# Enable nginx
sudo systemctl enable nginx
sudo systemctl start nginx

# Write a marker file so we can confirm this is an AMI-launched instance
sudo bash -c 'echo "Baked at: $(date)" > /etc/ami-build-info'

echo "Instance configured. Ready to bake AMI."
exit

Create the AMI

bash

export AMI_CUSTOM=$(aws ec2 create-image \
  --instance-id $INSTANCE_ID_2 \
  --name "ec2-lab-baked-$(date +%Y%m%d%H%M)" \
  --description "EC2 lab: nginx + app config baked in" \
  --no-reboot \
  --region $AWS_REGION \
  --query 'ImageId' --output text)

echo "Custom AMI ID: $AMI_CUSTOM"

# Wait for AMI to be available (~2-3 min)
aws ec2 wait image-available \
  --image-ids $AMI_CUSTOM --region $AWS_REGION

aws ec2 describe-images \
  --image-ids $AMI_CUSTOM --region $AWS_REGION \
  --query 'Images[0].{Name:Name,State:State,CreationDate:CreationDate}'

--no-reboot creates the AMI without stopping the instance first. Faster, but there's a small risk of filesystem inconsistency if writes are in progress. For production AMI baking, stop the instance first or use an application-consistent snapshot.

Checkpoint: custom AMI in available state.

Launch from the custom AMI

bash

export INSTANCE_ID_3=$(aws ec2 run-instances \
  --image-id $AMI_CUSTOM \
  --instance-type t3.micro \
  --key-name ec2-lab-key \
  --security-group-ids $SG_ID \
  --associate-public-ip-address \
  --iam-instance-profile Name=ec2-lab-profile \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ec2-lab-from-ami}]' \
  --region $AWS_REGION \
  --query 'Instances[0].InstanceId' --output text)

aws ec2 wait instance-running \
  --instance-ids $INSTANCE_ID_3 --region $AWS_REGION

export PUBLIC_IP_3=$(aws ec2 describe-instances \
  --instance-ids $INSTANCE_ID_3 \
  --query 'Reservations[0].Instances[0].PublicIpAddress' \
  --output text --region $AWS_REGION)

echo "Instance 3 IP: $PUBLIC_IP_3"

sleep 20

# Hit the baked nginx config -- no user data needed
curl http://$PUBLIC_IP_3:8080/status

# Confirm the build marker
ssh -i ~/.ssh/ec2-lab ec2-user@$PUBLIC_IP_3 'cat /etc/ami-build-info && nginx -v'

Checkpoint: new instance comes up with nginx running and custom config in place, zero manual setup. The AMI carried everything.

Break It: AMI

Break: Override AMI config with user data

bash

# Launch from the same AMI but override nginx config via user data
cat > override-userdata.sh << 'EOF'
#!/bin/bash
# Replace the baked config with something different
cat > /etc/nginx/conf.d/app.conf << NGINX
server {
    listen 8080;
    location /status {
        return 200 "Overridden by user data!\n";
        add_header Content-Type text/plain;
    }
}
NGINX
systemctl reload nginx
EOF

export INSTANCE_ID_4=$(aws ec2 run-instances \
  --image-id $AMI_CUSTOM \
  --instance-type t3.micro \
  --key-name ec2-lab-key \
  --security-group-ids $SG_ID \
  --associate-public-ip-address \
  --user-data file://override-userdata.sh \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ec2-lab-ami-override}]' \
  --region $AWS_REGION \
  --query 'Instances[0].InstanceId' --output text)

aws ec2 wait instance-running --instance-ids $INSTANCE_ID_4 --region $AWS_REGION

export PUBLIC_IP_4=$(aws ec2 describe-instances \
  --instance-ids $INSTANCE_ID_4 \
  --query 'Reservations[0].Instances[0].PublicIpAddress' \
  --output text --region $AWS_REGION)

sleep 30
curl http://$PUBLIC_IP_4:8080/status

Observe: AMI = base layer. User data = runtime override layer. They compose. This is the EC2 equivalent of a Docker image (AMI) + container entrypoint override (user data).

Launch Templates + Auto Scaling Groups + ALB

Goal

Codify instance configuration in a launch template. Build an ASG that self-heals and scales. Put it behind an ALB. Trigger instance refresh (rolling replacement) by updating the launch template.

Estimated time: 1 – 1.5 hours

Create a launch template

What's happening here

A launch template is the modern way to define instance configuration for an ASG, it replaces the deprecated launch configuration. It captures AMI, instance type, key pair, security groups, IAM profile, user data, and storage config in a versioned document. Versioning is key: when you update a launch template (new AMI, updated user data), you create a new version. The ASG can then perform an instance refresh: a controlled rolling replacement of all instances using the new version. Same concept as an ECS rolling deployment, but at the VM level.

bash

export LT_ID=$(aws ec2 create-launch-template \
  --launch-template-name ec2-lab-lt \
  --version-description "v1 - initial" \
  --launch-template-data "{
    \"ImageId\": \"$AMI_CUSTOM\",
    \"InstanceType\": \"t3.micro\",
    \"KeyName\": \"ec2-lab-key\",
    \"SecurityGroupIds\": [\"$SG_ID\"],
    \"IamInstanceProfile\": {\"Name\": \"ec2-lab-profile\"},
    \"UserData\": \"$(base64 -w0 userdata.sh 2>/dev/null || base64 userdata.sh)\",
    \"TagSpecifications\": [{\"ResourceType\": \"instance\", \"Tags\": [{\"Key\": \"Name\", \"Value\": \"ec2-lab-asg\"}]}]
  }" \
  --region $AWS_REGION \
  --query 'LaunchTemplate.LaunchTemplateId' --output text)

echo "Launch Template ID: $LT_ID"

aws ec2 describe-launch-templates \
  --launch-template-ids $LT_ID --region $AWS_REGION \
  --query 'LaunchTemplates[0].{Name:LaunchTemplateName,Version:LatestVersionNumber}'

Create ALB + target group

bash

export ASG_ALB_ARN=$(aws elbv2 create-load-balancer \
  --name ec2-lab-alb \
  --subnets $(aws ec2 describe-subnets \
    --filters Name=vpc-id,Values=$VPC_ID \
    --query 'Subnets[*].SubnetId' \
    --output text --region $AWS_REGION | tr '\t' ' ') \
  --security-groups $SG_ID \
  --region $AWS_REGION \
  --query 'LoadBalancers[0].LoadBalancerArn' --output text)

export ASG_ALB_DNS=$(aws elbv2 describe-load-balancers \
  --load-balancer-arns $ASG_ALB_ARN --region $AWS_REGION \
  --query 'LoadBalancers[0].DNSName' --output text)

export ASG_TG_ARN=$(aws elbv2 create-target-group \
  --name ec2-lab-asg-tg \
  --protocol HTTP --port 80 \
  --vpc-id $VPC_ID --target-type instance \
  --health-check-path / \
  --health-check-interval-seconds 15 \
  --healthy-threshold-count 2 \
  --unhealthy-threshold-count 3 \
  --region $AWS_REGION \
  --query 'TargetGroups[0].TargetGroupArn' --output text)

aws elbv2 create-listener \
  --load-balancer-arn $ASG_ALB_ARN \
  --protocol HTTP --port 80 \
  --default-actions Type=forward,TargetGroupArn=$ASG_TG_ARN \
  --region $AWS_REGION

echo "ALB DNS: $ASG_ALB_DNS"

Create the Auto Scaling Group

What's happening here

The ASG is the self-healing layer. It maintains a desired count of instances at all times:

If an instance fails its ALB health check and is terminated, ASG launches a replacement.
If CPU spikes above a threshold, ASG scales out (launches more instances).
If CPU drops, ASG scales in (terminates excess instances).

Key parameters:

min=1, max=3, desired=2: always 2 running, can scale between 1 and 3.
health-check-type ELB, ASG uses the ALB health check to determine instance health, not just EC2 system status. A running instance that fails /health is replaced.
health-check-grace-period 120: wait 2 minutes after launch before checking health. Gives user data time to finish.

bash

export SUBNET_LIST=$(aws ec2 describe-subnets \
  --filters Name=vpc-id,Values=$VPC_ID \
  --query 'Subnets[*].SubnetId' \
  --output text --region $AWS_REGION | tr '\t' ',')

aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name ec2-lab-asg \
  --launch-template LaunchTemplateId=$LT_ID,Version='$Latest' \
  --min-size 1 --max-size 3 --desired-capacity 2 \
  --target-group-arns $ASG_TG_ARN \
  --health-check-type ELB \
  --health-check-grace-period 120 \
  --vpc-zone-identifier $SUBNET_LIST \
  --tags Key=Name,Value=ec2-lab-asg,PropagateAtLaunch=true \
  --region $AWS_REGION

echo "Waiting for instances to be healthy..."
aws autoscaling wait group-in-service \
  --auto-scaling-group-name ec2-lab-asg 2>/dev/null || sleep 120

# Check ASG status
aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names ec2-lab-asg --region $AWS_REGION \
  --query 'AutoScalingGroups[0].{desired:DesiredCapacity,min:MinSize,max:MaxSize,instances:Instances[*].{id:InstanceId,state:LifecycleState,health:HealthStatus}}'

# Hit the ALB
curl http://$ASG_ALB_DNS

Checkpoint: 2 instances running, ALB returns nginx page, ASG shows both instances InService.

Self-healing: terminate an instance manually

bash

# Get one instance ID from the ASG
INSTANCE_TO_KILL=$(aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names ec2-lab-asg --region $AWS_REGION \
  --query 'AutoScalingGroups[0].Instances[0].InstanceId' --output text)

echo "Terminating: $INSTANCE_TO_KILL"

aws ec2 terminate-instances \
  --instance-ids $INSTANCE_TO_KILL --region $AWS_REGION

# Watch ASG respond
for i in $(seq 1 8); do
  echo "=== Poll $i ==="
  aws autoscaling describe-auto-scaling-groups \
    --auto-scaling-group-names ec2-lab-asg --region $AWS_REGION \
    --query 'AutoScalingGroups[0].Instances[*].{id:InstanceId,state:LifecycleState,health:HealthStatus}'
  sleep 15
done

Observe: terminated instance disappears. ASG detects desired=2 but running=1. Launches a replacement. Within 2-3 minutes, back to 2 healthy instances. The ALB continues serving traffic from the surviving instance during replacement.

Checkpoint: ASG returns to 2 instances automatically.

Instance refresh (rolling replacement via launch template update)

What's happening here

When you update a launch template (new AMI, updated config), existing instances are still running the old version. Instance refresh is the mechanism to roll out the new version across the ASG, it replaces instances in batches, respecting the minimum healthy percentage so there's no downtime. This is the EC2 equivalent of an ECS rolling deployment. Same concept: drain old, launch new, health check, shift traffic.

bash

# Create a new launch template version with updated user data
cat > userdata-v2.sh << 'EOF'
#!/bin/bash
dnf install -y nginx
echo "<h1>Version 2 - from instance refresh</h1><p>Instance: $(curl -s http://169.254.169.254/latest/meta-data/instance-id)</p>" > /usr/share/nginx/html/index.html
systemctl enable nginx && systemctl start nginx
EOF

aws ec2 create-launch-template-version \
  --launch-template-id $LT_ID \
  --version-description "v2 - updated user data" \
  --source-version 1 \
  --launch-template-data "{\"UserData\": \"$(base64 -w0 userdata-v2.sh 2>/dev/null || base64 userdata-v2.sh)\"}"\
  --region $AWS_REGION

# Trigger instance refresh with 50% min healthy
aws autoscaling start-instance-refresh \
  --auto-scaling-group-name ec2-lab-asg \
  --preferences MinHealthyPercentage=50,InstanceWarmup=60 \
  --desired-configuration LaunchTemplate="{LaunchTemplateId=$LT_ID,Version='$Latest'}" \
  --region $AWS_REGION

# Watch the refresh
for i in $(seq 1 12); do
  echo "=== Poll $i ==="
  aws autoscaling describe-instance-refreshes \
    --auto-scaling-group-name ec2-lab-asg --region $AWS_REGION \
    --query 'InstanceRefreshes[0].{Status:Status,PercentComplete:PercentageComplete,Remaining:InstancesToUpdate}'
  curl -s http://$ASG_ALB_DNS | grep -o '<h1>.*</h1>' || echo "(no h1)"
  sleep 20
done

Observe: instances are replaced one at a time. During the refresh, the ALB may return both "old" and "v2" responses as it round-robins between old and new instances, exactly like ECS rolling deployment.

Checkpoint: all instances replaced, ALB returns "Version 2" consistently.

Break It: ASG

Break: Set desired count below min

bash

aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name ec2-lab-asg \
  --desired-capacity 0 \
  --region $AWS_REGION 2>&1 || echo "Failed as expected"

Observe: ValidationError: DesiredCapacity must be between MinSize (1) and MaxSize (3). ASG enforces the min/max bounds, you can't set desired below min. This is the safety rail that prevents accidentally scaling to zero in production.

Modern Access: SSM Session Manager + EC2 Instance Connect

Goal

Eliminate SSH keys and open port 22 entirely. Access instances via SSM Session Manager (no open ports, full audit trail) and EC2 Instance Connect (browser-based, short-lived keys). Understand why this is the correct production access pattern.

Estimated time: 45 min

The problem with SSH + port 22

What's happening here

Traditional SSH has three problems in production:

Port 22 must be open: even restricted to your IP, it's a persistent attack surface. IPs change, rules get misconfigured.
Key management: who has keys? What happens when someone leaves the team? Keys don't expire.
No audit trail: you know someone SSH'd in, but not what commands they ran.

SSM Session Manager solves all three:

No port 22: the instance connects outbound to the SSM service. No inbound port needed.
IAM-controlled access: who can session is controlled by IAM policy, not SSH keys. Revoke access by removing IAM permission.
Full audit trail: every command logged to CloudWatch/S3.

Set up SSM Session Manager

bash

# Attach SSM managed policy to the instance role
aws iam attach-role-policy \
  --role-name ec2-lab-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

# Close port 22 entirely
SGR_SSH=$(aws ec2 describe-security-group-rules \
  --filters Name=group-id,Values=$SG_ID \
  --region $AWS_REGION \
  --query 'SecurityGroupRules[?FromPort==`22`].SecurityGroupRuleId' \
  --output text)

if [ -n "$SGR_SSH" ]; then
  aws ec2 revoke-security-group-ingress \
    --group-id $SG_ID \
    --security-group-rule-ids $SGR_SSH \
    --region $AWS_REGION
  echo "Port 22 closed"
fi

# Verify SSH is blocked
ssh -i ~/.ssh/ec2-lab -o ConnectTimeout=5 ec2-user@$PUBLIC_IP_2 2>&1 || echo "SSH blocked -- as expected"

Connect via SSM Session Manager

bash

# Install Session Manager plugin (if not already installed)
# macOS:
curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/mac/sessionmanager-bundle.zip" \
  -o sessionmanager-bundle.zip
unzip sessionmanager-bundle.zip
sudo ./sessionmanager-bundle/install -i /usr/local/sessionmanagerplugin -b /usr/local/bin/session-manager-plugin

# Linux:
curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/ubuntu_64bit/session-manager-plugin.deb" \
  -o session-manager-plugin.deb
sudo dpkg -i session-manager-plugin.deb

# Wait for SSM agent to register (~60s after policy attach)
sleep 60

# Check if instance is registered with SSM
aws ssm describe-instance-information \
  --region $AWS_REGION \
  --query 'InstanceInformationList[?InstanceId==`'$INSTANCE_ID_2'`].{Id:InstanceId,Status:PingStatus,Agent:AgentVersion}'

# Start a session -- no port 22, no SSH key
aws ssm start-session \
  --target $INSTANCE_ID_2 \
  --region $AWS_REGION

Inside the session:

bash

whoami
hostname
uname -a

# Run any command -- full shell
aws s3 ls  # Instance profile still works

# Check who you are via IAM
aws sts get-caller-identity

exit

Checkpoint: full shell on the instance with no open ports and no SSH key. Access controlled by IAM.

EC2 Instance Connect

What's happening here

EC2 Instance Connect is a different mechanism: it generates a one-time SSH public key: pushes it to the instance via the EC2 API (valid for 60 seconds), and then opens a standard SSH connection. After 60 seconds the temporary key is invalid. Advantages over traditional SSH:

No persistent key pair to manage
Access controlled by IAM (ec2-instance-connect:SendSSHPublicKey permission)

Requirements: port 22 must be open (or you use the browser-based console option which tunnels through AWS). For the pure no-open-port model, SSM Session Manager is better. This is why SSM Session Manager is the preferred production pattern. EC2 Instance Connect still needs port 22 for CLI use.

bash

# Re-open port 22 temporarily for EC2 Instance Connect CLI demo
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp --port 22 \
  --cidr 18.206.107.24/29 \
  --region $AWS_REGION  # EC2 Instance Connect IP range for us-east-1

# Send a temporary public key and SSH
aws ec2-instance-connect send-ssh-public-key \
  --instance-id $INSTANCE_ID_2 \
  --instance-os-user ec2-user \
  --ssh-public-key file://~/.ssh/ec2-lab.pub \
  --region $AWS_REGION

# SSH within 60 seconds
ssh -i ~/.ssh/ec2-lab \
  -o StrictHostKeyChecking=no \
  ec2-user@$PUBLIC_IP_2

# After 60s the temporary key is invalid -- try reconnecting to observe
exit

In the AWS console: EC2 → Instances → Connect → EC2 Instance Connect → browser-based terminal. No CLI setup needed, works entirely through the browser. Useful for quick access when SSM isn't set up yet.

Checkpoint: connected via one-time key. SSM preferred for production. EC2 Instance Connect useful as a fallback.

Break It: Modern Access: SSM Session Manager + EC2 Instance Connect

Break: SSM without the managed policy

bash

# Detach the SSM policy
aws iam detach-role-policy \
  --role-name ec2-lab-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

# Wait for the instance to deregister from SSM (~2 min)
sleep 120

# Check SSM registration
aws ssm describe-instance-information \
  --region $AWS_REGION \
  --query 'InstanceInformationList[?InstanceId==`'$INSTANCE_ID_2'`]'

# Try to start a session
aws ssm start-session \
  --target $INSTANCE_ID_2 \
  --region $AWS_REGION 2>&1 || echo "Session failed as expected"

# Reattach
aws iam attach-role-policy \
  --role-name ec2-lab-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

Observe: without AmazonSSMManagedInstanceCore: the SSM agent on the instance can't communicate with the SSM service. The instance disappears from SSM inventory. Session start fails. This is the single most common SSM setup mistake.

Bastion Hosts & Private Resource Access

Goal

Understand why bastion hosts exist, how they differ from Modern Access: SSM Session Manager + EC2 Instance Connect's SSM access pattern, and how to reach private resources (RDS, ElastiCache, private EC2) without exposing them to the internet. This is the access model used in the ElastiCache and other database labs.

Estimated time: 30 min (conceptual, no new resources to create in this session)

Why bastions exist

Some AWS resources have no public endpoint by design: ElastiCache Redis, RDS databases, and EC2 instances in private subnets all live on private IPs inside a VPC. Their DNS names resolve to addresses that are only routable within the VPC, not from your laptop on the internet. That is intentional. Exposing Redis or a database to 0.0.0.0/0 is a critical security risk. The practical problem during development: how do you connect from your laptop to inspect keys, run queries, or debug behaviour? A bastion (jump host) is an EC2 instance that acts as a relay: reachable from you (via SSM, as in Modern Access: SSM Session Manager + EC2 Instance Connect) and inside the VPC, so it can reach private resources on your behalf.

text

Your laptop
    ↓
  SSM / internet
    ↓
 EC2 Bastion  ← relay (inside VPC)
    ↓
  VPC-internal network
    ↓
ElastiCache / RDS / private EC2

Mental model

Think of a gated apartment complex. You cannot walk directly to your friend's apartment (Redis) because the gate is locked. The security guard (bastion) inside the complex can reach every apartment. You reach the guard via SSM; the guard relays your request. The gate stays locked.

The source security group model

Launch, Access, Security Groups, IMDS opened SSH to your IP (203.0.113.42/32). That breaks when your IP changes and does not help teammates. Opening Redis to your IP has the same problem. The production pattern: reference a security group as the source, not an IP range.

text

Client SG (CLIENT_SG):
  No inbound rules
  Attached to: bastion, Lambda, ECS tasks

Redis SG (REDIS_SG):
  Inbound: TCP 6379 from CLIENT_SG
  Attached to: ElastiCache cluster

Now the bastion (CLIENT_SG) can reach Redis. Your Lambda (CLIENT_SG) can reach Redis. Anything not in CLIENT_SG cannot, even other VPC instances in a different SG. The internet cannot reach Redis at all. Add a new service that needs Redis? Attach CLIENT_SG. No rule changes, no IP management.

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

The bastion is not in your production path

text

Production traffic:
  User → API Gateway → Lambda (CLIENT_SG) → Redis

Debug / dev traffic:
  You → SSM → Bastion (CLIENT_SG) → Redis

Both paths use CLIENT_SG. The bastion does not proxy production traffic, it is only you, an engineer, reaching a private resource during development. Terminate it when you are done.

SSH tunnels vs SSM for bastions

Launch, Access, Security Groups, IMDS used SSH directly; Modern Access: SSM Session Manager + EC2 Instance Connect replaced SSH with SSM on your lab instance. For bastions, both patterns exist:

SSH tunnel: map a local port through the bastion to a private endpoint (still needs a key pair and port 22): ssh -i key.pem -L 6379:<redis-endpoint>:6379 ec2-user@<bastion-ip> -N Then redis-cli -h 127.0.0.1 on your laptop.

SSM Session Manager (used in all course labs), no key pair, no port 22, no inbound rules on the bastion. Connect with aws ssm start-session --target $BASTION_ID: then run redis-cli from inside the bastion shell.

Bastion vs other access patterns

Pattern	How it works	Best for	Downsides
EC2 Bastion (SSH)	SSH to public EC2, then connect to private resource	Simple setups, occasional access	Key management, port 22 exposure
EC2 Bastion (SSM)	SSM Session Manager to EC2, no SSH	Labs, team access, no key management	Instance must be running (small cost)
AWS Client VPN	VPN client puts your laptop inside the VPC	Teams needing frequent VPC access	Setup complexity, hourly VPN endpoint cost
SSH tunnel only	SSH port-forward to local port	Local GUI tools (DB clients, redis-commander)	Still needs SSH key and bastion
AWS CloudShell	Browser terminal with VPC access	Quick CLI commands, no EC2 cost	Limited tools, session timeout

Debugging a connection timeout to a private resource

When redis-cli or psql hangs silently, work through these layers in order. The symptom is identical regardless of which layer is wrong.

1Is the bastion in CLIENT_SG?: aws ec2 describe-instances --instance-ids $BASTION_ID --query 'Reservations[0].Instances[0].SecurityGroups'
2Does the resource SG allow CLIENT_SG on the right port? (aws ec2 describe-security-groups --group-ids $REDIS_SG --query 'SecurityGroups[0].IpPermissions') look for UserIdGroupPairs containing CLIENT_SG
3Are both in the same VPC?: compare bastion VpcId with the resource subnet group VPC
4Does the endpoint resolve from inside the bastion?: nslookup <endpoint> via SSM shell; must return a private 10.x IP
5Are NACLs blocking traffic?: NACLs are stateless; both inbound and outbound must allow port 6379 (or DB port) and ephemeral return ports

Common bastion issues

Symptom	Root cause	Fix
`aws ssm start-session` hangs	SSM Agent not registered yet	Wait 2–3 min after launch; check `aws ssm describe-instance-information`
`TargetNotConnected`	Instance profile missing or wrong role	Instance must have `AmazonSSMManagedInstanceCore` on its IAM role
`AccessDeniedException` on start-session	Your IAM user lacks `ssm:StartSession`	Attach `AmazonSSMFullAccess` or scoped session policy
Client hangs on connect to Redis/RDS	SG rule wrong port/source, or bastion in wrong SG	Work through the 5-layer debug above
Works from bastion but not Lambda	Lambda not in CLIENT_SG or wrong VPC	Check `aws lambda get-function-configuration --query 'VpcConfig'`

Cost and lifecycle

t3.micro costs ~$0.0104/hour in ap-south-1: within Free Tier for lab sessions
Terminate the bastion when done, it is stateless; cleanup scripts in other labs handle this
Re-creating a bastion next session takes ~3 minutes

Quick reference: launch a bastion

Other labs (ElastiCache, RDS) use this pattern. The instance needs an IAM role with AmazonSSMManagedInstanceCore and membership in CLIENT_SG.

bash

# Launch bastion (Amazon Linux 2023, SSM-enabled)
AMI_ID=$(aws ec2 describe-images \
  --owners amazon \
  --filters Name=name,Values="al2023-ami-2023*-x86_64" Name=state,Values=available \
  --query 'sort_by(Images,&CreationDate)[-1].ImageId' \
  --output text --region $AWS_REGION)

export BASTION_ID=$(aws ec2 run-instances \
  --image-id $AMI_ID \
  --instance-type t3.micro \
  --iam-instance-profile Name=your-ssm-instance-profile \
  --security-group-ids $CLIENT_SG \
  --subnet-id $SUBNET_ID \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=lab-bastion}]' \
  --user-data $'#!/bin/bash\nyum install -y redis6\n' \
  --region $AWS_REGION \
  --query 'Instances[0].InstanceId' --output text)

aws ec2 wait instance-running --instance-ids $BASTION_ID --region $AWS_REGION

# Wait ~2 min, then connect
aws ssm start-session --target $BASTION_ID --region $AWS_REGION

# Terminate when done
aws ec2 terminate-instances --instance-ids $BASTION_ID --region $AWS_REGION

Cleanup

bash

# 1. Delete ASG (scales down instances automatically)
aws autoscaling delete-auto-scaling-group \
  --auto-scaling-group-name ec2-lab-asg \
  --force-delete --region $AWS_REGION

# 2. Delete ALB + listener + target group
LISTENER=$(aws elbv2 describe-listeners \
  --load-balancer-arn $ASG_ALB_ARN --region $AWS_REGION \
  --query 'Listeners[0].ListenerArn' --output text)
aws elbv2 delete-listener --listener-arn $LISTENER --region $AWS_REGION
aws elbv2 delete-load-balancer --load-balancer-arn $ASG_ALB_ARN --region $AWS_REGION
sleep 30
aws elbv2 delete-target-group --target-group-arn $ASG_TG_ARN --region $AWS_REGION

# 3. Terminate remaining instances
for ID in $INSTANCE_ID $INSTANCE_ID_2 $INSTANCE_ID_3 $INSTANCE_ID_4; do
  aws ec2 terminate-instances --instance-ids $ID --region $AWS_REGION 2>/dev/null
done
aws ec2 wait instance-terminated \
  --instance-ids $INSTANCE_ID_2 $INSTANCE_ID_3 --region $AWS_REGION 2>/dev/null

# 4. Delete EBS volume
aws ec2 detach-volume --volume-id $VOL_ID --region $AWS_REGION 2>/dev/null
sleep 10
aws ec2 delete-volume --volume-id $VOL_ID --region $AWS_REGION 2>/dev/null

# 5. Delete EFS
for MT in $(aws efs describe-mount-targets \
  --file-system-id $EFS_ID --region $AWS_REGION \
  --query 'MountTargets[*].MountTargetId' --output text); do
  aws efs delete-mount-target --mount-target-id $MT --region $AWS_REGION
done
sleep 30
aws efs delete-file-system --file-system-id $EFS_ID --region $AWS_REGION

# 6. Delete launch template
aws ec2 delete-launch-template \
  --launch-template-id $LT_ID --region $AWS_REGION

# 7. Deregister custom AMI + delete snapshot
SNAPSHOT=$(aws ec2 describe-images \
  --image-ids $AMI_CUSTOM --region $AWS_REGION \
  --query 'Images[0].BlockDeviceMappings[0].Ebs.SnapshotId' --output text)
aws ec2 deregister-image --image-id $AMI_CUSTOM --region $AWS_REGION
aws ec2 delete-snapshot --snapshot-id $SNAPSHOT --region $AWS_REGION

# 8. Delete key pair
aws ec2 delete-key-pair --key-name ec2-lab-key --region $AWS_REGION

# 9. Delete security groups
aws ec2 delete-security-group --group-id $EFS_SG --region $AWS_REGION
aws ec2 delete-security-group --group-id $SG_ID --region $AWS_REGION

# 10. IAM cleanup
for POLICY in AmazonS3ReadOnlyAccess AmazonSSMManagedInstanceCore; do
  aws iam detach-role-policy \
    --role-name ec2-lab-role \
    --policy-arn arn:aws:iam::aws:policy/$POLICY 2>/dev/null
done
aws iam remove-role-from-instance-profile \
  --instance-profile-name ec2-lab-profile \
  --role-name ec2-lab-role
aws iam delete-instance-profile --instance-profile-name ec2-lab-profile
aws iam delete-role --role-name ec2-lab-role

echo "All resources deleted."