PreviewLab8.5 hours

Redis Caching

Deploy ElastiCache Redis in a VPC, implement cache-aside patterns, and practice TTL, pub/sub, replication, and distributed locks.

Mental model

ElastiCache Redis is not a database. It's a speed layer in front of your database. Your DB is the source of truth. Redis holds hot copies of frequently-read data with expiry. A cache miss is a normal event, not a failure. Every key must have a TTL. The cache can always be rebuilt from the DB.

Prerequisites

0 of 5 checked

Concept: EC2 Bastion

Read this before Session 0.

This lab uses an EC2 bastion to connect to Redis. For the full bastion and CLIENT_SG model, read Session 7 of Launch to Production Patterns. The sections below are a condensed recap tailored to this lab.

What is a bastion?

A bastion (also called a jump host or jump box) is an EC2 instance whose sole purpose is to act as a bridge between you and resources that have no public access. ElastiCache Redis has no public endpoint, its DNS resolves to a private IP inside your VPC. You cannot connect to it from your laptop directly, because your laptop is on the internet and the Redis IP is not routable outside the VPC. The connection path with a bastion:

javascript

Your laptop  →  (internet)  →  EC2 bastion (in VPC)  →  (VPC-internal)  →  ElastiCache Redis

The bastion is the only thing with both a public presence (reachable from your laptop) and access inside the VPC (can reach Redis). It acts as a relay.

Why not just open Redis to the internet?

Short answer: never do this. A Redis instance exposed to the internet with no auth will be compromised within minutes by automated scanners. Even with AUTH, exposing port 6379 publicly is a significant attack surface. Redis was designed to run inside a trusted network boundary, the VPC is that boundary.

Two ways to reach a bastion

Traditional: SSH

The classic approach, you generate an EC2 key pair, download the .pem file, and run:

bash

ssh -i key.pem ec2-user@<bastion-public-ip>

# Then from inside the bastion:
redis-cli -h <redis-endpoint> -p 6379

To also forward a local port to Redis (so your laptop's redis-cli connects as if Redis is local):

bash

# SSH tunnel: maps localhost:6379 to the Redis endpoint via the bastion
ssh -i key.pem -L 6379:<redis-endpoint>:6379 ec2-user@<bastion-public-ip> -N

# Now on your laptop:
redis-cli -h 127.0.0.1 -p 6379

Downsides of SSH: you must manage key pairs (generate, distribute, rotate, never lose them), open port 22 on the bastion's security group, and handle firewall rules. Key management becomes a security problem at scale.

Modern: SSM Session Manager

AWS Systems Manager (SSM) Session Manager replaces SSH entirely. No key pairs, no port 22, no inbound rules on the bastion's security group. Instead:

The bastion runs the SSM Agent (pre-installed on Amazon Linux 2023)
The bastion's IAM role has AmazonSSMManagedInstanceCore attached
SSM Agent connects outbound to the SSM service (port 443, already open for outbound)
You run aws ssm start-session --target <instance-id> from your laptop
AWS routes the session through SSM's infrastructure, no direct connection to the bastion needed

bash

# Connect to the bastion (no key pair, no port 22 needed)
aws ssm start-session --target $BASTION_ID --region $AWS_REGION

# You're now inside the bastion
# Run redis-cli from here
redis-cli -h $REDIS_HOST -p 6379 --tls -a $AUTH_TOKEN

Why this is better

the bastion's security group has zero inbound rules. The only traffic that reaches it is outbound HTTPS to SSM. An attacker scanning port 22 finds nothing. No key files to lose or rotate. This lab uses SSM Session Manager throughout.

What the bastion is NOT

The bastion is not part of your application. Your application (Lambda, ECS tasks, EC2 app servers) connects to Redis directly from inside the VPC using the client security group (CLIENT_SG). The bastion is only for you (the engineer) to manually inspect Redis during development and debugging. In production:

javascript

Your app (Lambda/ECS)  →  CLIENT_SG  →  Redis SG (port 6379)  →  Redis
Your laptop (you)       →  SSM       →  EC2 bastion (CLIENT_SG)  →  Redis

Both paths use CLIENT_SG as the source. The Redis security group only allows inbound from CLIENT_SG: so both your app and your bastion can reach Redis, and nothing else can.

Security group model visualised

The REDIS_SG has one inbound rule: TCP 6379 from CLIENT_SG. Anything in CLIENT_SG (bastion, Lambda, ECS) can reach Redis. Nothing else can, not even other instances in the same VPC if they're in a different SG.

Bastion cost

A t3.micro bastion costs roughly $0.0104/hour in ap-south-1 (~₹0.87/hour). Over an 8-hour lab session: ~$0.08. It falls within Free Tier (750 hours/month of t3.micro). Terminate it when you're done, the cleanup script handles this.

Common bastion issues and fixes

IAM, Network, and the Redis Connection Model

Goal

Understand why ElastiCache is the most network-sensitive AWS service. Build the VPC, security group, and subnet group infrastructure before creating any Redis cluster. Get this wrong and nothing works, and the error messages won't tell you why.

Estimated time: 60 min

The ElastiCache network model

What's happening here

ElastiCache clusters live entirely inside your VPC. There is no public endpoint. You cannot connect from your laptop directly. The connection path is always:

javascript

Your app / Lambda / EC2  →  (same VPC)  →  ElastiCache cluster

Three things must be right simultaneously for a connection to work:

VPC: client and cluster in the same VPC
Security Group: cluster SG must allow inbound TCP 6379 from the client's SG (not from 0.0.0.0/0, from a specific source SG)
Subnet Group: ElastiCache must know which subnets to place nodes in

If any of these three is wrong, the connection hangs with a timeout. Redis does not send a rejection, the packet never arrives. This is why "I can't connect to Redis" is always a network/SG debugging exercise, not a Redis one.

IAM scope: IAM controls the ElastiCache control plane (create/delete/describe clusters). The data plane (GET/SET commands) uses Redis AUTH tokens, not IAM. There is a newer IAM auth feature for Redis 7+ via RBAC, but AUTH tokens are the standard pattern covered here.

bash

export AWS_REGION=ap-south-1
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

# Get the default VPC
export VPC_ID=$(aws ec2 describe-vpcs \
  --filters Name=isDefault,Values=true \
  --region $AWS_REGION \
  --query 'Vpcs[0].VpcId' --output text)

echo "VPC: $VPC_ID"

# Get all subnets in the default VPC
export SUBNET_IDS=$(aws ec2 describe-subnets \
  --filters Name=vpc-id,Values=$VPC_ID \
  --region $AWS_REGION \
  --query 'Subnets[*].SubnetId' --output text | tr '\t' ',')

echo "Subnets: $SUBNET_IDS"

# Get individual subnet IDs for later use
export SUBNET_1=$(aws ec2 describe-subnets \
  --filters Name=vpc-id,Values=$VPC_ID \
  --region $AWS_REGION \
  --query 'Subnets[0].SubnetId' --output text)

export SUBNET_2=$(aws ec2 describe-subnets \
  --filters Name=vpc-id,Values=$VPC_ID \
  --region $AWS_REGION \
  --query 'Subnets[1].SubnetId' --output text)

echo "Subnet 1: $SUBNET_1"
echo "Subnet 2: $SUBNET_2"

Checkpoint: VPC ID and at least 2 subnet IDs are set.

Create Security Groups

bash

# Security group for the Redis cluster
export REDIS_SG=$(aws ec2 create-security-group \
  --group-name redis-lab-cluster-sg \
  --description "ElastiCache Redis cluster - allow inbound 6379 from client SG" \
  --vpc-id $VPC_ID \
  --region $AWS_REGION \
  --query 'GroupId' --output text)

echo "Redis SG: $REDIS_SG"

# Security group for EC2 bastion (our Redis client)
export CLIENT_SG=$(aws ec2 create-security-group \
  --group-name redis-lab-client-sg \
  --description "Redis lab EC2 client" \
  --vpc-id $VPC_ID \
  --region $AWS_REGION \
  --query 'GroupId' --output text)

echo "Client SG: $CLIENT_SG"

# Allow inbound 6379 on Redis SG ONLY from the client SG
aws ec2 authorize-security-group-ingress \
  --group-id $REDIS_SG \
  --protocol tcp \
  --port 6379 \
  --source-group $CLIENT_SG \
  --region $AWS_REGION

echo "Inbound 6379 allowed from client SG to Redis SG"

# Verify the rule
aws ec2 describe-security-groups \
  --group-ids $REDIS_SG \
  --region $AWS_REGION \
  --query 'SecurityGroups[0].IpPermissions'

Observe: the inbound rule references the client SG ID as the source, not a CIDR block. This means only EC2 instances (or Lambdas) attached to CLIENT_SG can reach the Redis port. Any other resource in the VPC, even in the same subnet, cannot connect. This is the correct production pattern, not 0.0.0.0/0.

Checkpoint: Redis SG has one inbound rule: TCP 6379 from CLIENT_SG.

Create ElastiCache Subnet Group

What's happening here

A subnet group tells ElastiCache which subnets it can place cluster nodes in. ElastiCache will choose one subnet per AZ from the group. You should include subnets from at least 2 AZs so that Multi-AZ replication and failover work correctly. The subnet group is a control-plane concept, it doesn't affect connectivity directly, but if you pick subnets that your client can't reach (e.g. private subnets with no routing to your client), you'll get timeouts.

bash

# Create subnet group with subnets from at least 2 AZs
aws elasticache create-cache-subnet-group \
  --cache-subnet-group-name redis-lab-subnet-group \
  --cache-subnet-group-description "Redis lab subnet group" \
  --subnet-ids $(echo $SUBNET_IDS | tr ',' ' ') \
  --region $AWS_REGION

# Verify
aws elasticache describe-cache-subnet-groups \
  --cache-subnet-group-name redis-lab-subnet-group \
  --region $AWS_REGION \
  --query 'CacheSubnetGroups[0].{Name:CacheSubnetGroupName,Subnets:Subnets[*].SubnetIdentifier}'

Checkpoint: subnet group created with multiple subnets across AZs.

Create EC2 bastion for Redis access

bash

# IAM role for EC2 with SSM access (no SSH key needed)
cat > /tmp/ec2-trust.json << 'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}
EOF

aws iam create-role \
  --role-name redis-lab-ec2-role \
  --assume-role-policy-document file:///tmp/ec2-trust.json

aws iam attach-role-policy \
  --role-name redis-lab-ec2-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

aws iam create-instance-profile \
  --instance-profile-name redis-lab-ec2-profile

aws iam add-role-to-instance-profile \
  --instance-profile-name redis-lab-ec2-profile \
  --role-name redis-lab-ec2-role

sleep 10

# Launch EC2 in the same VPC with redis-cli pre-installed
AMI_ID=$(aws ec2 describe-images \
  --owners amazon \
  --filters Name=name,Values="al2023-ami-2023*-x86_64" Name=state,Values=available \
  --query 'sort_by(Images,&CreationDate)[-1].ImageId' \
  --output text --region $AWS_REGION)

export BASTION_ID=$(aws ec2 run-instances \
  --image-id $AMI_ID \
  --instance-type t3.micro \
  --iam-instance-profile Name=redis-lab-ec2-profile \
  --security-group-ids $CLIENT_SG \
  --subnet-id $SUBNET_1 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=redis-lab-bastion}]' \
  --user-data '#!/bin/bash
yum install -y redis6
' \
  --region $AWS_REGION \
  --query 'Instances[0].InstanceId' --output text)

echo "Bastion instance: $BASTION_ID"
aws ec2 wait instance-running --instance-ids $BASTION_ID --region $AWS_REGION
echo "Bastion running. redis-cli installing via user-data (~2 min)."

Checkpoint: EC2 instance running with CLIENT_SG. SSM Session Manager accessible.

Break It: IAM, Network, and the Redis Connection Model

Break 1: Wrong security group port, silent timeout

bash

# Add a rule for port 6380 instead of 6379 -- common typo
aws ec2 authorize-security-group-ingress \
  --group-id $REDIS_SG \
  --protocol tcp \
  --port 6380 \
  --source-group $CLIENT_SG \
  --region $AWS_REGION

echo "Rule added for 6380 (wrong port)."
echo "Connecting to Redis on 6379 with only this rule would hang indefinitely."
echo "The TCP SYN packet arrives at the instance but the SG drops it silently."
echo "No RST, no rejection -- just timeout. This is identical to the right port with wrong SG."

# Clean up the wrong rule
aws ec2 revoke-security-group-ingress \
  --group-id $REDIS_SG \
  --protocol tcp \
  --port 6380 \
  --source-group $CLIENT_SG \
  --region $AWS_REGION

both a wrong port and a wrong SG produce identical symptoms, connection timeout. There is no error message that says "security group blocked this". Debugging process: check SG rules first, then subnet routing, then NACLs, then try from a known-good host in the same SG.

Break 2: Attempt to connect from outside VPC

bash

# Demonstrate that there is no public endpoint
aws elasticache describe-cache-clusters \
  --region $AWS_REGION \
  --query 'CacheClusters[*].{ID:CacheClusterId,Endpoint:RedisConfiguration}' 2>/dev/null || echo "No clusters yet"

echo "ElastiCache endpoints are internal DNS names (*.cache.amazonaws.com)"
echo "They resolve to private IPs inside the VPC."
echo "Attempting to connect from your laptop: redis-cli -h <endpoint> -p 6379"
echo "Result: connection timed out. The private IP is not routable from the internet."
echo "Fix: use SSM Session Manager on the EC2 bastion, or set up a VPN/tunnel."

Cluster Setup and Basic Data Structures

Goal

Create a Redis cluster. Connect via the EC2 bastion. Master the 5 core data structures and understand which one to reach for in each situation.