Free previewConcept guide25 min read

Overview

Each section leads with what ECS actually does; mental model callouts add an analogy when it helps. Diagrams carry the structure. Read top to bottom once, then use as reference.

What is ECS?

Amazon ECS is a fully managed container orchestration service. You give it a Docker image and a spec (CPU, memory, port). ECS decides where to run it, starts it, monitors it, restarts it on failure, scales it, and routes traffic to it.

Mental model

Imagine you run a restaurant chain. You write a recipe (container image) and a staffing manual (task definition). ECS is the regional manager, it figures out which kitchen (EC2 or Fargate) to cook in, makes sure every shift is staffed, and replaces anyone who doesn't show up. You never touch the kitchen yourself.

What ECS does NOT manage: your VPC, subnets, security groups, IAM roles, load balancers. Those are yours to design.

ECS vs Kubernetes: ECS is simpler and AWS-native. Kubernetes is more powerful and portable. If you're on AWS and don't need multi-cloud, ECS is the pragmatic choice, less operational overhead, native integration with every AWS service.

Core Building Blocks

The hierarchy in one sentence: A *Cluster* holds *Services*, a *Service* runs *Tasks*, a *Task* is built from a *Task Definition*, and a *Task Definition* points to your Docker image.

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

ECS Concept	Real-world analogy
Task Definition	Recipe card
Task	One cooked dish from that recipe
Service	The chef ensuring 3 dishes are always on the table
Cluster	The restaurant
Capacity (Fargate/EC2)	The kitchen / cooking equipment

Cluster

A cluster is a logical grouping, a namespace boundary. It holds no compute by itself. It associates with capacity providers (Fargate or EC2 ASGs), and all services + tasks within it share that capacity pool.

Mental model

A cluster is like a floor in an office building. The floor doesn't do any work, it just defines which desks (compute) and which teams (services) belong together. You'd have a prod floor, a staging floor, a dev floor. Walls between them mean different access controls and separate billing visibility.

Cluster states

State	What's happening
`ACTIVE`	Healthy, accepting tasks
`PROVISIONING`	Spinning up capacity provider resources
`DEPROVISIONING`	Tearing down capacity provider resources
`FAILED`	Capacity provider setup broke
`INACTIVE`	Deleted, may ghost in API for a bit

Key settings

Capacity Providers: FARGATE, FARGATE_SPOT, or your EC2 ASG
Default Capacity Provider Strategy: which provider to use when a service doesn't specify. Set base=1 on FARGATE so at least one task is always on stable compute
Container Insights: CloudWatch metrics per cluster. Always enable on prod
AZ Rebalancing: ECS redistributes tasks if they pile up in one AZ

Best practice: one cluster per environment. Clean IAM separation, independent cost tracking, separate Container Insights dashboards.

bash

aws ecs create-cluster \
  --cluster-name prod-cluster \
  --capacity-providers FARGATE FARGATE_SPOT \
  --default-capacity-provider-strategy \
    capacityProvider=FARGATE,weight=1,base=1 \
    capacityProvider=FARGATE_SPOT,weight=4,base=0 \
  --settings name=containerInsights,value=enabled

aws ecs list-clusters
aws ecs describe-clusters --clusters prod-cluster --include STATISTICS SETTINGS
aws ecs delete-cluster --cluster prod-cluster

Task Definition

An immutable, versioned JSON document describing everything needed to run containers, image, CPU/memory, network mode, IAM roles, ports, env vars, secrets, health check, log config. Every change creates a new revision (:1, :2…). Old revisions are never deleted, only deregistered.

Mental model

A task definition is like a job posting. It says: "I need someone with Python 3.11 (image), 2 vCPUs of work capacity (CPU), fluent in English (env vars), and they must report to HR (IAM role)." Every time you update the requirements, you post a new version of the job. Old postings stay on file, that's your deployment history.

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

The most important fields

essential: true: if this container exits, the entire task stops. Put it on your main app container. Sidecars (log shippers, proxies) are essential: false: you don't want a flaky log agent killing your API.

CPU/Memory, two levels:

Task level: total budget for the whole task. Fargate enforces this as a hard cap.
Container level: per-container reservation. On Fargate, optional but recommended.

⚠️ On Fargate, task-level CPU and memory are required. ECS rejects task definitions that only set them on containers.

Valid Fargate CPU/Memory combos:

vCPU	Memory range
0.25	512 MB – 2 GB
0.5	1 GB – 4 GB
1	2 GB – 8 GB
2	4 GB – 16 GB
4	8 GB – 30 GB
8	16 GB – 60 GB
16	32 GB – 120 GB

Annotated Task Definition JSON

json

{
  "family": "my-app",                    // logical name, revisions group under this
  "networkMode": "awsvpc",               // required for Fargate
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "512",                          // 0.5 vCPU, task level
  "memory": "1024",                      // 1 GB, task level
  "executionRoleArn": "arn:aws:iam::123:role/ecsTaskExecutionRole",
  "taskRoleArn":      "arn:aws:iam::123:role/my-app-task-role",
  "containerDefinitions": [{
    "name": "my-app",
    "image": "123.dkr.ecr.us-east-1.amazonaws.com/my-app:latest",
    "essential": true,
    "portMappings": [{ "containerPort": 8080 }],
    "environment": [{ "name": "ENV", "value": "production" }],
    "secrets": [{
      "name": "DB_PASSWORD",             // injected as env var at task start
      "valueFrom": "arn:aws:secretsmanager:..."
    }],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/my-app",
        "awslogs-region": "us-east-1",
        "awslogs-stream-prefix": "ecs"
      }
    },
    "healthCheck": {
      "command": ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"],
      "interval": 30,      // check every 30s
      "timeout": 5,        // fail if no response in 5s
      "retries": 3,        // 3 consecutive failures = UNHEALTHY
      "startPeriod": 60    // don't count failures in first 60s (startup grace)
    }
  }]
}

bash

aws ecs register-task-definition --cli-input-json file://taskdef.json
aws ecs list-task-definition-families
aws ecs list-task-definitions --family-prefix my-app
aws ecs describe-task-definition --task-definition my-app:3
aws ecs deregister-task-definition --task-definition my-app:1  # marks INACTIVE, not deleted

Task

A task is a running instantiation of a task definition. One or more containers, running together on the same network namespace and lifecycle. Tasks are ephemeral, they start, do work, and stop. ECS tracks every state transition and records the stop reason and exit code.

Mental model

If a task definition is a recipe, a task is the dish you made from it right now. You can make many dishes from the same recipe simultaneously. When a dish goes bad, you throw it out and cook fresh. ECS is the kitchen manager who notices when a dish has gone off and calls in a replacement.

Two ways to run a task:

Via a Service: long-running things: APIs, background workers. Service keeps it alive.
Via run-task: one-shot things: DB migrations, batch jobs, cron. Runs once, exits.

Task lifecycle

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

Task states

State	What it means
`PROVISIONING`	ECS is finding a place to run this
`PENDING`	Found a place, pulling the image
`ACTIVATING`	Setting up Service Connect / Cloud Map
`RUNNING`	All essential containers are alive
`DEACTIVATING`	Starting graceful shutdown
`STOPPING`	SIGTERM sent, waiting up to `stopTimeout`
`DEPROVISIONING`	Releasing network interface
`STOPPED`	Done. Check `stoppedReason` if unexpected.

Most common failure: task cycles PENDING → STOPPED in under 30 seconds. Almost always: bad image tag, execution role missing ECR permission, or app crashing on startup. Check stoppedReason + containers[0].reason + CloudWatch logs.

bash

aws ecs run-task \
  --cluster prod-cluster \
  --task-definition my-app:3 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[subnet-abc],securityGroups=[sg-xyz],assignPublicIp=DISABLED}"

aws ecs list-tasks --cluster prod-cluster --desired-status RUNNING
aws ecs describe-tasks --cluster prod-cluster --tasks <task-arn>
aws ecs stop-task --cluster prod-cluster --task <task-arn>

Service

A Service is a long-lived desired-state controller. It continuously reconciles actual running task count with desired count. It auto-registers task IPs into ALB target groups on start, drains on stop, manages rolling deployments, and optionally triggers auto scaling.

Mental model

A Service is the shift supervisor who makes sure 3 cashiers are always at the tills. If one calls in sick, the supervisor immediately calls a replacement. If it's Black Friday, they call in extra staff. They also tell the front door (ALB) which cashiers are ready to serve customers, and remove anyone who went home.

Service scheduler strategies

Strategy	When to use	Fargate support
`REPLICA`	APIs, workers, run N tasks, spread across AZs	✅
`DAEMON`	Log agents, monitoring, exactly 1 task per EC2 node	❌

The deployment numbers, visualised

javascript

desiredCount = 4, minimumHealthyPercent = 50, maximumPercent = 200

Rolling deploy starts:
Before:  [v1][v1][v1][v1]                       ← 4 tasks
Step 1:  [v1][v1][v2][v2][v2][v2]               ← 4 new started (6 total, under 200% cap)
Step 2:  [v2][v2][v2][v2]                       ← 2 old drained and stopped

Key config fields:

desiredCount: target running tasks
minimumHealthyPercent: floor during deploy (50 = half must stay up)
maximumPercent: ceiling during deploy (200 = can temporarily double)
healthCheckGracePeriodSeconds: don't kill a new task for failing health checks during startup
deploymentCircuitBreaker: auto-rollback if new tasks keep failing

bash

aws ecs create-service \
  --cluster prod-cluster \
  --service-name my-api \
  --task-definition my-app:3 \
  --desired-count 3 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[subnet-a,subnet-b],securityGroups=[sg-xyz],assignPublicIp=DISABLED}" \
  --load-balancers "targetGroupArn=arn:aws:...,containerName=my-app,containerPort=8080" \
  --health-check-grace-period-seconds 60 \
  --deployment-configuration "minimumHealthyPercent=50,maximumPercent=200,deploymentCircuitBreaker={enable=true,rollback=true}"

aws ecs update-service --cluster prod-cluster --service my-api --force-new-deployment
aws ecs update-service --cluster prod-cluster --service my-api --desired-count 5
aws ecs update-service --cluster prod-cluster --service my-api --desired-count 0
aws ecs delete-service --cluster prod-cluster --service my-api

Launch Types & Capacity Providers

Mental model: Fargate vs EC2

Fargate = renting a fully furnished apartment. You move in, use it, leave. No plumbing, no maintenance. More expensive per unit.

EC2 = buying a house. Full control, cheaper at scale, but you maintain everything.

FARGATE_SPOT = Airbnb, cheap, but the host can kick you out with 2 minutes' notice. Great for batch jobs, terrible for stateful workloads.

Launch type comparison

	Fargate	EC2	FARGATE_SPOT
Manage servers	❌	✅	❌
Cost	Medium	Low at scale	Up to 70% off
Task isolation	Strong (micro-VM per task)	Weaker (shared host OS)	Strong
Interruption risk	None	None	Yes, 2 min warning
GPU workloads	❌	✅	❌

Capacity provider strategy: weight + base

bash

# 1 guaranteed on-demand task, rest mostly on SPOT
aws ecs create-service \
  --capacity-provider-strategy \
    capacityProvider=FARGATE,weight=1,base=1 \
    capacityProvider=FARGATE_SPOT,weight=4,base=0
# With desiredCount=5:
# → 1 task on FARGATE (base=1 guarantees at least 1)
# → 4 tasks on FARGATE_SPOT (weight ratio drives the rest)

FARGATE_SPOT rule: your app MUST handle SIGTERM and shut down within 120 seconds. ECS sends SIGTERM on interruption. If you don't handle it, requests die mid-flight with no drain.

Networking Modes

Mental model: network modes

awsvpc: each task gets its own apartment with its own front door and address (private IP). Safest, most isolated.

bridge: all tasks share the building lobby (host network), different room numbers (mapped ports). Higher density, less isolation.

host: the task IS the building, no separation, maximum performance.

none: the container has no phone and no address. For pure compute jobs with no network.

The four modes

Mode	Each task gets	Fargate	Best for
`awsvpc`	Own ENI + private IP	✅ Required	Everything, use this by default
`bridge`	Dynamic host port mapping	❌	Legacy EC2, high task density
`host`	EC2 host's network directly	❌	Max throughput, no isolation
`none`	No network	❌	Isolated batch jobs

awsvpc, why it's the default

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

Each task has its own ENI → its own security group → granular firewall per service. Task A can only talk to the database, Task B only to Redis. In bridge mode you'd manage this at the host level, which doesn't scale cleanly.

One gotcha: EC2 instances have ENI limits. A t3.medium has 3 ENIs max → 3 awsvpc tasks max per instance. On Fargate, no such limit.

bridge mode, how dynamic port mapping works

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

Multiple containers on one host, all listening on port 8080 internally but exposed on different host ports. The ALB knows each ephemeral port and routes accordingly. More tasks per instance = cheaper, but no per-task security groups.

IAM Roles in ECS, The Two Roles

🔑 This is the #1 source of confusion in ECS. Memorise the distinction once and debugging IAM becomes trivial.

Mental model: two access cards

Imagine a new employee starting at your company. The onboarding team (ECS Agent / Execution Role) handles admin before day 1: gets the badge (pull image), sets up the desk (create log stream), retrieves the laptop password from HR (fetch secrets). The employee themselves (your app / Task Role) then uses their own access card to do actual work: enter the server room (S3), use the database (DynamoDB), send emails (SES).

Two different people, two different access cards. Mixing them up is the most common ECS IAM mistake.

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

Role 1: Task Execution Role


Used by	ECS Agent, before your container starts
Set via	`executionRoleArn` in task definition
Needs	ECR pull, CloudWatch logs, Secrets Manager / SSM
AWS managed policy	`AmazonECSTaskExecutionRolePolicy`
Debug signal	Task stops before ever starting

Role 2: Task Role


Used by	Your application code at runtime
Set via	`taskRoleArn` in task definition
Needs	Whatever your app calls: S3, DynamoDB, SQS, SNS…
Credential delivery	ECS injects endpoint at `169.254.170.2`, SDK calls this automatically
Debug signal	App gets AccessDenied at runtime

Summary, never confuse these again

	Task Execution Role	Task Role
Who uses it	ECS Agent	Your app
When	Before container starts	While container runs
Set via	`executionRoleArn`	`taskRoleArn`
Example	Pull ECR image, write logs	Read from S3, write to DynamoDB

Security note: on EC2, without a Task Role, your container can hit 169.254.169.254 (EC2 IMDS) and inherit the EC2 instance role, which likely has far more permissions than intended. Always set a Task Role, even if it's empty.

Load Balancing

ECS integrates with ALB/NLB by auto-registering each task's IP + container port into a Target Group on start, and deregistering + draining on stop. The LB health checks each task independently and only routes to healthy ones.

Mental model

The ALB is the receptionist at the front desk. Tasks are staff in the back office. When a new staff member is ready to take calls (task passes health check), the receptionist adds them to the call list. When someone leaves (task stopping), the receptionist finishes routing their current calls before removing them.

Which LB to pick

LB	Protocol	Pick it when
ALB	HTTP/HTTPS, WebSocket	Default for everything. Path routing, host routing, sticky sessions.
NLB	TCP/UDP/TLS	Non-HTTP, or extreme throughput / low latency needed.
CLB	HTTP/TCP	Don't. Legacy. No Fargate support.

The request flow

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

Path-based routing, one ALB, many services

javascript

ALB Listener :443
  ├── /api/*   → Target Group A → ECS Service: my-api
  ├── /auth/*  → Target Group B → ECS Service: auth-service
  └── /*       → Target Group C → ECS Service: frontend

Critical config:

target-type: ip required for Fargate/awsvpc (registers task IP, not instance ID)
healthCheckGracePeriodSeconds on Service, prevents killing a slow-starting task
deregistrationDelay on Target Group, time for in-flight requests to complete

Service Discovery & Service Connect

Task A needs to call Task B, but task IPs change every deployment, restart, and scale event. You can't hardcode IPs.

Mental model

Imagine working in a big office where people change desks every day. Without a directory, you'd never find your colleague. Service Discovery is the office directory (DNS). Service Connect is the internal phone operator, smarter, handles busy signals, and automatically reroutes if someone is away.

Service Discovery (Cloud Map + DNS)

ECS registers each task IP → AWS Cloud Map → DNS hostname
Task A calls http://auth-service.prod.local:8080 → resolves to Task B's IP
Problem: DNS caches (TTL). During a deployment, DNS might still point to an old dead IP. Your app must handle retries.

Service Connect (recommended)

ECS injects Envoy proxy sidecar automatically into every task
Services call each other by short name: http://auth:8080
Envoy handles retries, circuit breaking, connection pooling
No DNS TTL issues, proxy has current state

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

💡 Use Service Connect for all new services. Service Discovery only for legacy integrations or non-ECS services that need to reach ECS via DNS.

Auto Scaling

Two completely separate scaling layers:

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

Mental model

Layer 1 = hiring more staff when the queue gets long. Layer 2 = opening more offices because there's no room for the new staff (EC2 only. Fargate skips this entirely).

Layer 1: Policy types

Policy	How it works	Best for
Target Tracking	Thermostat, maintain CPU/memory at a target %	Default. Set it and forget it.
Step Scaling	Scale by N tasks when metric crosses thresholds	Fine-grained control
Scheduled	Scale to N tasks at specific times	Known patterns (business hours)

bash

# Register service as scalable
aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --resource-id service/prod-cluster/my-api \
  --scalable-dimension ecs:service:DesiredCount \
  --min-capacity 2 \
  --max-capacity 20

# Attach CPU target tracking at 60%
aws application-autoscaling put-scaling-policy \
  --policy-name cpu-target-tracking \
  --service-namespace ecs \
  --resource-id service/prod-cluster/my-api \
  --scalable-dimension ecs:service:DesiredCount \
  --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration '{
    "TargetValue": 60.0,
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ECSServiceAverageCPUUtilization"
    },
    "ScaleOutCooldown": 60,
    "ScaleInCooldown": 300
  }'

Cooldown asymmetry: scale out fast (60s), scale in slow (300s). Better to have one extra task than to drop one and immediately need it back.

Deployment Strategies

Rolling Update (default)

Mental model

Like replacing crew on a ship one by one. The ship never stops sailing. There's always enough experienced people to keep things running while new crew come aboard.

javascript

4 tasks (v1). Deploy v2. minHealthy=50%, maxPercent=200%:

Before:  [v1][v1][v1][v1]
Step 1:  [v1][v1][v2][v2][v2][v2]   ← 4 new tasks start (6 total, under 200%)
Step 2:  [v2][v2][v2][v2]            ← 2 old tasks drained and stopped

Deployment Circuit Breaker: if new tasks keep failing health checks, ECS auto-rolls back. No manual intervention needed.

bash

aws ecs update-service \
  --cluster prod-cluster --service my-api \
  --task-definition my-app:4 \
  --deployment-configuration \
    "deploymentCircuitBreaker={enable=true,rollback=true},minimumHealthyPercent=50,maximumPercent=200"

Blue/Green (via CodeDeploy)

Mental model

Like opening a second restaurant with the new menu before closing the original. Both run simultaneously. You send 10% of diners to the new one first. If they love it, you send everyone. If there's a kitchen disaster, you send everyone back instantly.

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

Traffic shifting presets:

ECSCanary10Percent5Minutes, 10% canary for 5 min, then 100%
ECSLinear10PercentEvery1Minutes: gradual 10% per minute
ECSAllAtOnce: instant cutover

Use Blue/Green when: breaking schema changes, regulatory zero-downtime requirements, or you need instant rollback.

Skip Blue/Green when: routine deploys, it costs 2x capacity for the deployment window.

Logging & Observability

Mental model

Your container's stdout is someone shouting in a room. Without a log driver, nobody hears it and it vanishes when the task stops. awslogs is a microphone sending everything to CloudWatch. FireLens is a mixing board, take that signal and route it anywhere: CloudWatch, S3, Datadog, Splunk.

Log drivers

Driver	Where logs go	Complexity
`awslogs`	CloudWatch Logs	Simplest. Default choice.
`awsfirelens`	Anywhere via Fluent Bit	Production-grade routing
`splunk`	Splunk directly	Enterprise only
`json-file`	Local file on host	Dev/local only. Dies with task.

awslogs config

json

"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "/ecs/my-app",
    "awslogs-region": "us-east-1",
    "awslogs-stream-prefix": "ecs",       // stream: ecs/container-name/task-id
    "awslogs-create-group": "true"        // auto-create group if missing
  }
}

FireLens, production pattern

Add Fluent Bit as a sidecar. ECS manages it alongside your app:

json

{
  "name": "log-router",
  "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:stable",
  "essential": true,
  "firelensConfiguration": { "type": "fluentbit" }
}

Container Insights

Enabled at cluster level, gives you CloudWatch dashboards for:

CPU + memory per cluster / service / task / container
Task count history, scale events and failures
Network I/O

💡 Enable on prod. Fastest way to spot: memory leaks (memory climbing over hours), bad deploys (task failure rate spiking), capacity issues (CPU pegged at 100% across all tasks).

Storage & Persistence (EFS)

EFS (Elastic File System) is a managed NFS filesystem accessible from multiple tasks simultaneously, across AZs. Works with both EC2 and Fargate.

Mental model

Container local disk is like a whiteboard (fast, useful, but completely erased when the meeting ends (task stops). EFS is a shared Google Drive) all tasks read and write to the same filesystem, and data survives any individual task stopping or restarting.

When you need EFS

File uploads that must persist across deployments
Shared config files read by multiple tasks
ML model files too large to bake into the image
Any state that shouldn't live inside a container

json

"volumes": [{
  "name": "shared-data",
  "efsVolumeConfiguration": {
    "fileSystemId": "fs-0abc12345",
    "rootDirectory": "/app-data",
    "transitEncryption": "ENABLED",
    "authorizationConfig": {
      "accessPointId": "fsap-0xyz",
      "iam": "ENABLED"    // task role needs elasticfilesystem:ClientMount
    }
  }
}],
"mountPoints": [{
  "sourceVolume": "shared-data",
  "containerPath": "/data",
  "readOnly": false
}]

Most common EFS failure: EFS security group doesn't allow inbound NFS (port 2049) from the task's security group. Task starts, hangs at mount time. Fix: EFS SG → inbound 2049 → source = task SG.

ECS Exec. Debugging Running Containers

ECS Exec uses AWS Systems Manager Session Manager to create an encrypted WebSocket tunnel from your terminal to a running container, no SSH daemon, no bastion host, no open inbound ports.

Mental model

Normal debugging requires SSH, open ports, keys, bastion hosts, a security nightmare. ECS Exec is like calling an employee directly on their work phone (SSM). No need to unlock the building, no visitor badge. The call is encrypted, fully logged, and you never open any doors.

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

Setup

1Task Role needs SSM permissions:

json

{
  "Effect": "Allow",
  "Action": [
    "ssmmessages:CreateControlChannel",
    "ssmmessages:CreateDataChannel",
    "ssmmessages:OpenControlChannel",
    "ssmmessages:OpenDataChannel"
  ],
  "Resource": "*"
}

1Enable on service (force new deploy, existing tasks don't have SSM agent):

bash

aws ecs update-service \
  --cluster prod-cluster --service my-api \
  --enable-execute-command --force-new-deployment

1Exec in:

bash

aws ecs execute-command \
  --cluster prod-cluster --task <task-arn> \
  --container my-app --interactive --command "/bin/sh"

Inside the container, inspect the credential endpoint:

bash

# See the Task Role credentials ECS injected automatically
curl http://169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
# Returns: AccessKeyId, SecretAccessKey, Token, Expiration, RoleArn
# This is exactly what boto3/SDK calls, no hardcoded keys anywhere

CLI Cheatsheet

Cluster

bash

aws ecs create-cluster --cluster-name <name>
aws ecs list-clusters
aws ecs describe-clusters --clusters <name> --include STATISTICS SETTINGS
aws ecs delete-cluster --cluster <name>

Task Definitions

bash

aws ecs register-task-definition --cli-input-json file://taskdef.json
aws ecs list-task-definition-families
aws ecs list-task-definitions --family-prefix <family>
aws ecs describe-task-definition --task-definition <family>:<rev>
aws ecs deregister-task-definition --task-definition <family>:<rev>

Services

bash

aws ecs create-service --cluster <c> --service-name <s> --task-definition <td> --desired-count <n>
aws ecs update-service --cluster <c> --service <s> --task-definition <td>:<rev> --force-new-deployment
aws ecs update-service --cluster <c> --service <s> --desired-count <n>
aws ecs describe-services --cluster <c> --services <s>
aws ecs list-services --cluster <c>
aws ecs update-service --cluster <c> --service <s> --desired-count 0   # drain first
aws ecs delete-service --cluster <c> --service <s>

Tasks

bash

aws ecs run-task --cluster <c> --task-definition <td> --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[<sn>],securityGroups=[<sg>],assignPublicIp=DISABLED}"
aws ecs list-tasks --cluster <c> --desired-status RUNNING
aws ecs list-tasks --cluster <c> --service-name <s>
aws ecs describe-tasks --cluster <c> --tasks <arn>
aws ecs stop-task --cluster <c> --task <arn>

Debugging

bash

# Last 10 service events
aws ecs describe-services --cluster <c> --services <s> \
  --query 'services[0].events[:10]'

# Why did this task stop?
aws ecs describe-tasks --cluster <c> --tasks <arn> \
  --query 'tasks[0].{reason:stoppedReason,containers:containers[*].{name:name,exit:exitCode,reason:reason}}'

# Exec into running container
aws ecs execute-command --cluster <c> --task <arn> \
  --container <name> --interactive --command "/bin/sh"

ECR

bash

aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin <account>.dkr.ecr.us-east-1.amazonaws.com
aws ecr create-repository --repository-name my-app
docker build -t my-app .
docker tag my-app:latest <account>.dkr.ecr.us-east-1.amazonaws.com/my-app:latest
docker push <account>.dkr.ecr.us-east-1.amazonaws.com/my-app:latest

Common Gotchas

Things that break everyone at least once.

Task cycles PENDING → STOPPED in under 30 seconds
- Check stoppedReason + containers[0].reason + CloudWatch logs first
- App crash on startup (most common)
- Bad image tag → CannotPullContainerError
- Execution role missing ECR permission
- Container OOM'd immediately (increase memory)
Service never reaches desiredCount, tasks start then get killed by health checks
- Task SG doesn't allow inbound on container port from ALB SG
- Health check path isn't returning HTTP 200
- healthCheckGracePeriodSeconds is too short for your app's startup time
Task stuck in PENDING for minutes
- Fargate: subnet has no route to internet + no VPC endpoint for ECR (agent can't pull the image)
- EC2: no instance has free CPU/memory — check CPUReservation and MemoryReservation
App gets AccessDenied calling AWS services
- You added the permission to the Execution Role instead of the Task Role
- Execution role is only used by the agent before your app starts
- Your app needs the permission on taskRoleArn
ENI limit hit (EC2 + awsvpc)
- Tasks stuck in PROVISIONING; service events mention ENI exhaustion
- t3.medium = 3 ENIs max = 3 tasks max
- Fix: larger instances, awsvpcTrunking, or Fargate
FARGATE_SPOT tasks dying mid-request
- Spot interruptions give 2 minutes
- App must catch SIGTERM, stop accepting requests, drain in-flight, exit
- Set deregistrationDelay on the target group to match your drain time
Secrets not in container, task won't start
- Execution role missing secretsmanager:GetSecretValue
- KMS-encrypted secret also needs kms:Decrypt
- ECS won't start the container if secret injection fails — check service events for ResourceInitializationError
--force-new-deployment didn't update the image
- You pushed to the same tag (:latest) without changing the task definition
- Force-deploy starts new tasks but they pull the cached digest
- Fix: use immutable tags (:v1, :abc1234) or register a new task definition revision

Production Architecture

Multi-service ECS setup

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

Rolling deployment sequence

Rendering diagram…

Click to interact · then scroll to zoom, drag to pan

Quick Reference Card

Symptom / Question	Answer
App can't call S3 / DynamoDB	Add permission to Task Role (`taskRoleArn`)
ECS can't pull ECR image	Add `ecr:BatchGetImage` to Execution Role
Secret not in container, task won't start	Add `secretsmanager:GetSecretValue` to Execution Role
Service health check loop, never stabilises	SG not open on container port from ALB SG, or `healthCheckGracePeriodSeconds` too low
Task stuck in PENDING	No internet route / no ECR VPC endpoint (Fargate), or insufficient EC2 capacity
Task crashes after ~2 min randomly	On `FARGATE_SPOT`: handle SIGTERM within 120s
ENI exhaustion, tasks stuck PROVISIONING	Use larger EC2 instance, awsvpcTrunking, or switch to Fargate
Need a shell in a running container	`aws ecs execute-command --interactive --command /bin/sh`
Multiple services need shared files	Mount EFS: works across tasks, AZs, EC2 and Fargate
Fast rollback on bad deploy	Blue/Green via CodeDeploy
Simple deploy with auto-rollback	Rolling Update + circuit breaker
One task per EC2 node	DAEMON scheduling strategy
Services calling each other by name	Service Connect (preferred) or Service Discovery