Overview
Read this before the SQS hands-on lab, or any time a queue appears in a system design and you want to understand what's actually happening.
The Problem SQS Solves
Imagine two services: an Invoice Service that creates invoices, and a PDF Generator that renders them. The naive approach:
Invoice Service → HTTP call → PDF GeneratorThis works until it doesn't. What happens when:
- The PDF Generator is slow? Invoice Service waits, holding a thread, burning latency.
- The PDF Generator is down? Invoice Service fails. The invoice is lost or the user gets an error.
- 10,000 invoices arrive at once? The PDF Generator gets crushed.
Every direct HTTP call between services couples their availability, performance, and throughput together. If one struggles, the other suffers. SQS breaks that coupling:
Invoice Service → SQS Queue → PDF GeneratorNow:
- Invoice Service puts a message in the queue and returns immediately. No waiting.
- PDF Generator reads from the queue at its own pace. It can process 10 or 1000 messages per second, independently.
- If PDF Generator goes down, messages stay in the queue. When it comes back, it picks up where it left off.
- If 10,000 invoices arrive, the queue absorbs the spike. PDF Generator drains it steadily.
This is asynchronous decoupling: the sender and receiver don't need to be available, fast, or scaled identically at the same time.
What SQS Is
Amazon Simple Queue Service (SQS) is a fully managed message queue. It stores messages durably until a consumer retrieves and deletes them. Producers write to it. Consumers read from it. SQS handles storage, replication, scaling, and delivery. Key properties:
- Fully managed: no servers to run, no capacity to provision
- Durable: messages are stored across multiple AZs. A single AZ failure doesn't lose messages.
- At-least-once delivery: every message is delivered at least once. In rare infrastructure failure scenarios, a message may be delivered more than once. Consumers must be idempotent.
- Scalable: queues handle millions of messages per second without configuration
- Pull-based: consumers poll the queue for messages. SQS doesn't push to consumers.
The Message Lifecycle
Every message in SQS goes through a specific lifecycle. Understanding this is the foundation for understanding everything else.
Rendering diagram…
Click to interact · then scroll to zoom, drag to pan
- 1Producer sends:
SendMessagewrites the message to the queue. It's stored durably. - 2Consumer receives:
ReceiveMessagereturns the message AND makes it invisible to all other consumers for the visibility timeout period. - 3Consumer processes: the consumer runs its business logic on the message.
- 4Consumer deletes: if processing succeeds,
DeleteMessageremoves it permanently. Done. - 5Visibility timeout expires: if the consumer doesn't delete within the timeout, SQS assumes processing failed (crash, timeout, error) and makes the message visible again for another consumer.
- 6Max receive count: if a message is received more than N times without being deleted (keeps failing), SQS moves it to the Dead Letter Queue.
The most important thing to internalise
deleting a message is how you tell SQS "I processed this successfully". If you don't delete, SQS will re-deliver. This is by design, it's how SQS guarantees at-least-once delivery even when consumers crash mid-processing.
Queue Types: Standard vs. FIFO
Standard Queue
- Throughput: unlimited. Millions of messages per second.
- Ordering: best-effort. Messages are generally delivered in the order sent, but not guaranteed. Under high load or after a retry, ordering can change.
- Delivery: at-least-once. A message may be delivered more than once in rare cases.
- Use when: ordering doesn't matter, or your consumer handles duplicates. The vast majority of use cases.
FIFO Queue
- Throughput: 3,000 messages per second with batching, 300 without.
- Ordering: guaranteed. Messages are delivered exactly in the order they were sent, within a message group.
- Delivery: exactly-once. Deduplication removes duplicate sends within a 5-minute window.
- Use when: order matters, financial transactions, sequential state machine steps, audit logs where event order is significant.
| Property | Standard | FIFO |
|---|---|---|
| Throughput | Unlimited | 3,000 msg/sec (batched) |
| Ordering | Best-effort | Guaranteed (per group) |
| Delivery | At-least-once | Exactly-once |
| Deduplication | No | Yes (5-min window) |
| Queue name suffix | (none) | Must end in .fifo |
| Cost | $0.40/million requests | $0.50/million requests |
Default to Standard. Only use FIFO when you have a concrete ordering requirement you can articulate.
Key Configuration Parameters
Visibility Timeout
When a consumer receives a message, SQS hides it from other consumers for this duration. If the consumer doesn't delete the message within this time, SQS assumes processing failed and makes the message visible again.
Default: 30 seconds. Range: 0 seconds to 12 hours. Set it to longer than your longest expected processing time with some buffer. If your Lambda processes messages in under 5 seconds, set 30 seconds. If your worker processes messages in up to 2 minutes, set 5 minutes. Too short: messages re-appear while still being processed → duplicate processing. Too long: a crashed consumer holds messages invisible for a long time before they're retried.
Message Retention Period
How long SQS keeps a message if no consumer receives it. Default: 4 days. Range: 1 minute to 14 days. If your consumer is down for longer than the retention period, messages are permanently lost, they're not moved to DLQ, they're just deleted. Set retention to 14 days for important workloads.
Receive Wait Time (Long Polling)
When a consumer calls ReceiveMessage and the queue is empty, SQS can:
- Short poll (default, wait time = 0): return immediately with an empty response. The consumer must poll again. Burns API calls and costs money.
- Long poll (wait time = 1–20 seconds): hold the connection open and return as soon as a message arrives, up to the wait time. Much more efficient.
Always use long polling. Set ReceiveMessageWaitTimeSeconds to 20 on the queue.
Max Receive Count (for DLQ)
How many times a message can be received without being deleted before SQS moves it to the Dead Letter Queue. Default: no DLQ configured (messages retry indefinitely). Set to 3–5 for most workloads.
Dead Letter Queue (DLQ)
A DLQ is just another SQS queue. You configure your main queue to send messages to it after they've been received and not deleted N times (maxReceiveCount). Messages in the DLQ are not re-processed automatically, they sit there for you to inspect. Why this matters: without a DLQ, a message your code can't process (malformed payload, bug in your consumer, downstream dependency permanently down) will retry forever, blocking other messages and burning compute. With a DLQ, it fails fast, gets captured, and your queue keeps moving.
DLQ is not optional in production. Every SQS queue should have one. The DLQ must be the same type as the source queue: Standard → Standard DLQ, FIFO → FIFO DLQ.
Polling: How Consumers Read Messages
SQS is pull-based. Consumers call ReceiveMessage to get messages. SQS doesn't push to consumers. This has important implications:
- You control the polling rate: how many consumers poll, how often, how many messages per request (up to 10 per call)
- Multiple consumers can poll the same queue. SQS ensures each message goes to exactly one consumer (via visibility timeout). This is how you scale consumers horizontally.
- Lambda as a consumer is a special case. Lambda polls the queue on your behalf via an Event Source Mapping and invokes your function with a batch of messages. You don't write polling code.
Worker-based: Lambda-based:
Your code polls SQS Lambda polls SQS for you
while True: Event Source Mapping:
messages = sqs.receive() - polls queue automatically
for msg in messages: - invokes Lambda with batch
process(msg) - deletes on success
sqs.delete(msg) - retries on failureSQS as a Buffer
One of the most important SQS patterns: absorbing traffic spikes so downstream services aren't overwhelmed.
Normal load: 10 req/sec → Queue → Consumer (10/sec capacity) ✓
Spike: 500 req/sec → Queue → Consumer (10/sec capacity) ✓
(490 msgs queue up, drained over ~49 seconds)Without SQS, the spike hits the consumer directly, it either crashes or rejects requests. With SQS, the spike is absorbed. The queue depth grows during the spike and drains afterward. The consumer always sees a steady flow. This pattern is used for: image processing pipelines, invoice/document generation, email sending, payment processing, any workload where bursty input arrives faster than the downstream can process.
SQS vs. SNS vs. EventBridge
These three services are often confused because they all move messages between systems.
| Service | Model | Persistence | Consumers | Best for |
|---|---|---|---|---|
| SQS | Queue (pull) | Up to 14 days | One consumer per message | Work queues, buffering, decoupling |
| SNS | Topic (push) | None (fire and forget) | Many subscribers per message | Fan-out, notifications, broadcast |
| EventBridge | Event bus (push) | None (24h retry) | Many targets via rules | Event-driven routing, service integration |
SQS: one message, one consumer, persistent. Use when a task needs to be done exactly once by one worker.
SNS: one message, many consumers, no persistence. Use when multiple systems need to react to the same event.
EventBridge: events with routing rules, no persistence. Use when you need content-based routing or AWS service integration. They're often combined: SNS → SQS (fan-out with persistence) is one of the most common patterns in AWS. SNS broadcasts to multiple SQS queues; each queue has its own consumer processing independently.
Common Patterns
Work Queue
The basic pattern. One producer, one consumer (or pool of consumers). Each message represents a unit of work.
Producer → SQS → Consumer workers (scaled horizontally)Fan-out (SNS → SQS)
One event triggers multiple independent processing pipelines.
SNS Topic
├── SQS Queue A → PDF Generator
├── SQS Queue B → Email Sender
└── SQS Queue C → Audit LoggerPriority Queue (two queues)
High-priority messages processed before low-priority by routing to separate queues and having consumers drain the high-priority queue first.
High Priority SQS → Consumer (poll first)
Low Priority SQS → Consumer (poll if high is empty)Retry with Exponential Backoff (DLQ redrive)
Messages that fail move to DLQ. A separate process periodically moves DLQ messages back to the main queue for retry, with delay increasing between attempts.
Idempotency: The Non-Negotiable Rule
SQS guarantees at-least-once delivery. This means your consumer will receive the same message more than once at some point, during infrastructure failures, network retries, or visibility timeout expiry. Your consumer must produce the same result whether it processes a message once or ten times. Idempotency strategies:
- Database upsert:
INSERT ... ON CONFLICT DO NOTHING: writing the same record twice is a no-op - Idempotency key: store processed message IDs in a database/Redis. Check before processing.
- Natural idempotency: some operations are inherently idempotent, setting a value is idempotent, incrementing a counter is not
The message MessageId (set by SQS) or a messageDeduplicationId you provide (FIFO queues) can serve as the idempotency key.
Cost Model
- Free tier: 1 million requests/month, always free
- Standard: $0.40 per million requests after free tier
- FIFO: $0.50 per million requests after free tier
- One "request" = one API call:
SendMessage,ReceiveMessage,DeleteMessageeach count separately - Long polling doesn't cost more than short polling, you're billed per
ReceiveMessagecall regardless of wait time - Data transfer within the same region is free
For most workloads, SQS cost is negligible. At 10 million messages/month with send + receive + delete = 30 million API calls = ~$12/month.
Summary
SQS is a durable buffer between two systems that don't need to talk to each other directly.
The producer writes and forgets. The consumer reads at its own pace. SQS holds the message until the consumer confirms it's done. If the consumer fails, the message comes back. If it keeps failing, it goes to the DLQ for inspection. This is the foundation of resilient asynchronous systems on AWS.