PreviewLab6 hours

Queues & DLQs

Create SQS queues, configure visibility timeouts and DLQs, and practice polling, long polling, and Lambda event source mappings.

Prerequisites

0 of 4 checked

Queue Basics: Send, Receive, Delete

Goal

Create a standard queue from scratch. Send messages, receive them, understand why you must explicitly delete them. Observe what happens when you don't.

Estimated time: 1–2 hours

Create your first standard queue

What's happening here

SQS is a pull-based message queue, consumers call receive-message to fetch messages; SQS never pushes to them. A Standard queue offers at-least-once delivery (a message may be delivered more than once) and best-effort ordering (order is not guaranteed). These two properties are fundamental: your consumers must be idempotent. The VisibilityTimeout=30 means: once a message is received, it becomes invisible to other consumers for 30 seconds. If you don't delete it within that window, it reappears and gets delivered again. MessageRetentionPeriod=86400 means unprocessed messages are automatically deleted after 24 hours (max is 14 days).

bash

mkdir sqs-lab && cd sqs-lab

export QUEUE_URL=$(aws sqs create-queue \
  --queue-name sqs-lab-standard \
  --attributes \
    VisibilityTimeout=30 \
    MessageRetentionPeriod=86400 \
    ReceiveMessageWaitTimeSeconds=0 \
  --region $AWS_REGION \
  --query 'QueueUrl' --output text)

echo "Queue URL: $QUEUE_URL"

export QUEUE_ARN=$(aws sqs get-queue-attributes \
  --queue-url $QUEUE_URL \
  --attribute-names QueueArn \
  --region $AWS_REGION \
  --query 'Attributes.QueueArn' --output text)

echo "Queue ARN: $QUEUE_ARN"

Verify:

bash

aws sqs get-queue-attributes \
  --queue-url $QUEUE_URL \
  --attribute-names All \
  --region $AWS_REGION \
  --query 'Attributes.{VisibilityTimeout:VisibilityTimeout,Retention:MessageRetentionPeriod,ApproxMsgs:ApproximateNumberOfMessages}'

Checkpoint: queue exists, ApproximateNumberOfMessages is 0.

Send messages

What's happening here

send-message is synchronous, it returns only after SQS has durably stored the message across multiple AZs. The MessageId in the response is SQS's internal ID; the MD5OfMessageBody is a checksum you can verify client-side if message integrity matters. The message body is an opaque string to SQS, it doesn't parse or validate it. You can put anything there: plain text, JSON, XML, base64. MessageAttributes are typed key-value metadata you can use to route or filter without parsing the body.

bash

# Send a plain message
aws sqs send-message \
  --queue-url $QUEUE_URL \
  --message-body 'Hello from SQS lab' \
  --region $AWS_REGION

# Send a JSON payload with message attributes
aws sqs send-message \
  --queue-url $QUEUE_URL \
  --message-body '{"event": "invoice.created", "id": "inv-001", "amount": 9900}' \
  --message-attributes \
    'source={DataType=String,StringValue=billing-service}' \
    'priority={DataType=Number,StringValue=1}' \
  --region $AWS_REGION

# Send a batch (up to 10 messages, charged as 1 API call)
aws sqs send-message-batch \
  --queue-url $QUEUE_URL \
  --entries \
    'Id=msg1,MessageBody={"id":"inv-002","amount":5000}' \
    'Id=msg2,MessageBody={"id":"inv-003","amount":12000}' \
    'Id=msg3,MessageBody={"id":"inv-004","amount":7500}' \
  --region $AWS_REGION

Check queue depth:

bash

aws sqs get-queue-attributes \
  --queue-url $QUEUE_URL \
  --attribute-names ApproximateNumberOfMessages \
  --region $AWS_REGION
# Expected: 4 (may take a few seconds to reflect)

Checkpoint: queue depth shows 4.

Receive messages

What's happening here

receive-message is a destructive read with a timer: the moment you receive a message, the clock starts on VisibilityTimeout. The message is not gone; it's hidden. If you crash after receiving but before deleting, the message reappears after 30 seconds and another consumer picks it up. MaxNumberOfMessages=10 is the ceiling per call (SQS may return fewer even if more exist (this is intentional, not a bug). The ReceiptHandle is the token you must use to delete the message) it's different from MessageId and unique per receive operation. Receiving the same message twice gives you two different receipt handles.

bash

# Receive up to 3 messages
aws sqs receive-message \
  --queue-url $QUEUE_URL \
  --max-number-of-messages 3 \
  --attribute-names All \
  --message-attribute-names All \
  --region $AWS_REGION | jq '.'

# Save receipt handles
aws sqs receive-message \
  --queue-url $QUEUE_URL \
  --max-number-of-messages 10 \
  --region $AWS_REGION \
  --query 'Messages[*].ReceiptHandle' \
  --output json > receipt_handles.json

cat receipt_handles.json

Check depth immediately after receive:

bash

aws sqs get-queue-attributes \
  --queue-url $QUEUE_URL \
  --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible \
  --region $AWS_REGION

ApproximateNumberOfMessages (visible) drops. ApproximateNumberOfMessagesNotVisible (in-flight, hidden) goes up. Total is still 4. The messages aren't gone, they're invisible.

Delete messages (acknowledge processing)

What's happening here

Deleting a message is the explicit acknowledgement that processing succeeded. SQS doesn't auto-delete on receive by design, if your consumer crashes mid-processing, the message must reappear. This is the core durability guarantee. delete-message-batch deletes up to 10 at once and costs one API call. If a delete fails because the receipt handle expired (your processing took longer than VisibilityTimeout), SQS returns a failure entry, you should check for it. A missing delete is the most common source of duplicate processing in SQS.

bash

# Delete messages one by one using stored receipt handles
for handle in $(cat receipt_handles.json | jq -r '.[]'); do
  aws sqs delete-message \
    --queue-url $QUEUE_URL \
    --receipt-handle "$handle" \
    --region $AWS_REGION
  echo "Deleted: ${handle:0:40}..."
done

Verify queue is empty:

bash

aws sqs get-queue-attributes \
  --queue-url $QUEUE_URL \
  --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible \
  --region $AWS_REGION
# Both should be 0

Checkpoint: both visible and in-flight counts are 0.

Observe what happens when you don't delete

What's happening here

This is the most important thing to see with your own eyes. We send a message, receive it (starting the visibility timer), then do nothing. After 30 seconds, SQS makes the message visible again. If another consumer calls receive-message: it gets the same message, a duplicate. This is what at-least-once delivery means in practice. Every SQS consumer must be designed to handle receiving the same message body more than once without causing double-processing.

bash

# Send a message
aws sqs send-message \
  --queue-url $QUEUE_URL \
  --message-body '{"test": "visibility-timeout", "ts": '"$(date +%s)"'}' \
  --region $AWS_REGION

# Receive it (starts the 30s clock)
aws sqs receive-message \
  --queue-url $QUEUE_URL \
  --max-number-of-messages 1 \
  --region $AWS_REGION | jq '.Messages[0].Body'

echo "Message received. NOT deleting it. Waiting 35 seconds..."
sleep 35

# Receive again, same message reappears
echo "Receiving again after visibility timeout:"
aws sqs receive-message \
  --queue-url $QUEUE_URL \
  --max-number-of-messages 1 \
  --region $AWS_REGION | jq '.Messages[0].Body'

same message body both times. ReceiptHandle is different (SQS generates a new one per receive). ApproximateReceiveCount attribute increments with each delivery.

bash

# Clean up: receive and delete
RH=$(aws sqs receive-message --queue-url $QUEUE_URL --max-number-of-messages 1 \
  --region $AWS_REGION --query 'Messages[0].ReceiptHandle' --output text)
aws sqs delete-message --queue-url $QUEUE_URL --receipt-handle $RH --region $AWS_REGION

Break It: Queue Basics: Send, Receive, Delete

Break 1: Message too large

bash

# SQS max message size is 256 KB
python3 -c "print('x' * 300000)" > big_payload.txt

aws sqs send-message \
  --queue-url $QUEUE_URL \
  --message-body file://big_payload.txt \
  --region $AWS_REGION

InvalidMessageContents or MessageTooLong. For payloads \>256 KB, the pattern is: store the payload in S3, send only the S3 key in the SQS message, consumer fetches from S3. This is the claim-check pattern.

Break 2: Expired receipt handle

bash

# Temporarily shorten visibility timeout to 5s
aws sqs set-queue-attributes \
  --queue-url $QUEUE_URL \
  --attributes VisibilityTimeout=5 \
  --region $AWS_REGION

aws sqs send-message --queue-url $QUEUE_URL --message-body 'expire test' --region $AWS_REGION

RH=$(aws sqs receive-message --queue-url $QUEUE_URL --max-number-of-messages 1 \
  --region $AWS_REGION --query 'Messages[0].ReceiptHandle' --output text)

echo "Waiting 8 seconds for receipt handle to expire..."
sleep 8

# Try to delete with the expired receipt handle
aws sqs delete-message \
  --queue-url $QUEUE_URL \
  --receipt-handle "$RH" \
  --region $AWS_REGION

ReceiptHandleIsInvalid. The message reappeared when the visibility timeout expired, and SQS no longer accepts the stale handle. Fix: use change-message-visibility mid-processing to extend the lease, or set a longer initial timeout.

bash

# Restore
aws sqs set-queue-attributes --queue-url $QUEUE_URL --attributes VisibilityTimeout=30 --region $AWS_REGION
RH=$(aws sqs receive-message --queue-url $QUEUE_URL --max-number-of-messages 1 \
  --region $AWS_REGION --query 'Messages[0].ReceiptHandle' --output text)
[ "$RH" != "None" ] && aws sqs delete-message --queue-url $QUEUE_URL --receipt-handle $RH --region $AWS_REGION

Visibility Timeout + Polling Mechanics

Goal

Understand long polling vs short polling. Extend visibility timeout mid-processing. Simulate a slow consumer and observe what happens. Build a correct consumer loop.