PreviewLab5 hours

Logs

Create log groups, stream application logs, query with Logs Insights, and build metric filters for security events.

Mental model

CloudWatch Logs is a managed log aggregation system. Your apps emit lines of text (or JSON). CloudWatch stores them in a hierarchy: Log Group → Log Stream → Log Events. You query them with Logs Insights. Think of a Log Group as a table, a Log Stream as a partition, and each log event as a row.

Prerequisites

0 of 5 checked

Log Groups, Streams, and Retention

Goal

Understand the three-level hierarchy. Create log groups with correct retention policies. Write log events manually via CLI. Observe what no retention policy costs.

Estimated time: 60 min

The three-level hierarchy

What's happening here

Every piece of log data in CloudWatch lives inside a strict three-level hierarchy:

Log Group: logical container for a service or component. /aws/lambda/my-fn, /myapp/invoice-service. You control retention at the log group level.
Log Stream: a sequence of log events from a single source within the group. Lambda auto-creates one stream per function instance per day. EC2 CloudWatch Agent creates one stream per instance.
Log Event: a single line: a timestamp (milliseconds since epoch) and a message string. The message can be plain text or JSON, CloudWatch stores both identically, but Logs Insights can only extract fields from JSON.

Retention: by default, log groups never expire. Without a retention policy, every log event is kept forever and you pay storage costs indefinitely. Always set retention at log group creation.

Cost: $0.50 per GB ingested + $0.03 per GB stored per month. A service logging 100MB/day without retention = 36GB/year = ~$1.08/month in storage alone, growing every day.

bash

export AWS_REGION=ap-south-1
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

# Create a log group with 7-day retention
aws logs create-log-group \
  --log-group-name /cw-lab/invoice-service \
  --region $AWS_REGION

aws logs put-retention-policy \
  --log-group-name /cw-lab/invoice-service \
  --retention-in-days 7 \
  --region $AWS_REGION

# Create a second log group WITHOUT retention (observe the default)
aws logs create-log-group \
  --log-group-name /cw-lab/no-retention \
  --region $AWS_REGION

# Compare retention
aws logs describe-log-groups \
  --log-group-name-prefix /cw-lab \
  --region $AWS_REGION \
  --query 'logGroups[*].{Name:logGroupName,RetentionDays:retentionInDays,StoredBytes:storedBytes}'

Observe: /cw-lab/invoice-service shows RetentionDays: 7. /cw-lab/no-retention shows RetentionDays: null: never expire. In a real service running for a year, null retention turns into an unbounded storage bill.

Checkpoint: two log groups exist. One has retentionInDays: 7: the other null.

Create a log stream and write log events

What's happening here

Log streams are created explicitly (for custom sources) or auto-created (by Lambda, ECS, the CloudWatch Agent). When writing via the API directly you need to create the stream first, then call put-log-events. Timestamps in put-log-events are milliseconds since Unix epoch: not ISO strings. A common bug is passing seconds instead of milliseconds, which places your events in 1970. Events within a single put-log-events call must be in chronological order. Out-of-order events within a call are rejected. Events can be at most 2 hours in the future or 14 days in the past.