PreviewLab5 hours

Metrics

Publish custom metrics, use metric math and SEARCH expressions, and build dashboards from AWS and application data.

Mental model

CloudWatch Metrics is a time-series database. Every number your AWS infrastructure produces (CPU %, invocation count, latency, queue depth) lands here as a data point. You query it, graph it, and alarm on it. Think of it as a append-only ledger of what your system has done, at fixed time intervals.

Prerequisites

0 of 5 checked

The Metric Data Model

Goal

Understand how CloudWatch organises metric data. Query existing AWS service metrics. Learn what namespace, dimension, statistic, and period actually mean.

Estimated time: 60 min

The four-level hierarchy

What's happening here

Every metric in CloudWatch lives inside a strict four-level hierarchy:

Namespace: logical container, usually the AWS service. AWS/Lambda, AWS/SQS, AWS/EC2. Your custom metrics go in a namespace you define, e.g. MyApp/Orders.
Metric Name: what is being measured. Invocations, Duration, Errors, CPUUtilization.
Dimensions: key-value pairs that identify *which resource* the metric belongs to. FunctionName=my-fn, QueueName=my-queue. A metric without dimensions applies to all resources in that namespace.
Data Points: the actual numbers, each with a timestamp, value, and unit.

The combination of Namespace + MetricName + Dimensions uniquely identifies a metric time series. If you change even one dimension value, you get a completely separate metric, not an update to the existing one. This is the most common source of "I published a metric but I can't find it" bugs.

bash

export AWS_REGION=ap-south-1
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

# List all namespaces in your account
aws cloudwatch list-metrics \
  --region $AWS_REGION \
  --query 'Metrics[*].Namespace' --output text \
  | tr '\t' '\n' | sort -u

# List all metrics in the Lambda namespace
aws cloudwatch list-metrics \
  --namespace AWS/Lambda \
  --region $AWS_REGION \
  --query 'Metrics[*].{Namespace:Namespace,Name:MetricName,Dims:Dimensions}'

# List metrics for a specific function (filter by dimension)
aws cloudwatch list-metrics \
  --namespace AWS/Lambda \
  --metric-name Invocations \
  --dimensions Name=FunctionName,Value=scheduler-lab-fn \
  --region $AWS_REGION

Observe: if you've run the Scheduler lab, scheduler-lab-fn appears as a dimension value under AWS/Lambda. If not, substitute any Lambda function name in your account. Notice that each unique FunctionName value creates a completely separate metric time series even though the metric name (Invocations) is the same.

Checkpoint: you can list metrics by namespace and filter by dimension. At least one Lambda metric is visible.

Statistics and what they mean

What's happening here

When you query a metric, CloudWatch aggregates data points in each time bucket using a statistic. The same underlying data produces completely different numbers depending on which statistic you choose:

Sum: total of all values in the period. Right for counters: invocation count, error count, bytes transferred.
Average (mean of all values. Right for utilisation: CPU %, memory %. Wrong for counters) averaging invocation counts across Lambda instances produces a meaningless number.
Max / Min: highest or lowest value seen. Right for latency spikes: you care about the worst case, not the average.
SampleCount: number of data points in the period. Useful for verifying data is arriving.
Percentiles (p50, p90, p99, p99.9): the value below which N% of data points fall. Right for latency: p99 latency tells you what your slowest 1% of requests experience. Average latency hides tail latency completely.

Choosing the wrong statistic is one of the most common observability mistakes. Average latency on an API can look healthy at 50ms while p99 is 5 seconds.

bash

# Get Lambda Invocations with different statistics over the last hour
# Replace scheduler-lab-fn with any function that has recent invocations
START=$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)
END=$(date -u +%Y-%m-%dT%H:%M:%SZ)

for STAT in Sum Average SampleCount; do
  echo "=== $STAT ==="
  aws cloudwatch get-metric-statistics \
    --namespace AWS/Lambda \
    --metric-name Invocations \
    --dimensions Name=FunctionName,Value=scheduler-lab-fn \
    --start-time $START \
    --end-time $END \
    --period 300 \
    --statistics $STAT \
    --region $AWS_REGION \
    --query 'sort_by(Datapoints, &Timestamp)[*].{time:Timestamp,value:'$STAT'}'
done

# Now do the same for Duration -- Sum is meaningless, Max and p99 are useful
for STAT in Average Maximum; do
  echo "=== Duration $STAT ==="
  aws cloudwatch get-metric-statistics \
    --namespace AWS/Lambda \
    --metric-name Duration \
    --dimensions Name=FunctionName,Value=scheduler-lab-fn \
    --start-time $START \
    --end-time $END \
    --period 300 \
    --statistics $STAT \
    --region $AWS_REGION \
    --query 'sort_by(Datapoints, &Timestamp)[*].{time:Timestamp,value:'$STAT',unit:Unit}'
done

Observe: Sum for Invocations gives total calls per 5-minute window. Average for Invocations gives average calls per data point in the window, often less than 1, which is useless. Average for Duration is meaningful (mean execution time). Maximum for Duration shows your worst-case invocation in each window. Same metric, completely different signals.

Checkpoint: you can retrieve the same metric with different statistics and explain why each is appropriate or not.

Period and data retention tiers

What's happening here

CloudWatch does not keep all data at full resolution forever. It automatically downsamples older data into coarser buckets:

High-resolution (1-second) data: kept for 3 hours
Standard (1-minute) data: kept for 15 days
5-minute aggregates: kept for 63 days
1-hour aggregates: kept for 455 days (15 months)

This means if you query 1-minute data for events 20 days ago, you get nothing, it has been downsampled to 5-minute buckets. If you query 1-hour data for events 6 months ago, it still exists.

Period in your query must match the available resolution for the time range. Querying Period=60 (1 minute) for data from 3 weeks ago silently returns empty. CloudWatch doesn't error, it just returns no datapoints. This is a very common debugging trap.

bash

# Try querying 1-minute data from 20 days ago -- returns empty
START_OLD=$(date -u -d '20 days ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -v-20d +%Y-%m-%dT%H:%M:%SZ)
END_OLD=$(date -u -d '19 days ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -v-19d +%Y-%m-%dT%H:%M:%SZ)

echo "=== 1-minute period, 20 days ago (likely empty) ==="
aws cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name Invocations \
  --dimensions Name=FunctionName,Value=scheduler-lab-fn \
  --start-time $START_OLD \
  --end-time $END_OLD \
  --period 60 \
  --statistics Sum \
  --region $AWS_REGION \
  --query 'Datapoints'

# Now query 1-hour period for the same range -- data exists (if function was invoked then)
echo "=== 1-hour period, 20 days ago ==="
aws cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name Invocations \
  --dimensions Name=FunctionName,Value=scheduler-lab-fn \
  --start-time $START_OLD \
  --end-time $END_OLD \
  --period 3600 \
  --statistics Sum \
  --region $AWS_REGION \
  --query 'Datapoints'

Observe: 1-minute query for 20-day-old data returns []. Same time range with 1-hour period may return data. No error is thrown in either case. CloudWatch silently returns nothing when the resolution doesn't match the retention tier. If you're debugging "why is my graph empty", always check that your Period matches the age of the data.

Checkpoint: you understand why period selection matters and how data downsampling affects query results.

Break It: The Metric Data Model

Break 1: Query a metric before any data exists

bash

# Query a metric for a function that doesn't exist
aws cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name Invocations \
  --dimensions Name=FunctionName,Value=function-that-does-not-exist \
  --start-time $START \
  --end-time $END \
  --period 300 \
  --statistics Sum \
  --region $AWS_REGION

returns {"Datapoints": [], "Label": "Invocations"}. No error, no warning. CloudWatch treats missing data and zero data identically from the API perspective. This is why alarms have a treat_missing_data config, you must explicitly decide what missing data means for your alarm.

Break 2: Wrong statistic for a counter

bash

# Average on Invocations -- misleading result
aws cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name Invocations \
  --dimensions Name=FunctionName,Value=scheduler-lab-fn \
  --start-time $START \
  --end-time $END \
  --period 300 \
  --statistics Average \
  --region $AWS_REGION \
  --query 'Datapoints[*].{time:Timestamp,avg:Average}'

if Lambda was invoked 10 times in a 5-minute window, Sum returns 10. Average returns something like 1.0: because Lambda publishes one data point per invocation with value 1, and the average of ten 1s is 1. It looks like "1 invocation per period" regardless of actual traffic. Always use Sum for invocation counts and Average/Maximum/percentiles for latency.

Break 3: Period smaller than minimum for the namespace

bash

# Try period=1 (1 second) on a standard-resolution metric
aws cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name Invocations \
  --dimensions Name=FunctionName,Value=scheduler-lab-fn \
  --start-time $START \
  --end-time $END \
  --period 1 \
  --statistics Sum \
  --region $AWS_REGION

InvalidParameterValue: period must be a multiple of 60 for standard-resolution metrics. 1-second resolution requires publishing with StorageResolution=1 and querying with period=1. Standard AWS service metrics (Lambda, SQS, EC2) are standard-resolution, minimum period is 60 seconds.

Publishing Custom Metrics

Goal

Publish your own metrics to CloudWatch from the CLI and from Lambda. Understand dimensions, units, and storage resolution. Observe how wrong dimension values silently create new metric series.