Data Streams

SQS is a great message queue. But it has a fundamental constraint: each message is consumed by exactly one consumer and then deleted. If you need five different systems to process the same event, a payment event that must go to fraud detection, analytics, audit logging, the invoice service, and a real-time dashboard simultaneously. SQS requires you to fan out via SNS or maintain five separate queues. More importantly, SQS is a queue of discrete tasks. It is not a log. Once a message is consumed and deleted, it's gone. You cannot go back and re-read the last hour of events. You cannot replay messages after a bug is fixed. You cannot add a new consumer six months from now and have it process historical data. Kinesis Data Streams solves a different class of problem: continuously ingesting and processing high-volume, ordered streams of data where multiple consumers need the same data, and where the ability to replay is essential. Typical use cases:

Amazon Kinesis Data Streams (KDS) is a real-time data streaming service. Producers write records to a stream. Multiple independent consumers read from the stream in parallel. Records are retained for a configurable period (1–365 days), enabling replay. Key properties:

A stream is the top-level resource. It has a name, a shard count, and a retention period. Producers write to it; consumers read from it. A stream is region-specific.

The unit of data in Kinesis. Every record has:

The fundamental throughput unit of a Kinesis stream. A shard provides:

A stream with 10 shards handles 10MB/s writes and 20MB/s reads. If your throughput exceeds the shard capacity, you get ProvisionedThroughputExceededException and must add more shards (resharding).

The partition key determines which shard a record lands on. Kinesis hashes the partition key to a 128-bit integer and maps it to a shard. Records with the same partition key always go to the same shard, this is how ordering is guaranteed within a logical grouping. Choose partition keys based on what ordering you need:

Hot shard problem: if most of your traffic has the same partition key (e.g. all events for one viral user), one shard gets all the write traffic while others sit idle. Monitor IncomingBytes and IncomingRecords per shard.

Producers write records using the Kinesis API:

AWS provides higher-level producer libraries:

Service integrations (no code needed): CloudWatch Logs, DynamoDB Streams, IoT Core, and Kinesis Data Firehose can all write directly to a Kinesis stream.

This is where Kinesis differs most significantly from SQS. Multiple independent consumers read the same data simultaneously, each maintaining their own checkpoint (position in the stream).

The default consumption model. All consumers of a stream share the 2MB/s read capacity per shard. If you have 5 consumers and 5 shards, each consumer gets 2MB/s per shard, but they share it, so effectively 400KB/s each. At high consumer counts this becomes a bottleneck.

Each registered consumer gets its own dedicated 2MB/s per shard pipe, independent of other consumers. Kinesis pushes records to the consumer (push model) rather than the consumer polling (pull model). Sub-100ms latency. Cost: $0.015 per shard-hour + $0.013 per GB of data retrieved. More expensive than shared throughput but essential when you have many consumers or need the lowest latency.

Use enhanced fan-out when: you have more than 2–3 consumers, you're hitting read throttling, or you need consistent sub-second latency.

Lambda via Event Source Mapping: Lambda polls the stream on your behalf. Invokes your function with a batch of records from one or more shards. One Lambda invocation per shard per batch interval. Simplest option, no polling code, no checkpoint management.

Kinesis Client Library (KCL): a Java library that manages shard enumeration, checkpointing (via DynamoDB), load balancing across consumer workers, and failover. Best for high-throughput, complex processing logic, or non-Lambda consumers.

GetRecords API (DIY): you poll shards directly using GetShardIterator + GetRecords. Complete control, but you manage checkpointing, shard discovery, and failover yourself. Only use if KCL doesn't fit your language/environment.

Unlike SQS (where deletion marks completion), Kinesis consumers track their position via checkpoints: a pointer to the last successfully processed sequence number per shard. If a consumer crashes and restarts, it resumes from its last checkpoint: no messages lost, no re-reading from the beginning. Checkpoint storage:

At-least-once delivery: if your consumer crashes after processing but before checkpointing, records are re-delivered after restart. Your processing logic must be idempotent. Use the sequence number as an idempotency key.

When a consumer begins reading, it specifies where in the shard to start via a shard iterator type:

As your throughput grows, you add shards (split). As it shrinks, you merge shards to reduce cost.

Split: divide one shard into two. The original shard becomes a parent; the two new shards are children. Existing data remains in the parent until its retention period expires, consumers must process the parent before the children.

Merge: combine two adjacent shards into one. The two originals become parents; the merged shard is the child. Resharding is not instant, it takes a few seconds to a minute. During resharding, the stream continues to accept writes. Old shards become CLOSED (no new writes) and EXPIRED (after retention).

Auto-scaling shards: Kinesis does not auto-scale shards. You must monitor WriteProvisionedThroughputExceeded and ReadProvisionedThroughputExceeded metrics and reshard manually or via custom automation (Lambda + CloudWatch Alarm + Kinesis scaling API).

Retention is one of Kinesis's most powerful features and the clearest differentiator from SQS.

What you can do with retention:

Cost implication: extended retention ($0.023 per shard-hour beyond 24h) adds up at scale. Only enable long retention on streams where replay is genuinely needed.

These three services are the most commonly confused in AWS architectures.

Choose Kinesis when: multiple consumers need the same data, ordering matters, you need replay, or you're processing high-throughput continuous data streams.

Choose SQS when: one consumer processes each message, ordering doesn't matter (or use FIFO), you don't need replay, and you want simplicity.

Choose SNS when: you need to broadcast a single event to multiple systems immediately, with no persistence requirement.

Kinesis has multiple services with similar names. They solve different problems:

Kinesis Data Streams (KDS): a real-time stream you manage. You write producers. You write consumers. You manage shards. You get full control and very low latency (\<1 second). Use when you need custom processing logic or very low latency.

Kinesis Data Firehose: a managed delivery pipeline. You configure a source (KDS, MSK, or direct PUT) and a destination (S3, Redshift, OpenSearch, Splunk, HTTP endpoint). Firehose buffers data, optionally transforms it via Lambda, and delivers in batches (60 second minimum interval). No consumers to write. No shards to manage. Use when you need to land streaming data into a storage system with minimal code.

Common pattern: KDS → Lambda (real-time processing) + KDS → Firehose → S3 (raw archival). Both consumers read from the same stream simultaneously.

Every user action processed within seconds. Lambda scales with shard count.

All three read the same records. None affect the others.

Centralised log pipeline without managing Logstash or Fluentd infrastructure.

Example: 10 shards, 24h retention, 100 million records/day at 1KB average:

For comparison, SQS at 100M messages/day = ~$40/day (after free tier). Kinesis can be cheaper at high throughput because the per-record cost is very low once shards are provisioned.