Amazon Kinesis

Overview

Amazon Kinesis is the name AWS uses for a group of managed services that handle data in motion: events that arrive continuously, such as website clicks, application logs, payment events or readings from sensors. The central service, Kinesis Data Streams, is a durable, ordered log that many producers can write to and many consumers can read from within seconds. Around it sit services for delivering streams to storage and analytics tools, for processing them and for video.

Streaming changes how a system reacts. In a batch system, data is collected all day and processed at night. With a stream, each event is available to readers almost immediately, so a fraud check, a dashboard or an alert can respond while the event still matters. Kinesis is fully managed: AWS runs the servers, replicates the data across availability zones and handles scaling, especially in on-demand mode.

An everyday analogy is a conveyor belt in a busy sorting office. Parcels (records) are placed on the belt by many senders. The belt is divided into lanes (shards), and parcels with the same postcode always go in the same lane, in order. Several teams can watch the belt and pick up copies of parcels at their own pace, and the belt keeps parcels for a day or longer, so a team that falls behind can catch up.

What it is

Where it came from

AWS announced Amazon Kinesis at re:Invent in November 2013 as a limited preview, and public availability followed in December 2013. The original service is what is now called Kinesis Data Streams. Kinesis Firehose launched in October 2015 for loading streams into storage, and Kinesis Analytics launched in August 2016 for SQL over streams. Kinesis Video Streams arrived in November 2017.

Data Streams gained enhanced fan-out in August 2018, retention of up to one year in November 2020 and on-demand capacity mode in November 2021. In August 2023 Kinesis Data Analytics was renamed Amazon Managed Service for Apache Flink, and in February 2024 Kinesis Data Firehose was renamed Amazon Data Firehose. The Kinesis family therefore now refers mainly to Data Streams and Video Streams, with the other two services continuing under new names.

How it works

Writing records

A producer sends a record with a data payload and a partition key. Kinesis hashes the partition key to choose a shard. All records with the same key land in the same shard, in the order they were written, and receive increasing sequence numbers. Data is replicated across three availability zones in the region before the write is acknowledged.

In provisioned mode each shard accepts up to 1 MB per second or 1,000 records per second of writes. Producers that exceed this receive throttling errors and should retry with backoff. Batching records with PutRecords, up to 500 records per request, raises efficiency.

Reading records

Consumers read each shard from a position, such as the latest record, the oldest retained record or a given sequence number or time. In the shared-throughput model, each shard supports up to 2 MB per second of reads in total, shared by all consumers that poll it. With enhanced fan-out, each registered consumer gets its own 2 MB per second per shard, pushed over an HTTP/2 connection, which suits several independent applications reading the same stream.

The Kinesis Client Library handles the hard parts for consumer applications: it balances shards across workers, tracks progress with checkpoints stored in DynamoDB and reacts when shards split or merge. AWS Lambda can also read a stream and invoke a function on batches of records.

Retention and replay

Records stay in the stream for the retention period, 24 hours by default and configurable up to 365 days. Any consumer can rewind and re-read within that window, which helps with recovering from bugs or adding a new consumer to historical data.

Scaling

In provisioned mode you split hot shards and merge cold ones, or set the shard count with UpdateShardCount. In on-demand mode Kinesis adjusts capacity automatically based on traffic, starting at 4 MB per second of writes, and you pay per gigabyte written and read rather than per shard-hour.

import boto3, json
kinesis = boto3.client("kinesis")
kinesis.put_record(StreamName="orders",
                   Data=json.dumps({"order_id": 42, "total": 19.99}),
                   PartitionKey="customer-1001")

Key concepts and terms

A worked example

This scenario is illustrative. An online grocery service wants live order dashboards, fraud checks within seconds and an archive of all events in its data lake. At peak it sees about 3,000 order events per second, each about 1 KB.

1. Peak traffic is roughly 3 MB per second of writes. In provisioned mode that needs at least three shards; the team provisions six to leave headroom and spread hot keys. Later they switch to on-demand mode because traffic varies sharply between weekdays and holidays.
2. Order services write events with customer_id as the partition key, so each customer's events stay in order.
3. A fraud service, built with the Kinesis Client Library, registers for enhanced fan-out, reads every event within about a second and flags suspicious patterns.
4. A Flink application on Amazon Managed Service for Apache Flink computes revenue per store in one-minute windows for the operations dashboard.
5. Amazon Data Firehose reads the same stream, buffers records for a few minutes, converts them to Parquet and writes them to S3 for the data lake.
6. Retention is set to seven days. When a bug is found in the fraud service, the team deploys a fix and replays the past two days.

Strengths and limitations

Strengths:

• Fully managed, durable and replicated across availability zones.
• Low latency from write to read, typically within seconds or less.
• Several consumers can read the same data independently, with replay.
• On-demand mode removes most capacity planning.
• Deep integration with Lambda, Firehose, Flink, Redshift and other AWS services.

Limitations:

• Ordering is guaranteed only within a shard, not across a whole stream.
• Per-shard limits require careful partition keys to avoid hot shards in provisioned mode.
• AWS-specific APIs, so moving to another platform means changing producers and consumers.
• Retention beyond 24 hours and enhanced fan-out add cost.
• Fewer ecosystem tools and connectors than Apache Kafka.

How it compares with alternatives

Practical guidance and best practices

1. Choose partition keys with many distinct values so load spreads evenly across shards.
2. Batch writes with PutRecords or the Kinesis Producer Library, and retry throttled records with backoff.
3. Use on-demand mode for unpredictable traffic and provisioned mode for steady, well-understood loads.
4. Use enhanced fan-out when several consumers need full throughput and low latency.
5. Make consumers idempotent; retries can deliver the same record more than once.
6. Monitor iterator age, throttling and incoming bytes in CloudWatch, and alert on growing lag.
7. Set retention long enough to cover realistic recovery and replay needs.
8. Encrypt streams with KMS and grant least-privilege IAM permissions to producers and consumers.

Common mistakes

• Using a constant or low-cardinality partition key, which sends everything to one shard.
• Assuming records are ordered across the whole stream rather than within each shard.
• Writing records one at a time at high volume instead of batching.
• Letting many polling consumers share one shard's 2 MB per second read limit.
• Ignoring duplicates, which leads to double counting downstream.
• Using a stream as a long-term database instead of archiving to S3.

Frequently asked questions

What is the difference between Kinesis Data Streams and Data Firehose?

Data Streams is a durable stream that your own applications read and process, with replay. Amazon Data Firehose, formerly Kinesis Data Firehose, is a delivery service: it buffers incoming data and loads it into destinations such as S3, Redshift or OpenSearch, with optional transformation and format conversion. They are often used together.

How long does Kinesis keep data?

By default, 24 hours. You can extend retention up to 365 days, at extra cost. Within that window consumers can re-read records.

Does Kinesis guarantee ordering?

Within a shard, yes: records are stored and read in the order they were written. Across shards there is no global order. Use the same partition key for records that must stay in order.

How do I scale a Kinesis stream?

In provisioned mode, add shards by splitting or with UpdateShardCount. In on-demand mode Kinesis scales automatically based on recent traffic. Either way, even key distribution matters.

Kinesis or Kafka?

Kinesis is simpler to operate on AWS and integrates closely with its services. Kafka, self-managed or through Amazon MSK, offers wider tooling, portability and more configuration control. Teams standardised on Kafka elsewhere often choose MSK.

Can Kinesis deliver exactly once?

Kinesis Data Streams provides at-least-once delivery, so consumers may see duplicates after retries. Exactly-once results need idempotent processing or a framework such as Flink with transactional sinks.

Glossary

• Availability zone: an isolated data centre location within an AWS region.
• Backoff: waiting longer between retries after throttling.
• Hot shard: a shard receiving more traffic than its limits allow.
• Idempotent consumer: a consumer whose result is the same if a record is processed twice.
• KCL: Kinesis Client Library for building consumer applications.
• KPL: Kinesis Producer Library for efficient, batched writes.
• Replay: re-reading older records within the retention period.
• Throttling: rejection of requests that exceed capacity limits.

Summary

Amazon Kinesis gives AWS users managed real-time streams. Data Streams stores ordered records in shards, replicates them across availability zones and lets many consumers read and replay them, while Amazon Data Firehose and Amazon Managed Service for Apache Flink handle delivery and processing. Good partition keys, batching, idempotent consumers and lag monitoring are the core habits. For portability or a wider connector ecosystem, compare it with Apache Kafka and Amazon MSK, and see real-time systems and event-driven architecture for the wider design picture.

Related

It belongs to Streaming in my stack, under Big data. Others in the same group: