Advanced9 min read

DynamoDB Streams: React to Every Item Change

DynamoDB Streams is a change-data-capture log: every change to an item (an insert, an update, or a delete) becomes a stream record you can react to. It's how you turn a table into an event source without polling it.

The running example is an append-only audit log for a multi-tenant SaaS: one item per event, partitioned by tenant (PK = TENANT#acme, SK = EVENT#<timestamp>#<id>).

You want to react the moment a sensitive event lands, such as an invoice export or an admin-role grant, without scanning the table on a timer.

How do DynamoDB Streams work?

You enable a stream on the table, choose what each record carries, and attach a consumer that receives the changes in batches.

  • A stream holds item-level changes for 24 hours. Each record appears once in the stream, and a write that changes nothing produces no record.
  • StreamViewType sets what a record carries: keys only, the new image, the old image, or both. It's fixed once the stream exists.
  • Order is guaranteed per item only. Changes to one item arrive in the order they happened; nothing orders records across different items.
  • Lambda is the usual consumer. Its event source mapping polls the shards and invokes your function with a batch of records.
  • For more than 24 hours, use Kinesis Data Streams for DynamoDB (retention up to 365 days), and Amazon Data Firehose to archive the changes in S3.

The problem: reacting without polling

You need "alert me when a role.granted event is written." The naive approach is a job that scans the table, or queries each tenant's recent events, every minute. It pays for every read and is always up to a minute late.

What you want is a push: the change record delivered to your code the moment the item changes. That's what Streams provides.

How Streams works

When you enable a stream you pick a StreamViewType, which controls how much of the changed item each record carries (change data capture for DynamoDB):

StreamViewTypeeach record contains
KEYS_ONLYonly the key attributes of the changed item
NEW_IMAGEthe entire item as it looks after the change
OLD_IMAGEthe entire item as it looked before the change
NEW_AND_OLD_IMAGESboth the before and after images

You can't edit the view type later: you disable the stream and enable a new one, which gets a new stream ARN. Each open shard maps to exactly one table partition, so the stream grows shards as the table splits.

DynamoDB doesn't call your code. Lambda's event source mapping polls each shard about four times a second and invokes your function with a batch of new records (Lambda with DynamoDB). Those stream reads aren't billed for a standard Lambda trigger.

LambdaStream"DynamoDB"AppLambdaStream"DynamoDB"App"Put EVENT role.granted""change record(NEW_IMAGE)""poll shard (about 4x persecond)""batch of records""if action issensitive, alert"

A worked example: alert on sensitive audit events

Turn the stream on with NEW_IMAGE, so each record carries the full new event:

aws dynamodb update-table --table-name audit-log \
  --stream-specification StreamEnabled=true,StreamViewType=NEW_IMAGE

Then attach the alert function. The filter lets Lambda drop every record except the two sensitive actions before your code runs, so you aren't invoked for logins:

aws lambda create-event-source-mapping --function-name audit-alert \
  --event-source-arn "$STREAM_ARN" --starting-position LATEST \
  --filter-criteria '{"Filters":[{"Pattern":"{\"dynamodb\":{\"NewImage\":{\"action\":{\"S\":[\"invoice.export\",\"role.granted\"]}}}}"}]}'

$STREAM_ARN is the LatestStreamArn that aws dynamodb describe-table returns. The records play out like this:

stream record (NEW_IMAGE)consumer action
TENANT#acmeEVENT#…#a2action=invoice.exportsend to SIEM
TENANT#globex EVENT#…#b9 action=role.grantedpage on-call
TENANT#acmeEVENT#…#a1action=login.successfiltered out, no invocation

The function never reads the table. It reacts to what the stream hands it, and the alert fires within seconds of the write.

This is also the standard way to maintain a downstream copy: a stream consumer can project each event into OpenSearch for full-text audit search, or keep running counts, all derived from the same change log.

Keep changes longer than 24 hours

Stream records are trimmed after 24 hours. If your consumer is down longer than that, those changes never reach it. The audit-log table still holds the events, but the SIEM and the search index miss them. Pick the durable path by need:

You needUseWhy
A replay window past 24 h, or more consumersKinesis Data Streams for DynamoDBRetention from 24 h up to 365 days; enhanced fan-out
A permanent archive of every changeKinesis Data Streams for DynamoDB, then Amazon Data Firehose to S3Firehose reads a Kinesis data stream directly
Per-item order from Streams, plus an archiveThe DynamoDB stream into an EventBridge Pipe (or a Lambda), FirehoseA Pipe keeps record order and can target Firehose directly

Kinesis Data Streams for DynamoDB is a separate change feed you turn on per table. Create the Kinesis data stream first (on-demand mode needs no shard count), wait for it, point the table at it, and raise its retention for a longer replay window:

aws kinesis create-stream --stream-name audit-changes \
  --stream-mode-details StreamMode=ON_DEMAND
aws kinesis wait stream-exists --stream-name audit-changes
aws dynamodb enable-kinesis-streaming-destination --table-name audit-log \
  --stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/audit-changes
aws kinesis increase-stream-retention-period --stream-name audit-changes \
  --retention-period-hours 8760

For the permanent archive, create a Firehose stream that reads audit-changes and writes to S3. The IAM role must let Firehose read the stream and write the bucket:

aws firehose create-delivery-stream --delivery-stream-name audit-archive \
  --delivery-stream-type KinesisStreamAsSource \
  --kinesis-stream-source-configuration KinesisStreamARN=arn:aws:kinesis:us-east-1:123456789012:stream/audit-changes,RoleARN=arn:aws:iam::123456789012:role/audit-firehose \
  --extended-s3-destination-configuration BucketARN=arn:aws:s3:::audit-archive,RoleARN=arn:aws:iam::123456789012:role/audit-firehose

Kinesis Data Streams for DynamoDB has two costs. Its records can arrive out of order and more than once, so order and dedupe on each record's ApproximateCreationDateTime (Kinesis Data Streams for DynamoDB).

And you pay twice: change data capture units on the table, plus Kinesis charges, which rise for retention above 24 hours (extended retention).

Firehose can't read a DynamoDB stream itself. Its sources are Direct PUT, a Kinesis data stream or Amazon MSK (Firehose sources), so the archive goes through Kinesis, a Pipe, or a Lambda that calls PutRecordBatch.

Handle consumer failures

Lambda processes each record at least once, so a retried batch can page on-call twice. Make the function idempotent: record each event's SK (or the stream record's eventID) and skip one you've already handled.

With default settings, a batch that keeps failing blocks its shard until the records expire, up to a day (Lambda error handling). Cap that on the event source mapping:

aws lambda update-event-source-mapping --uuid "$ESM_UUID" \
  --maximum-retry-attempts 3 --maximum-record-age-in-seconds 3600 \
  --bisect-batch-on-function-error \
  --function-response-types ReportBatchItemFailures \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:s3:::audit-stream-failures"}}'

Bisecting splits a failing batch to isolate the bad record. What still fails goes to the on-failure destination.

ReportBatchItemFailures does nothing until the function returns the failures: { batchItemFailures: [{ itemIdentifier: record.dynamodb.SequenceNumber }] }. Lambda then retries from the lowest failed sequence number, so the records before it aren't re-run (partial batch response).

An S3 destination receives the full batch payload. An SQS or SNS destination receives only metadata pointing back into the stream, which you must read before the 24 hours run out.

Do it in DynoTable

Before you wire up a stream consumer, you need the exact shape of the item your Lambda will receive: which attributes exist, how nested maps and lists look, what a NEW_IMAGE record will actually contain.

To convert a sample item between plain JSON and the attribute-value shape a stream record uses, the DynamoDB JSON Converter does it in your browser. And in DynoTable, you can inspect the full item, including its DynamoDB-JSON form, so you model the NEW_IMAGE record against real data instead of guessing the field shape.

Inspecting an audit event item in DynoTable to model the NEW_IMAGE stream record its Lambda consumer will receive.
Inspecting an audit event item in DynoTable to model the NEW_IMAGE stream record its Lambda consumer will receive.

DynamoDB Local supports streams, so you can run the table locally, read the records your consumer will get with GetShardIterator and GetRecords, and inspect the items the same way. See connect to DynamoDB Local.

TTL deletes are the exception: on Local their records carry no userIdentity and no REMOVE event name, so a TTL-archive filter matches nothing there.

Pitfalls and next steps

  • 24 hours is a reaction buffer, not an archive. If a downstream system must never miss a change, add Kinesis Data Streams for DynamoDB and Firehose, as shown above.
  • Pick the view type by what the consumer must know. Use NEW_IMAGE to act on the new state and NEW_AND_OLD_IMAGES to diff. KEYS_ONLY plus a re-read costs a read per record and returns the item as it is now, or nothing after a delete.
  • Order holds per item, not across items. If the consumer needs order across items, carry it in the data (a time-sortable sort key or a sequence number you write) and order on that. Lambda's ParallelizationFactor keeps each item's order while it processes a shard in parallel.
  • Two readers per shard. A third consumer reading the same stream gets throttled (on a global table AWS recommends one). Fan out from one function, or use Kinesis Data Streams for DynamoDB.
  • TTL deletes carry a marker. A TTL delete's record has userIdentity set to type: "Service" and principalId: "dynamodb.amazonaws.com". To archive expiring items, filter on it and check it again in the function, as DynamoDB TTL shows.

Streams turns the audit log into an event source. The next operational concern is the opposite end of an item's life: expiring old events automatically with DynamoDB TTL.

Download DynoTable to inspect the exact item shape your stream consumer will receive before you write a line of Lambda code.

By Dmitriy · Updated