DynamoDB Parallel Scans
A parallel scan splits one Scan into N independent requests, each claiming a
Segment of the table, so multiple workers read it at once.
What is a DynamoDB parallel scan?
A DynamoDB parallel scan splits one Scan into N independent requests, each claiming a Segment of the table via Segment and TotalSegments, so multiple workers read it concurrently. It's the only way the Scan API offers to read a whole table faster than a single partition's throughput allows. It's still a full read, so you pay for every item scanned.
- A sequential
Scanreads one partition at a time. Its speed is capped at a single partition's throughput, no matter how big the table is. Segment+TotalSegmentsshard the read acrossTotalSegmentsworkers; each worker scans its own slice in parallel.- DynamoDB hashes the partition key to assign segments, so slices can be lopsided and more workers doesn't always mean faster.
- It raises the read rate, not the total. You pay to read every item either way, and a wide parallel scan can drain the throughput your live traffic needs.
Why a sequential Scan is slow
Coming from SQL, a full-table read feels like one streaming operation. In DynamoDB it
isn't. The table's data lives across many physical partitions, but a single Scan
walks them one at a time, 1 MB per page.
So a plain Scan can only pull from one partition's throughput budget at a moment,
even if the table is spread across dozens of partitions with idle capacity. The bigger
the table, the longer it crawls.
(AWS: Parallel scan)
Split the read with Segment and TotalSegments
A parallel scan fixes the bottleneck. You pick a worker count, set TotalSegments
to that number, and give each worker a distinct zero-based Segment. Every worker
issues its own Scan; DynamoDB serves them concurrently.
Worker 0 → Scan Segment=0 TotalSegments=4
Worker 1 → Scan Segment=1 TotalSegments=4
Worker 2 → Scan Segment=2 TotalSegments=4
Worker 3 → Scan Segment=3 TotalSegments=4
Each worker still pages with LastEvaluatedKey independently and owns its segment
from first page to last.
A worker's LastEvaluatedKey only resumes its own segment, so a resumable job stores
Segment and TotalSegments next to it (pagination).
With partition keys spread evenly, four workers read up to four times as fast as one.
A worked example: fixing every reading in place
Say you run a telemetry table, sensor-readings. Each item is one reading from a
field device:
PK = "DEVICE#a83f" (partition key, the device id)
SK = "TS#2026-06-22T03:14" (sort key, ISO timestamp)
batteryMv = 3120
tempC = 41.8
firmwareTag = "fw-7.2.1"
A firmware bug made devices on fw-7.2.1 report tempC in Fahrenheit, and every
affected reading needs correcting in place. An export can't do this: the job has to
read each live item and write the fix back.
A sequential Scan of 80 GB takes about an hour at best. One partition serves at
most 3,000 read units per second, which is ~23 MB/s of eventually consistent reads,
before any client round-trip time. So you fan it out across eight workers:
Scan sensor-readings Segment=0 TotalSegments=8 ConsistentRead=false
…
Scan sensor-readings Segment=7 TotalSegments=8 ConsistentRead=false
Eight workers read up to eight times as fast, when the devices hash evenly across segments. AWS is explicit that more segments don't guarantee it (AWS: Parallel scan).
Each fix sets a tempFixedAt marker, conditioned on
attribute_not_exists(tempFixedAt), so a rerun after a crash skips the readings
already done.
How DynamoDB assigns items to segments
DynamoDB assigns each item to a segment by hashing its partition key, not by row count or byte count.
So every item sharing a PK lands in the same segment. In sensor-readings, all readings for DEVICE#a83f go to one worker, regardless of
how many timestamps that device has or how big its item collection
is.
(AWS: Parallel scan)
Segments end up uneven. One worker might own three chatty devices with millions
of readings; another might draw an empty slice. Raising TotalSegments won't split
one device's readings, because they all share a hash.
What to do about skew:
- Measure, then tune. Start with a modest
TotalSegments, log each worker'sScannedCountand run time, and adjust. AWS's own advice is to experiment with the segment and worker counts. - Accept the floor. The biggest item collection sets the minimum run time, since one worker reads it alone. More segments only help the rest.
- For a full copy, skip the scan. Export to S3 doesn't read through your segments or your throughput at all.
See the read cost before you run it
A parallel scan costs the same total as a sequential one: you just pay faster. Price it before you start.
DescribeTable returns TableSizeBytes (updated about every six hours), and an eventually consistent full read costs about that size ÷ 4 KB × 0.5
read units.
For the 80 GB table that's about 10.5 million read units, roughly $1.31 per run at us-east-1 on-demand list price ($0.125 per million). On a provisioned table, compare the rate your workers will draw with the capacity your live traffic needs.
DynoTable checks the size for you: before it reads a large table to the end (over about 1 GiB or 5 million items, by DynamoDB's approximate table size) it asks you to confirm, showing the approximate size and item count. Then it streams live items-scanned progress as the read runs.
Pitfalls and when not to bother
- The throughput cliff. A high-
TotalSegmentsscan can consume the table's entire read capacity in seconds, starving live traffic. On a table serving users, cap each worker's request size with theLimitparameter, pause between pages, or scan off-peak. Throttling covers what to do when requests start failing. (AWS: Parallel scan) - It's still the wrong tool for an access pattern. Parallel scans are for deliberate full-table jobs: backfills, migrations, audits. If you're reaching for one to answer a recurring query, add a GSI and make it a Query instead.
- PartiQL
SELECTwithout a key is the same scan, sequential. Put the partition key in theWHERE, or block full scans in IAM: why Scan is slow and expensive shows both. DynoTable's SQL Workbench computes aGROUP BYorJOINon your machine, but an unkeyed query still Scans its base table. - Strong consistency doubles the bill. A
Scandefaults to eventually consistent reads. LeaveConsistentRead=falseunless each page must reflect the very latest writes. Even a strongly consistent scan spans minutes, so it's not a point-in-time snapshot: Export to S3 from a chosen point in time is.
Next steps
Model your keys so day-to-day reads never need a scan: start with single-table design, Query vs Scan, and why Scan is slow and expensive.
Build the key conditions with the Expression Builder. When a full-table job is the right call, try DynoTable to run it with a size warning up front on large tables and live scan progress.