Advanced8 min read

DynamoDB Parallel Scans

A parallel scan splits one Scan into N independent requests, each claiming a Segment of the table, so multiple workers read it at once.

What is a DynamoDB parallel scan?

A DynamoDB parallel scan splits one Scan into N independent requests, each claiming a Segment of the table via Segment and TotalSegments, so multiple workers read it concurrently. It's the only way the Scan API offers to read a whole table faster than a single partition's throughput allows. It's still a full read, so you pay for every item scanned.

  • A sequential Scan reads one partition at a time. Its speed is capped at a single partition's throughput, no matter how big the table is.
  • Segment + TotalSegments shard the read across TotalSegments workers; each worker scans its own slice in parallel.
  • DynamoDB hashes the to assign segments, so slices can be lopsided and more workers doesn't always mean faster.
  • It raises the read rate, not the total. You pay to read every item either way, and a wide parallel scan can drain the throughput your live traffic needs.

Why a sequential Scan is slow

Coming from SQL, a full-table read feels like one streaming operation. In DynamoDB it isn't. The table's data lives across many physical partitions, but a single Scan walks them one at a time, 1 MB per page.

So a plain Scan can only pull from one partition's throughput budget at a moment, even if the table is spread across dozens of partitions with idle capacity. The bigger the table, the longer it crawls. (AWS: Parallel scan)

Split the read with Segment and TotalSegments

A parallel scan fixes the bottleneck. You pick a worker count, set TotalSegments to that number, and give each worker a distinct zero-based Segment. Every worker issues its own Scan; DynamoDB serves them concurrently.

Worker 0 →  Scan  Segment=0  TotalSegments=4
Worker 1 →  Scan  Segment=1  TotalSegments=4
Worker 2 →  Scan  Segment=2  TotalSegments=4
Worker 3 →  Scan  Segment=3  TotalSegments=4

Each worker still pages with LastEvaluatedKey independently and owns its segment from first page to last.

A worker's LastEvaluatedKey only resumes its own segment, so a resumable job stores Segment and TotalSegments next to it (pagination). With partition keys spread evenly, four workers read up to four times as fast as one.

A worked example: fixing every reading in place

Say you run a telemetry table, sensor-readings. Each item is one reading from a field device:

PK = "DEVICE#a83f"          (partition key, the device id)
SK = "TS#2026-06-22T03:14"  (sort key, ISO timestamp)
batteryMv  = 3120
tempC      = 41.8
firmwareTag = "fw-7.2.1"

A firmware bug made devices on fw-7.2.1 report tempC in Fahrenheit, and every affected reading needs correcting in place. An export can't do this: the job has to read each live item and write the fix back.

A sequential Scan of 80 GB takes about an hour at best. One partition serves at most 3,000 read units per second, which is ~23 MB/s of eventually consistent reads, before any client round-trip time. So you fan it out across eight workers:

Scan  sensor-readings  Segment=0  TotalSegments=8  ConsistentRead=false
…
Scan  sensor-readings  Segment=7  TotalSegments=8  ConsistentRead=false

Eight workers read up to eight times as fast, when the devices hash evenly across segments. AWS is explicit that more segments don't guarantee it (AWS: Parallel scan).

Each fix sets a tempFixedAt marker, conditioned on attribute_not_exists(tempFixedAt), so a rerun after a crash skips the readings already done.

How DynamoDB assigns items to segments

DynamoDB assigns each item to a segment by hashing its partition key, not by row count or byte count.

So every item sharing a PK lands in the same segment. In sensor-readings, all readings for DEVICE#a83f go to one worker, regardless of how many timestamps that device has or how big its is. (AWS: Parallel scan)

sensor-readings tablehash partition keySegment 0DEVICE#a83fDEVICE#1c20Segment 1DEVICE#9be4Segment 2 (empty)

Segments end up uneven. One worker might own three chatty devices with millions of readings; another might draw an empty slice. Raising TotalSegments won't split one device's readings, because they all share a hash.

What to do about skew:

  • Measure, then tune. Start with a modest TotalSegments, log each worker's ScannedCount and run time, and adjust. AWS's own advice is to experiment with the segment and worker counts.
  • Accept the floor. The biggest item collection sets the minimum run time, since one worker reads it alone. More segments only help the rest.
  • For a full copy, skip the scan. Export to S3 doesn't read through your segments or your throughput at all.

See the read cost before you run it

A parallel scan costs the same total as a sequential one: you just pay faster. Price it before you start.

DescribeTable returns TableSizeBytes (updated about every six hours), and an eventually consistent full read costs about that size ÷ 4 KB × 0.5 read units.

For the 80 GB table that's about 10.5 million read units, roughly $1.31 per run at us-east-1 on-demand list price ($0.125 per million). On a provisioned table, compare the rate your workers will draw with the capacity your live traffic needs.

DynoTable checks the size for you: before it reads a large table to the end (over about 1 GiB or 5 million items, by DynamoDB's approximate table size) it asks you to confirm, showing the approximate size and item count. Then it streams live items-scanned progress as the read runs.

Pitfalls and when not to bother

  • The throughput cliff. A high-TotalSegments scan can consume the table's entire read capacity in seconds, starving live traffic. On a table serving users, cap each worker's request size with the Limit parameter, pause between pages, or scan off-peak. Throttling covers what to do when requests start failing. (AWS: Parallel scan)
  • It's still the wrong tool for an access pattern. Parallel scans are for deliberate full-table jobs: backfills, migrations, audits. If you're reaching for one to answer a recurring query, add a GSI and make it a Query instead.
  • PartiQL SELECT without a key is the same scan, sequential. Put the partition key in the WHERE, or block full scans in IAM: why Scan is slow and expensive shows both. DynoTable's SQL Workbench computes a GROUP BY or JOIN on your machine, but an unkeyed query still Scans its base table.
  • Strong consistency doubles the bill. A Scan defaults to reads. Leave ConsistentRead=false unless each page must reflect the very latest writes. Even a strongly consistent scan spans minutes, so it's not a point-in-time snapshot: Export to S3 from a chosen point in time is.

Next steps

Model your keys so day-to-day reads never need a scan: start with single-table design, Query vs Scan, and why Scan is slow and expensive.

Build the key conditions with the Expression Builder. When a full-table job is the right call, try DynoTable to run it with a size warning up front on large tables and live scan progress.

By Dmitriy · Updated