DynamoDB Batch Write in Python (boto3 batch_writer)

batch_writer() is the one DynamoDB call where Python is less work than the other SDKs. It buffers puts and deletes, cuts them into BatchWriteItem requests of 25, and resends unprocessed items itself. What it does not do is protect you from the two failures that break most bulk loads, and both of them surface at the flush rather than at the line that supplied the bad item.

Code

import boto3

dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table("Music")

songs = [
    {"Artist": "Arturo Sandoval", "SongTitle": "Cubano Chant", "AlbumTitle": "Danzon", "Year": 1994},
    {"Artist": "Arturo Sandoval", "SongTitle": "A Mis Abuelos", "AlbumTitle": "Danzon", "Year": 1994},
    {"Artist": "Arturo Sandoval", "SongTitle": "Groovin' High", "AlbumTitle": "Swingin'", "Year": 1996},
]

with table.batch_writer() as batch:
    for song in songs:
        batch.put_item(Item=song)
    # batch_writer buffers deletes too — target a key you're NOT also putting
    # (two writes to the same key in one batch are rejected as a duplicate)
    batch.delete_item(Key={"Artist": "Ella Fitzgerald", "SongTitle": "Misty"})

print(f"Buffered {len(songs)} puts + 1 delete; the batch flushes on exit")

Explanation

Deferred flush — batch.put_item() appends to a list. Nothing is validated, serialised, or sent until the buffer reaches 25 or the with block exits, so the traceback for a bad item comes from the flush and not from the put_item call that supplied it. If you are loading from an iterator, keep your own index of what went into the buffer.
Plain Python values — this is the resource API, so you write 1994, not {"N": "1994"}. Decimals are required for anything fractional; a float is accepted into the buffer and rejected on flush.
batch_writer() is a Table method. The read-side counterpart is not: batch_get_item lives on the ServiceResource, and table.batch_get_item does not exist. There is no buffering, chunking, or retry helper for batch reads at all.
UnprocessedItems, not errors — that is the only retry it handles. A throttled write is resent; a ValidationException propagates. Going through client.batch_write_item instead hands you the whole loop, as in the Node.js example.
It cannot lift the service limits. 25 writes per request, 400 KB per item, 16 MB per request, no conditions and no updates, and every put replaces the entire stored item. Need a guard, or all-or-nothing? TransactWriteItems.

What batch_writer actually does on flush

Buffer 30 puts and watch the calls it makes. Wrapping table.meta.client.batch_write_item and recording the request sizes, against DynamoDB Local 3.3.0:

batch sizes sent: [25, 5]

Two requests, cut at the service limit, with the remainder flushed by __exit__. That flush is unconditional: raise a RuntimeError inside the block and the buffered items are still written on the way out. A bulk load that dies halfway leaves a partial load behind, not a clean slate.

Now the two failures. Buffer the same key twice, which is what happens the moment your source data has a repeat:

with table.batch_writer() as batch:
    batch.put_item(Item={"Artist": "Dup", "SongTitle": "Key", "Year": 1})
    batch.put_item(Item={"Artist": "Dup", "SongTitle": "Key", "Year": 2})

botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the
BatchWriteItem operation: Provided list of item keys contains duplicates

Neither put_item complained. batch_writer() does not deduplicate unless you ask it to, and asking is table.batch_writer(overwrite_by_pkeys=["Artist", "SongTitle"]). Run the same two puts through that and the item stores as Year: 2 — the buffer keeps the last write per key, so the dedupe is silent data loss if your two rows were meant to be different items under a key you got wrong.

The second one is boto3's alone and never reaches DynamoDB:

TypeError: Float types are not supported. Use Decimal types instead.

A Rating of 4.5 sits in the buffer without complaint and blows up on the flush. Decimal("4.5") round-trips correctly as {"N": "4.5"}. Read a price or a rating out of JSON with json.loads and every number is a float, so this is a first-run failure for most import scripts. Passing parse_float=Decimal to json.loads fixes it at the source.

If you are moving between native Python values and the wire format by hand, the DynamoDB JSON converter shows both sides of the same item so you can see what your Decimal actually becomes.

To bulk-load from CSV or JSON without writing the type mapping yourself, download DynoTable.

DynamoDB BatchWriteItem in Node.js — the manual retry loop batch_writer hides.
DynamoDB BatchWriteItem with the AWS CLI — the same batch write from the shell.
DynamoDB PutItem in Python — the single-item write this batches.
Batch operations in DynamoDB — limits, partial failure, and when batching pays off.
"Too many items requested for the BatchWriteItem call" — more than 25 put/delete requests in one batch.
"Provided list of item keys contains duplicates" — two requests touching the same key in one batch.

References

Last verified 2026-07-28 against the official AWS documentation linked above.

DynamoDB Batch Write in Python (boto3 batch_writer)

Code

Explanation

What batch_writer actually does on flush

Related examples

References