DynamoDB Filtering Strategies
"Filtering" in DynamoDB covers four different mechanisms. Three narrow the data
before it's read and billed. The one actually named Filter narrows it
after, which is why it never saves you money.
How does filtering work in DynamoDB?
DynamoDB has four ways to filter, and only one runs after you're billed. The partition key picks one item collection, the sort key narrows a slice of it, and a sparse index filters by attribute presence: all three cut your read cost before metering. A FilterExpression runs after the read, so it shrinks the response but never the bill.
- Partition key is the cheapest filter: it names one item collection, so you never touch the rest of the table.
- Sort key filters within that collection with
begins_with,between,<,>, still before billing. - Sparse index filters by absence: an item only appears in the index if it has the index's key attributes, so the index is the filtered set.
FilterExpressionruns after DynamoDB meters the read, so it cuts your response size but never your bill.
Set up the example
A support-ticket table. Each team's tickets share a partition, sorted by ticket id:
PK = "TEAM#billing" SK = "TICKET#00194"
Every ticket also carries priority (1 to 4) and status (open or closed).
Escalated tickets get two more attributes, written only while the ticket is
escalated: escalated = "ESCALATED" and escalatedAt (a unix timestamp).
You have four access patterns. Each one maps to a different filtering strategy, and the wrong choice on any of them turns into a read you pay for on every request.
The partition key picks one item collection
"Give me every ticket for the billing team." The partition key answers this directly:
Query PK = "TEAM#billing"
DynamoDB reads exactly one item collection: the items sharing that partition-key
value. Nothing else in the table is touched or billed. That's the difference between
Query and Scan.
Coming from SQL, this feels backwards: there's no WHERE team = 'billing' scanning
an index, you just name the partition. If you can't name it, add a
Global Secondary Index whose partition key is an attribute you can name,
and Query the index.
The sort key narrows a slice of that collection
"Give me billing tickets TICKET#00100 through TICKET#00200." The sort key
narrows inside the item collection, before the read is metered:
Query PK = "TEAM#billing" AND SK between "TICKET#00100" AND "TICKET#00200"
Sort-key conditions are limited on purpose: =, <, <=, >, >=, between,
and begins_with. No OR, no arbitrary predicate. That constraint is what keeps
the read targeted: DynamoDB walks a contiguous slice, not the whole collection.
What decides this is how you encode the sort key. If your pattern is "by
priority", a TICKET#<id> sort key won't help.
Don't bake priority into the base table's sort key either: priority changes, and a base-table key attribute can't be updated (see mutable sort keys).
Put it on a GSI instead: GSI1PK = "TEAM#billing", GSI1SK = "PRIO#1#TICKET#00194".
A GSI key can change with a plain UpdateItem, and the item moves in the index.
That's a sort-key strategy decision, made at
design time, not query time.
A sparse index holds only the items you want
"Give me every escalated ticket, across teams." Few tickets are escalated, so you don't want to read the whole table to find them.
A sparse index solves this by absence. A Global Secondary Index only contains an item if that item has both of the index's key attributes.
Build EscalatedIndex with escalated as its partition key and escalatedAt as its
sort key, and it holds the escalated tickets and nothing else.
AWS spells this out: items missing the index's key attributes are not written to the index (AWS: take advantage of sparse indexes).
Now the query reads only the escalated tickets, billed only for them:
Query ON EscalatedIndex escalated = "ESCALATED" (sorted by escalatedAt)
The filter happened when you wrote the data, by choosing whether to set the index key attributes at all. Sparse indexes goes deeper, and GSI vs LSI covers which index type fits.
One constant partition-key value ("ESCALATED") concentrates every index write on
one key, and a partition serves at most 1,000 write units per second.
A slow trickle of escalations is fine. For a flag that changes thousands of times a
second, shard the key (ESCALATED#0 to ESCALATED#9) and Query each shard: see
hot partitions, and
GSI throttling for why a hot index slows
base-table writes.
A FilterExpression runs after you're billed
"Give me the billing team's open tickets." status isn't a key attribute, so you
reach for a FilterExpression:
Query PK = "TEAM#billing"
Filter status = "open"
DynamoDB reads every item in the TEAM#billing collection, meters the capacity
for all of them, and then drops the closed ones.
AWS states a filter expression is "applied after a Query finishes, but before the
results are returned," and "a Query consumes the same amount of read capacity,
regardless of whether a filter expression is present."
You already paid for the full read (AWS: filter expressions for Query).
So if the billing team has 10,000 tickets and 12 are open, you pay to read 10,000.
The response is small; the bill is not. FilterExpression shrinks the payload
crossing the wire, never the read.
Pagination is metered before filtering too: a page is 1 MB of read items, not
1 MB of matches, so a filtered page can come back empty with a LastEvaluatedKey.
Keep paging until the key is absent (pagination).
If "open tickets" is a common pattern, it belongs in a key: a sparse index on an attribute written only while the ticket is open. A filter is for the last little bit of narrowing.
Build the expression with the
DynamoDB Expression Builder. status is a
DynamoDB reserved word, so it needs a #status placeholder, and the builder writes
the names and values for you. It's preset below to the Query + filter above:
dynamodb-expression-builder is the open-source (MIT) library behind this tool.
Compare the four
| When it filters | Cuts read cost? | Predicate power | Cost to set up | |
|---|---|---|---|---|
| Partition key | Before read | Yes, one item collection | Equality only | Free (it's the key) |
| Sort key | Before read | Yes, a slice | Range / begins_with | Sort-key design |
| Sparse index | Before read | Yes, index items only | Presence of an attribute | Extra GSI + write cost |
| FilterExpression | After read | No | Almost any condition | None |
Read the table top to bottom: predicate power goes up, cost control goes down.
FilterExpression can express almost any condition on non-key attributes precisely
because it runs on already-read items, and that's the same reason it can't save you
money.
See it in DynoTable
When you run a Query with a filter, the gap between items read and items
returned tells you what the filter costs.
DynoTable shows items scanned next to items returned as a filtered read streams, so a filter reading the whole collection shows up on screen, not in your monthly bill.
For cross-item questions a filter can't answer ("open tickets per team", "tickets
joined to their account"), DynoTable's SQL Workbench runs GROUP BY, JOIN and
aggregates on your machine.
Each JOIN lookup is a key read. The base read is a Query when you give the base
table a partition-key condition and a Scan when you don't, and the status bar tells
you which before you run.
Pitfalls and next steps
- Don't use
FilterExpressionas your primary access path. If a pattern is common, model it into a key or a sparse index. - Watch empty pages. A filtered query can page for a long time returning
nothing. Honor
LastEvaluatedKey; an empty page doesn't mean "done". - A sparse index isn't free. Every item that lands in it costs index writes and
storage. Keep it small by projecting
KEYS_ONLY, orINCLUDEjust the attributes your list view needs, and fetch the rest by key.
Estimate what a filtered read will actually cost with the pricing calculator, and try DynoTable to watch items scanned against items returned on your own tables.