Why we hand-wrote a PartiQL parser for DynamoDB

DynamoDB accepts a narrow slice of PartiQL. GROUP BY, a statement-level LIMIT, multiplication with *, CAST and subqueries all look like valid SQL. Each one travels to AWS and comes back as a ValidationException.

DynoTable's query editor is built on CodeMirror. Its linter, the code that underlines mistakes as you type, caught most of these by keyword. It couldn't see structure, so we hand-wrote a parser for DynamoDB's PartiQL dialect.

A precise underline and a safe quick fix both need that structure. In August we open-sourced the parser as dynamodb-partiql-parser: TypeScript, zero dependencies, MIT.

The CodeMirror integration ships separately as codemirror-lang-partiql.

A regex linter can't tell multiplication from SELECT *

The first PartiQL linter in DynoTable was about 650 lines of regular expressions and token scanning. It caught real mistakes: GROUP BY, DISTINCT, CASE, aggregates in WHERE, and string functions DynamoDB doesn't have.

It also caught the most common slip. DynamoDB writes an IN list with square brackets, as in WHERE OrderID IN [100, 300, 234], and the linter flagged the SQL habit of parentheses.

Its quick fixes were few, such as turning a double-quoted string value into a single-quoted one.

Its blind spot was structure. In SELECT price * quantity, the * is multiplication, which DynamoDB rejects. In SELECT * it means every attribute. Telling the two apart takes a parser.

The ranges were approximate too. The rule for aggregates in WHERE found its match, then searched the text again to locate it. That was close enough to point at a line, but too coarse for a fix that replaces text at exact positions.

Each new rule had to work around the others. The alias rule skipped matches inside CAST(...) so one span wasn't flagged twice.

The rule for aggregates in the SELECT list skipped any list that held a subquery, because its regex would grab the wrong FROM.

No parser on npm spoke DynamoDB's dialect

We had met this problem once before. DynoTable's SQL Workbench runs real SQL over your tables, and its first parser setup returned wrong answers. We replaced it with a concrete-syntax-tree parser.

That replacement taught us which tree to build. A concrete syntax tree (CST) keeps every token and its position. An abstract syntax tree (AST) keeps only the meaning. A quick fix edits the source, so it needs the CST.

A SQL grammar doesn't fit PartiQL, though. DynamoDB's dialect has bracketed IN lists, bag literals (<<'a', 'b'>>), map literals with quoted keys ({'rating': 5}) and document paths with list indexes (Devices.FireStick.DateWatched[0]).

It also has a MISSING literal for an attribute the item doesn't have, and RETURNING ALL OLD * on writes. A SQL grammar knows none of these, and it accepts much that DynamoDB rejects.

The PartiQL parsers we found on npm were generic. @aws/lsp-partiql bundles a WebAssembly build of the PartiQL team's Rust parser. partiql-js and piqel publish an independent Rust implementation.

None of them modeled what DynamoDB rejects, and no parser for DynamoDB's dialect was on npm. So we wrote a small lexer and a recursive-descent parser, shaped like the Workbench's new parser.

Every node in its tree carries source offsets, the positions where the node starts and ends in the query text.

The package has zero runtime dependencies, and its CI fails the build if one appears. That keeps it easy to embed, in a browser page or in your own project.

A linter's parser spends its life on broken input

The grammar was the easy half. A linter mostly parses unfinished code: a statement mid-keystroke, a missing bracket, a typo in the third clause. A parser that stopped at the first error would lint nothing after it.

So the parser recovers. A broken expression becomes an error node plus a diagnostic, and parsing continues. A broken statement makes the parser skip to the next ; or statement keyword, so the statements around it still lint.

Exact offsets let a diagnostic carry an edit. IN (1, 2) gets a fix that swaps the parentheses for brackets. Title LIKE '%foo%' gets a rewrite to contains(Title, 'foo'), and Title LIKE 'foo%' one to begins_with(Title, 'foo').

Every spec fixture cites the AWS page it encodes

The dialect rules come from AWS's PartiQL reference. Behavior derived from documentation can drift. The doc changes, the code doesn't, and nobody notices.

The spec corpus makes each rule traceable to its source. It holds 103 fixtures, each a query file with its expected diagnostics, and every one opens with the AWS documentation URL its rule was taken from.

A coverage table maps each documented rule to at least one fixture. The suite fails if a rule has no fixture or a fixture is missing from the table.

The corpus does not detect a doc change by itself. Someone still has to notice it. The cited URLs then show which fixtures to re-check, and the update lands as a fixture diff with a citation.

Some documented rules need context the linter doesn't have. The AWS operators page caps an IN list at 50 values on the partition key and 100 on a non-key attribute.

Without your table's schema, the linter can't tell which cap applies to a column. It warns once a list passes 100 values, and the warning names both caps.

Pasted garbage could overflow or freeze the parser

Among the bugs code review found before launch, two had never been hit by a real query, and stress tests now pin them. Deep nesting overflowed the call stack. Pasting 100 KB of garbage froze the lexer for 15.8 seconds on our machine.

CodeMirror runs its linter after a short pause in typing. The editor also checks for errors synchronously on every change, to decide whether to enable the button that runs the query.

An exception there escaped the parser and left the editor's error state wrong.

A recursive-descent parser can throw on its own, through the call stack. Each nesting level is a stack frame. Thousands of nested [ brackets or a long NOT NOT NOT chain end in V8's RangeError: Maximum call stack size exceeded.

Expression recursion now stops at 500 levels, far past anything a person writes and well under the stack budget. Past that, the parser emits one diagnostic instead of throwing. A NOT chain is now parsed in a loop.

Pasted input chains far deeper, such as A UNION B UNION C thousands of arms long. Those chains are now flat lists. The parser keeps one frame and an array of set operations instead of one frame per arm.

The lexer's cost on garbage grew with the square of the input. It now caps the errors it emits, and on the same machine the 100 KB paste lexes in 22 milliseconds.

The stress tests run on every build. They feed the parser 100 KB of garbage, 30,000-deep operator chains and a 5,000-arm UNION.

The public lint() entry point also catches any exception and returns a single diagnostic. The next editor to embed the parser may have nothing above its linter to catch errors.

The frozen regex linter refereed the swap

By the time the parser was ready, other code depended on the regex linter. One of its functions decided whether a statement was safe to run automatically. A silent change there would make the editor refuse queries it used to run.

So we froze the old linter in place and gave the new one the same four functions. A parity corpus of 53 fixtures runs every query through both linters and compares the results.

The comparison matches only each diagnostic's severity and the first word of its message, because the two linters word their messages differently.

A difference fails the test unless the fixture records a reason for it. 15 of the 53 do: 14 are marked as cases where the parser is stricter, and one as an intentional change.

The old linter is still in the app, frozen. The parity corpus now checks every package upgrade the app takes against it.

What transfers if you're building one

• Hand-write the parser for a small dialect. Our lexer and recursive-descent parser took less than a week, and we own every error message.
• Build a CST. Source offsets on every node turn a diagnostic into a quick fix. A tree that drops the source can't edit it.
• Recover from errors. A parser that stops at the first one lints nothing after it.
• Swap engines behind a frozen interface. Pin the old implementation against the new with a parity corpus, and make every allowed difference carry a written reason.
• Cap recursion depth and flatten chains. Anywhere input can nest, test with garbage as well as queries.
• Cite your sources in the tests. A fixture that names the doc page it encodes can be checked again when that page changes.

The parser is on GitHub and on npm (npm install dynamodb-partiql-parser). The editor integration is codemirror-lang-partiql.

For the dialect itself, PartiQL vs SQL covers what DynamoDB's subset can and can't do, and PartiQL examples walks through real statements.

The editor all of this was built for is in DynoTable, and you can try it free.