How we made cheap LLMs reliable at DynamoDB: tool validators + evals

DynoTable's AI assistant runs on your own AWS Bedrock credentials, which means you pick the model — and many people pick a cheap one. So we don't tune the assistant against a frontier model. Our floor is Amazon Nova Lite, a model that costs a rounding error per query and fails in ways frontier models don't.

Here's the punchline of this whole post: on one Eval scenario — a join that needs the composite-key table as the base — Nova Lite scored 0%. We didn't switch models, didn't add few-shot examples, didn't fine-tune. We rewrote one validator error message to say exactly what to do instead. It went to 100%, flipping the join on the very next step.

The model was never the bottleneck. The message was.

This is the engineering story behind that lesson: the validators that sit around every tool call, the things we deliberately do not validate, and the eval harness that arbitrates every change. If you're building an agent on a budget model — over DynamoDB or anything else — most of it transfers.

Why cheap models fail at DynamoDB specifically

DynamoDB is a hostile target for a small model because the query surface looks like SQL and isn't. PartiQL accepts SELECT syntax but supports no JOIN, no GROUP BY, no DISTINCT, no LIKE — and a model that learned SQL from the internet emits all of them confidently.

The failures we actually observed, each from a real logged run:

• Dropped limit arguments. Cheap models routinely omit the optional row limit on a query tool. The backend happily returns everything, a 200 KB tool result lands in the conversation, and the next model turn dies on the context ceiling — a ValidationException mid-loop, blamed on the wrong component.
• Unsupported constructs. JOIN, GROUP BY, aggregates in PartiQL — statements DynamoDB will never run, emitted because they'd be correct on Postgres.
• Identifier swaps. Nova Lite, given SELECT * FROM customers and the instruction "add a WHERE for Germany", would sometimes return SELECT * FROM orders WHERE … — a table the user never mentioned. The statement parses perfectly. Nothing syntactic catches it.
• Tool misrouting. Asked about an attached file, Nova Lite called the open-tabs listing tool looking for files in tabs — it rarely used the tool-search mechanism at all, so anything not directly visible didn't exist.

None of these are hypothetical. Each one drove a specific guardrail below.

Validate at the tool boundary, and make the error a lesson

The single most important architectural decision: validation lives inside the tool's execute step, not in the input schema. A schema rejection is a dead end — the loop sees a type error and learns nothing. A tool that runs, checks, and returns a structured error gives the model a normal tool result it can react to on the next step.

And that error is written for the model, not for a human log. Our export tool's errors read like this, verbatim:

startExport requires exactly one of {tabId} or {sql}.
FIX: call listOpenTabs and pass {tabId}, or pass {sql} with a single SELECT.

startExport {sql} must be a read-only single SELECT.
FIX: remove any INSERT/UPDATE/DELETE/DDL and pass one SELECT statement.

Query validation failures carry a validation: prefix — validation: parse-error: …, validation: partiql-unsupported: JOIN is not supported by DynamoDB PartiQL. — which the system prompt establishes as the signal to fix the statement rather than retry the same input. A rejection message is a teaching signal. Before we adopted the FIX: convention, the same failures ended with the model apologising and telling the user to click the export button themselves. After, it self-corrects and completes the task.

The corollary took longer to learn: where the concrete example lives is a design axis. If a mistake is caught by a validator at runtime, keep the system prompt generic and let the dynamic error message carry the specific tables and fix — the prompt stays small and the lesson arrives exactly when it's needed. Only mistakes no validator can catch (like routing an aggregate question to the right tool in the first place) earn a concrete example in the prompt itself.

Parse the things you're going to reject

Rejecting JOIN with a generic syntax error at offset 27 teaches nothing. So our hand-written PartiQL parser does something slightly perverse: it deliberately parses constructs DynamoDB doesn't support into a valid syntax tree — just so a walker can emit a specific, teachable diagnostic. All verbatim, all model-facing:

• JOIN is not supported by DynamoDB PartiQL.
• GROUP BY is not supported by DynamoDB PartiQL.
• TOP N is not supported. Use the API limit parameter.
• IN list has 120 items. DynamoDB PartiQL caps IN at 50 values (PK column) or 100 values (non-key column).
• IS NOT NULL is not supported in DynamoDB PartiQL. Use attribute_exists.
• lower(path) is not a DynamoDB PartiQL function. Use … instead. — with the replacement resolved per function.

Each message names the construct and the escape hatch. Paired with the prompt rule "if PartiQL rejects an unsupported construct, switch to the SQL workbench immediately", a cheap model recovers from its Postgres instincts in one step instead of thrashing.

Know what NOT to validate

The schema-aware tier of our SQL validator resolves table and column references against the real table descriptions the app already holds. Early on it also flagged references to attributes DynamoDB didn't declare — which sounds right and is exactly wrong. DynamoDB's attributeDefinitions only lists key attributes, so the check rejected WHERE <non-key> = … — the single most common legitimate query shape in existence. That check is now a warning that never gates.

Every validator is a bet that the rejected shape is more likely wrong than right. When the data model can't support the check, don't make the bet.

Result caps as behaviour, not just safety

The fix for the dropped-limit blow-up wasn't "add the limit to the prompt" (cheap models drop it anyway). Query tool results are capped hard — 10 rows or 5 KB, whichever comes first, always keeping at least the first row — and the truncation notice is itself prescriptive:

Showing 10 of 4,200. To filter: re-run with WHERE.
To view in UI: emit proposeOpenTable.
To aggregate: use runWorkbenchSql GROUP BY.

The caps are intentionally far below what the context could hold. The goal is behavioural: force the model to hand the user a real table view or an exact aggregate, never to enumerate rows into chat. The recovery hint names the exact tools, so the cap teaches the routing at the moment the model needs it. The same envelope is mirrored byte-for-byte in the eval harness, and a regression scenario pins truncated: true against a 10,000-item table so the cap can never silently disappear.

Evals are the arbiter

None of the above was designed on intuition and kept on faith. Every prompt clause, validator message, and cap is gated by an eval suite that runs real agent loops against a seeded DynamoDB Local — real tools, real query execution, the production prompt builders and validators imported directly rather than reimplemented — scored by binary checks on the tool trace, the final state of the database, and the text.

What the evals caught that intuition missed:

• A "cleaner" prompt genericisation regressed a scenario from 100% to 0%. It read better to us. The floor model disagreed. Only the eval noticed.
• Frontier-model prompt advice is actively wrong for cheap models. Published guidance says to soften CRITICAL/MUST language because big models over-trigger on it. Cheap models under-comply and need the firm version. We now eval-gate any borrowed advice instead of applying it on faith.
• Strict tool assertions punished correct behaviour. A scorer requiring the model to call the ask tool for clarification passed 19% across the model ladder — almost every "failure" was a model that had correctly asked for clarification in text. The scorer now accepts either. Score the user-facing intent, not the mechanism.
• The model ladder earned its narrowing. We started broad — Mistral, AI21, GLM, Qwen, Nova Pro, more — and cut it to two rungs (Nova Lite, Claude Haiku) for reasons only real runs surface: one family cost ~10× per call because Bedrock offered it no prompt caching; Nova Pro hallucinated concatenated attribute names like customers.customerName on open-ended joins. And one family had to be blocklisted outright — Bedrock's unified API happily accepts tool definitions for models whose adapter then emits function calls as plain text, something the API metadata won't tell you. We found it with a paid smoke probe, not a spec.

Two disciplines keep the suite honest. Example tables never equal test tables: prompt examples use neutral names that don't exist in the seed data, and the evals run on different tables — so a pass proves the model generalised the pattern, not that it memorised our example. And the $0 watch mode refuses recorded fixtures: replayed model outputs would green-pass assertions that depend on real tool activity, so the free tier validates wiring only, and anything that claims a score had to spend real tokens on a real model.

The checklist, if you're building one of these

• Pick your floor model first and eval against it always; frontier models hide your bugs.
• Put validation in the tool's execute step; return errors the loop can act on, never schema dead-ends.
• Write every rejection as what's wrong + FIX: exact next action, and interpolate the user's actual tables into it.
• Parse what you intend to reject, so the diagnostic can be specific.
• Audit each validator against your data model — delete the ones that reject legitimate shapes.
• Cap tool results below what fits, and make the truncation notice route the model to the right next tool.
• Run evals as real loops against a real (local) backend, reusing production code paths; keep example data disjoint from test data; make the eval — not taste — the arbiter of every prompt and message change.

Does any of this matter if you use a frontier model?

Less often, but yes. The validators are load-bearing at every tier — a frontier model still can't run JOIN on DynamoDB, and prescriptive errors just get consumed faster. On our own runs the chip-emission behaviour of a frontier-class model once dropped to 0% under a prompt phrasing that smaller models handled — the eval harness caught that too. Reliability engineering isn't a small-model tax; small models just make the invoice arrive sooner.

Where this runs

All of this ships inside DynoTable's assistant and its gated tool catalog: schema-aware natural-language querying on your own Bedrock credentials, exact whole-table aggregates, and writes that only ever land in a reviewable staging area. External agents get the same validated toolkit through the MCP server — how we built that safely is its own story, from OAuth to credential isolation.

And when you want determinism instead of generation — the query you'll commit to code — skip the model entirely: the Expression Builder assembles the exact request by hand.