GPT-5.6 Luna vs the cheap models we ship: benchmarked on real agent work

On July 30, OpenAI cut GPT-5.6 Luna's price by 80%, and the same cut landed on Amazon Bedrock. Every comparison page on the internet updated a pricing table that day. None of them answered the question that matters: at that price, is Luna actually good enough to drive an agent?

We had a way to answer it. DynoTable's AI agent runs real agent loops against DynamoDB — tool calls, PartiQL queries, SQL routing, staged writes — and every model we ship is gated by an Eval suite that scores those loops end to end. So we pointed the suite at Luna and let it run: 201 test cases, three runs each, against a live database, with every call priced from what AWS actually charged us.

The verdict: Luna scored 93.94% against 96.00% for Amazon Nova 2 Lite — the cheap model we ship as our tuning floor — on the same 201 cells, same scorers, same day. Two points behind a model our prompts have been tuned against for months. And it cost 5.3× less per call — the cheapest model we have measured.

Cheap and does the job. Here is the data, and the two mechanisms behind it — one of which is a billing behavior of Bedrock Mantle that none of the pricing tables capture.

What this benchmark is (and is not)

These are not public benchmark scores. Our suite drives the production agent through real DynamoDB sessions: pick the right tool for a plain question, read an item by composite key, stage a write for human review, respect PartiQL's no-JOIN boundary, route a GROUP BY to the SQL surface instead, ask a clarifying question only when the request is genuinely ambiguous, refuse a "delete everything". A scorer asserts the outcome of each session — we learned the hard way what happens when a scorer asserts tool names instead.

Three honesty notes before any chart:

• "Verified" in our model picker is a coverage claim, not a quality bar. It means the eval suite runs against that model. Luna is verified and scored below our floor model — both facts are published here.
• Nova 2 Lite and Claude Haiku 4.5 are the two models this app ships against, so months of prompt tuning have gone toward them specifically. Luna got zero tuning. Read the 2-point gap with that in mind.
• Three runs per case absorb flukes, not variance. One run going the other way moves a single case by 33 points, so no claim below rests on one.

The headline pair: Luna vs Nova 2 Lite

Measured on the 201 test cases both models ran, after we fixed a bias in how we were grading — the same yardstick, held to both:

The gap is real and we are not smoothing it: Luna loses on a handful of behaviors — it refused a task it should have attempted, routed an indexed top-N query through a full-table pass, and once deferred to the user instead of finishing an export. Those are agent-behavior losses, not formatting noise, and they are why Nova 2 Lite remains our floor.

But the shape of the result is the story. An untuned model, on someone else's benchmark, tuned for someone else's models, landing within ~2 points of the incumbent — at a fifth of the cost.

What every call actually cost

Sticker prices say Luna should be cheap. What we were actually charged says it is cheaper still — here is every call we made, priced at the rates AWS billed:

These are current Bedrock prices — we ran everything after the July 30 cut. Terra appears for cost only: we ran it across far fewer test cases than the others, and a score from a smaller sample does not belong beside a full one.

On sticker prices alone, Luna ($0.22 in / $1.32 out per million tokens on Bedrock) is already cheaper than Nova 2 Lite ($0.33 / $2.75) on both axes. But the measured gap per call is wider than the stickers explain. The difference is caching.

The implicit prompt caching nobody configures

An agent loop re-sends a large, mostly-static prompt on every step: system instructions, tool schemas, conversation so far. Whether you pay full price for those repeated tokens is the single biggest cost lever in agent workloads.

On Bedrock's Converse API, prompt caching is opt-in: you place cache breakpoints in the request, and models without cache support simply pay full rate. GPT-5.6 on Bedrock Mantle — the OpenAI-compatible endpoint these models run on — behaves differently: caching is implicit and on by default. No breakpoints, no request changes, nothing to configure.

We measured what that default is worth. Share of input charged at the discounted cache rate (90% off):

Across 4,728 calls, Luna paid full price on 9,456 input tokens — two per call. The static prompt prefix was billed at the read rate essentially every single step, without a cache breakpoint anywhere in our requests. Nova 2 Lite, on explicit Converse caching with a 0.25× read rate (vs 0.1× for Luna and Haiku), left 28% of its input at full price — and that, more than the stickers, is why the per-call gap is 5.3× and not 1.5×.

AWS shipped explicit cache breakpoints for GPT-5.6 the same day as the price cut. Our numbers are the default behavior — what you get before reading that post. For an agent loop with a stable prefix, the default already captured 94.7%.

Mantle is a different transport — budget a day for these

GPT-5.6 is the first Bedrock family that the Converse API cannot reach at all. It lives on the bedrock-mantle.{region} endpoint, speaking the OpenAI Responses protocol. Three things cost us real debugging time:

• temperature is rejected, not ignored. OpenAI's own endpoint silently drops unsupported sampling parameters; Mantle returns 400 unsupported_parameter. Our first full run failed everywhere on this before a single token was generated.
• The base URL is /openai/v1, not /v1. The wrong path also returns a 400 — one that reads misleadingly like an entitlement problem.
• A fresh account gets 401s until the AWS Marketplace subscription for the model is active. The error text talks about authorization, not subscriptions; if you hit it, check Marketplace first.

None of this is exotic once you know it, but no pricing table mentions any of it.

The full ladder: 12 models, one instrument, one day

After the price cut we re-ran every model we track in one session — 12 models, 39 scenarios, 3 runs each, $20.49 of Bedrock spend — so every row below is measured the same way on the same day. The table uses the 36 scenarios all twelve models ran — the three autocomplete ones are left out, because the GPT-5.6 models cannot do autocomplete here at all:

(1) The two models this app ships against — months of prompt tuning went toward them specifically, and Haiku's perfect score should be read with that squarely in view. (2) No Bedrock prompt caching, so every step of an agent loop pays full input rate — which is how DeepSeek ends up costing 5× Claude Haiku per run while scoring seven points below it.

An agent run here is a full multi-step session — tool calls, queries, staged writes — and the dollar column is what one such session costs, averaged over the 100+ we ran per model.

Four readings we'd defend from this table, and one we wouldn't. Luna is the only model in the top half whose cost sits in the bottom half — the price-performance outlier of the ladder. Terra, the mid-tier GPT-5.6 at nearly ten times Luna's measured cost per run, lands 0.78 points below its cheaper sibling — that gap is inside the noise band, so the honest reading is a tie, and a tie is the damning version: on this workload, paying 10× for the bigger variant bought nothing we could detect. The caching column of the economy matters more than the sticker: Nova Lite Gen 1 at $0.0013/run is the budget surprise, and the no-cache tier is uniformly mispriced for agent work. And the two models we tuned for hold ranks 1 and 3, which is the tuning bias made visible, not a neutral ranking. What we would not defend: reading any two adjacent rows as meaningfully different — with three runs each, neighbours can swap on a single one going the other way.

One more thing this run measured: our own harness, again. The three autocomplete scenarios initially scored 0% for all four no-cache models — 36 runs of zero — and every one was rejected before the model generated a token: we were sending a request field those models don't accept. Same bug class we published about before, new field. Those 36 zeros measured our bug, not the models, so they are excluded from every number above — which raised those four scores by about 4 points each.

Then we fixed the builder and re-ran exactly those cells. DeepSeek V3.2 — which the broken harness had at 0% — scored 100% on all three autocomplete scenarios. The other three landed mid-range. Had we published the first run uncorrected, we'd have reported a model as incapable of autocomplete when it is, measurably, the best of the four at it. A benchmark that won't audit itself will happily rank vendors on its own bugs.

What transfers if you're picking a cheap agent model

• Benchmark on your workload, not on leaderboards. Luna's public scores and our agent scores rank models differently. The only number that predicts your bill and your failure rate is one measured on your own loops.
• Price the loop, not the token. Cache behavior differs per model family — read rates, write premiums, and whether caching is implicit or needs breakpoints. Two models with similar stickers can differ 5× per call.
• A model being 2 points behind may not matter — where it loses does. Read the failing behaviors, not the aggregate. A wrong refusal is recoverable; a silent wrong answer is not.
• Record what every call costs, from day one. Every number here exists because we kept that record. Estimating from published prices would have missed the caching story entirely.

Luna is now a verified, selectable model in DynoTable — bring your own Bedrock credentials and pick it in the model picker. Our default remains Claude Haiku 4.5, and our tuning floor remains Nova 2 Lite; the gap is real. But if your bill is the constraint, two points is a very cheap trade.

Download DynoTable · How the AI agent works · The eval story behind our scorers