StewAI Blog · Research · 13 min read
We Paid 28x More for the Same Answer (And the Only Failure Was the Reasoning Model)
Fifteen runs of a source-grounded legal citation check across four models and three providers. Three models spanning a 30x price range produced identical output on every unambiguous item. The one model that failed was a reasoning model costing six times the cheapest. And the deterministic gate was right 15 times out of 15 regardless.
Frequently asked questions
Do cheap models match expensive reasoning models behind strong guardrails?
On this task, yes for the unambiguous items. Three models spanning a 30x price range, across two providers, scored 100% on every unambiguous item of a source-grounded citation check. The cheapest arm cost 38 credits per run and the most expensive 1,073, a 28x difference for indistinguishable output. The finding is not that cheap models are always fine: the single arm that failed was a mid-priced reasoning model, so tier does not predict quality and you have to measure per model.
What does a deterministic gate do when the model gets an item wrong?
It absorbs the error. In this pilot the gate reached the correct routing decision in 15 of 15 runs on every arm, including the arm that misclassified 30% of items, because the blocking decision aggregates over several item verdicts and a single misclassification cannot flip it. The business-level output survived model errors that would have quietly corrupted a free-text summary.
Does anything actually improve with a more expensive model?
Consistency did. The frontier model produced one byte-identical verdict set across five runs, while the two cheap models each produced two distinct sets across four runs. The variation was always the single genuinely contested item, and both cheap models were perfectly stable on the five clear ones. If you need bit-reproducible output the expensive model buys that; on this task it did not buy accuracy.
Read the full article on StewAI