Choosing the Cheapest Model That Passes

Most model providers sell a family of sizes. The smallest is fast and cheap, and each size up costs more per call. The method here is to write down what a passing answer looks like, test the smallest model against it, and move up one size only when a measured check fails.

The bar is a short list of numbers you decide before any model runs. For a question-answering page it might say: answer at least 85% of the questions the sources can answer, refuse at least 90% of the ones they can't, and keep 90% of answers fully supported by their sources. Each line names something a reader would notice if it went wrong.

Writing the bar first matters because results are persuasive. Once you have seen that a model answers almost everything, a faithfulness score a little under your target starts to look acceptable. A bar fixed in advance takes that decision away from the person who just saw the numbers.

Ask the Archive answers questions about Hawaiian words and early recordings from public-domain sources. Its comparison, run in October 2026, tested several models against the bar above. The smallest model in one family answered 39 of 40 answerable questions. A second model checked each answer against its sources and scored its faithfulness at 0.744: about one in four of its answers included a sentence the sources did not support.

Each call costs little, but a reader who catches one invented definition has a reason to doubt the next one. The next size up answered fewer questions (0.875) and kept 0.971 of its answers supported, so it passed every line of the bar.

It did not stay that way. The page then switched to matching words instead of searching by meaning, and on that version the next size up answered 0.800 and missed the bar. A pricier open-weight model passed, though its faithfulness cleared the line by a single judgment. It serves the page now, with the next size up as the fallback when it fails or times out. The Ask page publishes the first comparison and the bar it had to clear.

Freeze the cases and the bar before the first run. The cases should include questions the system ought to refuse, since a model that answers everything looks great until you count what it made up.

Run each candidate fresh, with no cached or replayed answers. Treat a wrong fact as a failure even when everything else is right. The HTML Artifact Studio agent, tested in October 2026, is an example. One case asked it to move a "Most popular" label from one pricing card to another. The smallest model dropped the label, or once put one plan's name on another plan's price, in all four runs that tried it. A wrong fact fails the case however good the styling is, so the agent moved up one size, which passed that case in three runs. The agent replays page records a similar result: on seven held-out tasks, the smaller model failed one that the next size passed, and the agent runs on the next size.

Compare against a plain baseline too. In the Studio's tests, the same model given a plain prompt passed the page's safety and structure checks on 1 of 7 cases. With the full instructions it passed all 11 cases it ran on.

Keep the size of the evidence in view. Eleven cases or 40 questions can show a clear failure. They cannot show that a model handles every request well. The bounded decisions guide works through how many error-free cases a given error limit needs, and the evals page lists each project's measured results alongside their limits.

This table is an authored illustration. Three sizes ran on the same 40 frozen questions. Answered is the share of the 40 that got an answer, and faithful is the share of those answers fully supported by their sources. The bar is at least 0.85 answered and at least 0.90 faithful.

Which one do you pick?

Medium. Small has the best answered rate, which makes it tempting, but it fails faithfulness, and that failure means invented content reaching readers. Large beats Medium on both lines, but the bar doesn't ask for that margin, and you would pay five times as much for it. If the extra quality turns out to matter, the bar was wrong, and you fix the bar.

Then look at Medium's margin. At 0.875 against 0.85, Medium answered 35 of 40 questions. Two more misses would leave 33, or 0.825, and fail the bar. Run it again on fresh cases before trusting the pass, and keep a fallback ready. The Ask example above is what happens when a thin margin meets a change in its inputs.

Sometimes the cheapest option that passes is no model. Ask the Archive finds its sources by matching words. Searching by meaning added 2 of 40 in recall and 0.05 in answers, and would have made each visitor download about 43 MB first. Rules, exact arithmetic, and lookups belong in code, as the bounded decisions guide explains.

A larger model is the right start when a wrong answer is costly, when the task is open-ended, or when you have no cases yet and need a reference to compare smaller models against. OpenAI's model selection guide takes that view: if cost and latency are not a concern, default to the most capable model, then step down to reduce them.

You can also use both sizes. Anthropic's Building effective agents describes routing easy, common questions to a smaller model and hard ones to a more capable one. The FrugalGPT paper describes a cascade that learns which combination of models to use for each query, to cut cost, and RouteLLM trains a router to choose between a stronger and a weaker model. Each still needs the same frozen cases and bar, now for the router as well as the models.