Should a Model Make This Decision?

Should this request go to a person, or can software handle it? When the answer comes from a known set, you have a bounded decision. Picking a route and checking a reply can work this way too. The Jev workshop lets you compare ways to make these decisions and the evidence each needs.

Write down the allowed outcomes first. If you cannot list them, clarify the task before choosing a model. If the task needs an open-ended answer, use a generator. If the outcomes are fixed, measure the task as classification.

Define the candidates, the rule for choosing one, and what happens when none fits. Keep a versioned record of that policy and its decisions. Scoring the candidates cannot reveal a right answer that was left off the list.

Start with the simplest option that meets your error limit. Include the cost of running it, labeling examples, and maintaining it:

1. Code. If a rule decides it from structured fields, write the rule and test it. Permissions, spending limits, retries, exact arithmetic, and hard safety rules stay in code in every design below.
2. A classifier. If the label set is stable and you have labeled history, a classifier is the baseline to beat: a smaller model trained only to pick one label from your list. Count its training and retraining cost.
3. A structured-output LLM. If the policy is written in prose and changes often, or the decision needs generated arguments, a general model with a strict schema is often simplest. If it writes a confidence value in its output, that value is generated text rather than a measured probability.
4. A decision model like Jev. If code supplies the candidates, try a model that returns a score for each. TypeSafe's confidence docs describe thresholds over the returned distribution. The workshop uses its top probability. Test any threshold on your own task; a high model score alone does not establish accuracy.
5. A cascade. A cheap first stage accepts confident cases and a stronger model or a person handles the rest. Cases both stages get wrong cap the benefit. Choosing the cheapest model that passes covers how to pick a model's size.
6. A person. If your evidence cannot bound the error below your limit, keep a person deciding and collect labels.

The workshop's routing exercise runs keyword rules and a scored policy on the same requests. The rules handle a duplicate charge and can miss the same problem in other words. Hard rules in code run before any score, and an instruction written inside a customer's message is customer text, not policy.

Pick the error rate you can accept on decisions made without a person, for example 5%. For a fixed policy evaluated on independent, representative accepted cases, 0 errors in 59 cases puts the exact one-sided 95% upper bound below 5%. A 10% limit needs 29 error-free cases. These counts apply to decisions the policy accepts, not all requests. They do not account for selecting a threshold on the same cases or for related cases that share a source.

In the workshop, set an error limit. Once there are labeled judgments, the workshop uses them to select a gate, a rule for accepting decisions. It reports one of 3 outcomes: a selected gate with its coverage and remaining errors, no gate that meets the limit, or insufficient evidence.

When a case pack includes a holdout, a separate set for evaluation, selection uses only the development split, the cases used to choose the gate. Without labeled judgments, selection is unavailable.

Once you use cases to choose a threshold, those cases are part of the choice. Freeze the policy and its identity, then evaluate it once on cases nobody has tuned against. Report the result whatever it shows. A frozen policy does not make public or already-inspected cases blind.

The workshop's support-review protocol freezes a separate holdout outside the site's public files. It requires 2 human reviewers who did not write the cases to label them before seeing model output. That review and model collection are pending. Some holdout cases share a source, so the raw case count is not a count of independent observations.

The recorded pilot has 4 authored cases. Both judges matched all 4 reference labels, and every Jev judgment put probability 1 on its answer. The threshold therefore could not trade coverage for error. This tiny development set provides an integration check. It does not validate a policy. The larger experiment has not been run.

Record the allowed outcomes, hard rules, error limit, and fallback before testing. Keep the cases used to choose a threshold separate from those used to evaluate it. Report accepted decisions, accepted errors, coverage, and unavailable results with their denominators. When the evidence cannot support the limit, keep the fallback and collect more labels.

A team routes support messages to billing, repairs, or review. Their new policy sent 20 test messages, chosen at random from last month, and all 20 went to the right place. They want to stop reviewing routed messages if errors stay under 5%. Can they?

Answer: not yet. With zero errors in 20 independent cases, the true error rate could still be as high as about 13.9%. Reaching the 5% limit needs 59 error-free cases, and those cases should not be the ones used to tune the policy. Twenty perfect results are a good reason to keep testing, with people still reviewing.

Writing versus choosing separates the kinds of job a model can do, and reading model claims applies the same questions to someone else's published numbers. For the wider system around a decision, read designing the system around a model.