Jev confidence vs probability: when not to trust it

Jev confidence is not a second opinion from the model. It is a statistic computed from the probability distribution the answer already gave you, collapsed into one number between 0 and 1 so your code can threshold on it. Probability tells you what the model thinks. Confidence tells you whether to act on it. Confusing the two is how a routing layer ends up automating its least reliable decisions.

TypeSafe is unusually candid about this. Its confidence documentation calls the value "a solid default" that "fits most use-cases", says you are "never locked into our definition", and points out that the full probabilities array is returned precisely so you can compute a different measure. That sentence is the licence to do the thing this post argues for: read the distribution, not just the summary.

Where the jev confidence number comes from

Every Choice and Score answer carries probabilities — the distribution across your options or your levels — and a confidence derived from it. A concentrated distribution means a confident answer; a flat one means an uncertain answer. Low confidence on a Choice usually means no option is a clear winner. Low confidence on a Score usually means the levels are ambiguous, multi-dimensional, or the state simply does not contain enough to go on.

The docs publish the arithmetic for their interactive explainer: for a three-option Choice, confidence is approximated as (3 × largest probability − 1) / 2, and the explorer's code generalises it to (n × peak − 1) / (n − 1) for n options. TypeSafe describes this as the demo's approximation rather than a specification of the production statistic, so treat the numbers below as the shape of the thing, not as a contract. The shape is what matters.

Probabilities across three options	Chosen option	Confidence
90% / 6% / 4%	A	0.85
55% / 44% / 1%	A	0.325
55% / 22.5% / 22.5%	A	0.325
40% / 33% / 27%	A	0.10
33.3% / 33.3% / 33.3%	tie	0.00

Look at rows two and three. Both return option A. Both return the same confidence. In one the model is nearly split between A and B; in the other the runner-up probability is spread evenly across two options it does not much like. If your code sends A and B to different teams, those are not the same situation, and the confidence number cannot tell them apart. The margin between the top two probabilities can.

That is the practical rule. Gate on confidence when the question is "should a machine handle this at all". Gate on the top-two margin when the question is "which of these two specific branches". Both numbers come back in the same response, so using both costs nothing.

Probability and confidence answer different questions

A Noul makes the distinction impossible to ignore, because a Noul has no confidence value at all. It returns a single number from 0 to 1: the probability that the statement is true. Near 1 is a strong yes, near 0 a strong no, and near 0.5 is the model saying it has no view.

That changes the code you write. A Choice or a Score takes a one-sided threshold — act above it, escalate below it. A Noul needs a two-sided band around the middle, because both 0.02 and 0.98 are confident answers and 0.5 is the uncertain one. Teams reach for if noul > 0.5, which quietly treats 0.51 as a yes. What you want is closer to an explicit uncertainty band whose edges you set from your own data:

refund = answers["refund_requested"].noul

if refund >= 0.85:       # strong yes — start the refund workflow
    begin_refund(ticket)
elif refund <= 0.15:      # strong no — carry on with normal triage
    continue_triage(ticket)
else:                     # the model has no view; a person decides
    queue_for_agent(ticket)

The 0.85 and 0.15 in that snippet are placeholders, not recommendations. They are the two numbers you are supposed to derive from labelled examples, and the rest of this post is about how.

Thresholds scale with risk, not with the model

TypeSafe's own guidance is that a confidence threshold is not one number: different actions in the same system should be gated at different levels depending on the consequences of getting it wrong. The confidence-gated routing pattern works a voice banking example in which any answer below 0.6 goes to a support agent, check_balance proceeds at 0.6 because the worst case is a balance read out to its owner, and approve_transfer only executes above 0.85 — otherwise the system asks the user to confirm. The intent routing pattern uses a 0.5 floor for the same purpose.

Those figures are examples in TypeSafe's documentation, and the docs say plainly that the correct values depend on your domain and on how the model performs for your use case: start conservative, test on your own data, adjust. Copying 0.6 into your own router because it appears in a vendor example is the single most common mistake we see when we review this kind of AI engineering work.

There is a structural reason it cannot transfer. On the formula the docs publish, the confidence a given peak probability produces depends on how many options the question has. The same 0.6 gate demands a very different level of agreement from a three-option question than from a wide one, and Jev supports a cardinality of up to 255 options, a limit TypeSafe states in its launch post.

Options in the Choice	Top probability needed to reach confidence 0.6
3	73.3%
10	64.0%
20	62.0%
255	60.2%

So a threshold is a property of one question, not of your system. Add a fifth department to a four-option Choice and the same gate has quietly become more permissive. Re-measure after any change to the option set, the criteria wording or the shape of the state.

How to choose a jev confidence threshold from your own labelled data

The method is ordinary classifier evaluation, and it takes an afternoon. What makes it worth doing is that the output is a number you can defend in a review, rather than one someone picked because it looked safe.

1. Sample real inputs. A few hundred items pulled from production, stratified so the rare classes appear at all. Synthetic examples will make every threshold look better than it is, because they lack the ambiguity that causes escalation in the first place.
2. Label them before you look at the model. Have the people who own the decision label each item the way they would decide it. Where two labellers disagree, record the disagreement instead of resolving it — those items are your irreducible floor.
3. Run the question set once and keep everything. Store the chosen option, the full probabilities array, the confidence and the human label for every item. You will want the probabilities later even if you think you only need confidence now.
4. Plot accuracy against coverage. Sort by confidence descending. At each candidate threshold, compute two numbers: the accuracy of the items at or above it, and the share of total volume they represent. This curve is the whole decision.
5. Pick the lowest threshold that meets your accuracy bar, then read the coverage. If 94% accuracy on the automated path requires a threshold that only covers 30% of volume, the automation is not worth building yet — and that is a useful thing to learn before the routing code exists.
6. Set a separate threshold per destination. Weight each by what a wrong answer costs. A misrouted product question costs a transfer. A wrongly approved transfer costs money and trust.
7. Check calibration, not just ordering. Bucket the items by confidence and compare observed accuracy in each bucket. TypeSafe claims its training method produces calibrated decisions, so that higher confidence means higher accuracy; treat that as a claim to verify on your data, because it is the assumption every threshold rests on.

Version the thresholds beside the question set and re-run the evaluation as part of your test suite, the same way you would any other behavioural contract. This is not a model-tuning exercise, it is QA and automation work, and it belongs in CI.

The strongest argument against confidence gating

It is this: a confidence gate can be a way of hiding a model that is not good enough. You pay for the model and you pay for the humans who clear the escalation queue, and if the automated share is small you have added a system without removing work. That argument is correct, and the accuracy-versus-coverage curve is exactly the instrument that settles it. If the coverage at your required precision is low, the honest conclusion is that the question is wrong or the task is not a fit — not that the threshold needs lowering.

Two more cases where gating earns nothing. When every decision must be reviewed by a person anyway for regulatory reasons, the gate adds latency and no saved effort. And when your own labellers disagree with each other on an item, a low confidence answer is the model reporting genuine ambiguity correctly; no threshold resolves ambiguity that the definition never resolved.

The case where it earns a great deal is the one TypeSafe designed for: a cheap, fast classifier deciding which expensive handler runs next. A confidence floor in front of a model router is the difference between a fallback policy and a guess, which is why we treat it as part of the routing layer rather than an afterthought — the same argument we made about routing between models and the one behind our piece on when a decision model beats an LLM.

What to log, so the threshold can be argued about later

Log the full probability distribution on every decision, not the chosen option and not the confidence alone. Distributions are what let you re-tune a threshold months later against real traffic without re-running anything, test a different uncertainty measure on historical data, and answer the question an auditor eventually asks: on what basis did the system act by itself that day? A stored confidence of 0.62 answers none of those. The distribution behind it answers all three.