3 min read
The default 0.50 threshold is quietly breaking your AI evals
Decision models for automated evals are only as good as the threshold you tune them to. Most teams never touch the default—and pay for it in missed failures and false alarms.
By intermac
The problem isn't which decision model you picked. It's that you're probably running it at 0.50 and calling it done.
Hamel Husain's post on decision models for evals makes this concrete: a model could assign every real failure a score of 0.40 and every passing trace a score of 0.10—classifying everything perfectly at a 0.30 threshold—but at 0.50, it catches nothing. That's not a model problem. That's a configuration problem that's entirely within your control.
The threshold isn't a detail
Decision models return a probability score. The threshold converts that score into a binary call. Leaving it at 0.50 is a bet that the model's internal scoring happens to align with that cutoff on your specific data, for your specific task. That bet is usually wrong, and you won't notice unless you measure it.
The post shows how to tune thresholds using labeled data and cross-validation, and how to set minimum floors on both precision and recall before you start optimizing. This matters because precision and recall pull in opposite directions: lowering the threshold catches more failures but generates more false alarms. The right tradeoff is a business decision, not a technical default. An automated alerting system with noisy alerts gets ignored. A system that misses failures gives you false confidence.
Of the models tested—Jev, Perplexity Decider, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions—only Perplexity, Liquid d1-ultra, and OpenAI could meet both 80% precision and 80% recall on the apartment leasing trace dataset. After threshold tuning, OpenAI reached an F1 of 0.87 at 3.72 cents per 100 eval runs. Perplexity came in at 0.79 F1 for 0.82 cents. That's a real cost-performance tradeoff worth reasoning about, not a default to accept.
Calibration is a separate question
If you want to use the model's score as a confidence estimate—not just a pass/fail signal—you need to check whether the probabilities are calibrated. A score of 0.80 should mean roughly 80% of traces with similar scores actually contain failures. The post shows that some models are well calibrated out of the box (Jev and Liquid d1-ultra on this dataset) and others are not. Platt scaling can help, but it can also make things worse, as it did slightly for OpenAI Decisions. You have to measure.
What this means for small teams
None of this is new methodology. Threshold tuning, precision-recall tradeoffs, and probability calibration are standard classifier evaluation practices. What's new is that more product teams—without classical ML backgrounds—are now building eval pipelines and reaching for decision models as a cheap, fast alternative to LLM judges.
The risk is treating a decision model like a black-box API call: send in a trace, get back a pass or fail, move on. That approach skips the part where you verify the output is actually useful for your task.
The practical steps are straightforward: collect labeled traces, save model scores, tune thresholds with cross-validation, set explicit precision and recall floors, and check calibration if you need scores as probabilities. A coding agent can do most of this with a well-structured prompt, as the post demonstrates.
The cost of skipping this work isn't obvious until your eval pipeline starts missing failures it should catch—or flagging things it shouldn't. By then, you've shipped on bad signal.
Sources
- How To Use Decision Models for Evals (original)
- Found via the TLDR newsletter