2 min read
AI-generated tests should surface signals, not block merges
Meta's JiTTesting system caught 74 serious privacy bugs before they landed in production. The real lesson isn't the AI—it's the deliberate choice to keep humans on the loop, not in it.
By intermac
The instinct when you build an AI-powered quality gate is to make it hard: fail the build, block the merge, force a resolution. Meta made the opposite call with JiTTesting, and the outcome justifies it.
The system generates executable tests for code diffs before they land—not by reviewing code the way a human would, but by producing tests that demonstrate a concrete behavioral difference introduced by the change. Two parallel workflows feed it: one that reasons about a diff's intent and targets likely failure points, another that treats every diff as if it were already buggy and tries to expose that. Weak catches then pass through an LLM ensemble and a rule-based filter before anything reaches an engineer.
The result in the first half of 2026: 74 serious privacy bugs caught at the diff stage, and a 20% reduction in privacy severity events overall. Those are real numbers.
The 60% figure is the honest part
Meta reports a 60% true-positive rate among findings that reached engineers. That means four out of ten reports that made it through automated filtering and ranking were still noise. At that ratio, a hard gate would be a disaster. Engineers would learn to dismiss alerts, real bugs would hide in the flood of false ones, and adoption would collapse.
This is the trade-off every team building AI-assisted tooling has to name explicitly. A higher recall catches more bugs; a stricter precision filter improves the signal but may miss things. The right operating point depends on how much your team trusts the system and how much friction you're willing to add to the review flow. Meta's answer—route only ranked, high-confidence findings to engineers, let them confirm whether a behavioral change was intended—keeps velocity intact while still surfacing the catches that matter.
Human-on-the-loop is an architectural decision
The distinction Meta draws between
Sources
- Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale (original)
- Found via the TLDR newsletter