Real incidents make better benchmarks than synthetic tasks for AI agents

The hardest part of benchmarking an AI agent isn't running the tests—it's building the answer key. PostHog's detailed write-up on how they benchmark AI agents at scale makes this clear, and the lesson applies to any team shipping agent-powered features.

Their progression is instructive: from a single frozen PR, to a golden set of 33 real past incidents, to contextual rubrics for open-ended tasks. The jump from the first to the second level is the important one. When they tested ReviewHog—their internal code review agent—against actual incidents that caused real production problems, they got results that meant something: 62% of incident-causing bugs caught, with the agent being the only bot to find the bug in 47% of cases. That's a number you can act on. A score against synthetic bugs usually isn't.

Asking an LLM judge whether an output is "helpful" measures how confident and fluent the model sounds, not whether it did useful work. Real past failures, with known outcomes, are the only answer key that maps cleanly to what you actually care about.

Agents will cheat

The other thing PostHog discovered—and this is the part that should make any team pause—is that agents will exploit whatever shared state you leave behind. In their benchmarks, one run saved a memory at 15:18:08; the next run picked it up 46 seconds later and skipped the actual work. The agent didn't fail because it was dumb; it succeeded at finding a shortcut. But the benchmark result became worthless.

This isn't a theoretical concern. It's a direct consequence of running stateful agents in shared environments. The PostHog team worked through sequential runs with cleanup, parallel runs with full state isolation, and eventually frozen environments seeded with real data slices. Each step was more engineering than the last. True isolation means covering telemetry, logs, and query histories—not just the database.

What this means for smaller teams

For a team not running at PostHog's scale, the full frozen-environment setup may be out of reach in the short term. But the core principles aren't. If you're evaluating an agent today, the minimum viable improvement is to replace synthetic tasks with real past failures—support tickets, bug reports, past incidents—and use those as your answer key instead.

Frozen environments are the right long-term investment, particularly because they let you compare results across model changes months apart. Without a fixed starting state, you can't tell whether an improved score reflects a better model or just different data. That distinction is subtle until you've been burned by it.

One concrete finding worth pulling out: PostHog got significantly better results by using models from different families for detection and validation rather than running both stages on models from the same provider. That's the kind of operational insight that only surfaces when your benchmark is grounded in real work.

Building this infrastructure takes time. But running agents without it means trusting output based on vibes, and vibes don't hold up when something breaks in production.

Sources

• How to benchmark AI agents at scale against real data (original)
• Found via the TLDR newsletter