3 min read
The bottleneck in AI-generated code is verification, not generation
Giving AI agents a self-check loop matters more than model quality. Here's what that means for teams building with AI today.
By intermac
The limiting factor in AI-assisted coding is no longer the model's ability to write code. It's the model's ability to know when the code is correct. That shift has a concrete implication: the teams that invest in verification infrastructure will get compounding returns; the ones that don't will spend their time reading diffs.
Addy Osmani's piece on agentic verification makes this case cleanly. The argument is simple. A self-verifying agent—one that writes a change, runs a check, reads the output, and retries—operates on a tight loop that never involves a human until the work is already passing. A human-in-the-loop agent does the opposite: you read the diff, run the app, find the bug, report it back. That second loop doesn't scale past one person.
Tests are now agent infrastructure
The reframe we find most useful here is treating your test suite as something the agent depends on, not just something humans use. That means the same properties apply: it has to be fast, deterministic, and give specific failure messages. A slow or flaky suite doesn't just frustrate humans—it teaches the agent the wrong lesson. If retrying a failing test occasionally makes it pass, the agent learns to retry rather than fix. That's a silent quality regression with no visible signal.
Osmani's hierarchy is worth internalizing. End-to-end tests are the highest-value investment for agents because they define "done" in terms the agent can chase independently. "Make this E2E test pass" is a target. "Implement this feature" is not. Property-based tests—using tools like fast-check, Hypothesis, or proptest—go a layer deeper, specifying what must never happen rather than what should happen in a particular example. Agents tend to write tests that confirm their own assumptions; property tests force them to handle inputs they wouldn't have chosen themselves.
The trade-off is real
E2E tests are slower and flakier than unit tests. That's not a reason to avoid them, but it is a reason to be selective. A small set of high-value flows—the ones a real user actually takes—is worth far more than exhaustive coverage of edge paths no one travels. Flaky tests in the main suite are worse than no tests, because they erode trust in every result that follows.
The practical setup Osmani describes is one we'd adopt directly: a fast core loop for the agent (subsecond where possible), with heavy E2E and differential tests gated to pre-integration. Flaky tests get isolated immediately, not tolerated. Output stays short and specific—verbose logs of passing tests waste the agent's attention on every turn.
What this asks of us
None of this is passive. Building loops the agent can run itself is deliberate engineering work. Writing E2E tests that cover real flows, specifying properties before implementing code, maintaining a deterministic test environment—these are investments that pay off only if you treat them as first-class infrastructure.
The role that remains firmly human is defining what success looks like before the agent starts. An E2E test, a property, a differential comparison against an old system—pick the right oracle, keep it honest, and the agent can iterate. Skip that step, and you're back to reading every line yourself.
The judgment call isn't which model to use. It's how good your feedback loop is.
Sources
- Agentic Verification (original)
- Found via the TLDR newsletter