A green build from an AI agent proves less than you think

When an AI agent writes your code and your tests, a green build tells you the agent was internally consistent. It does not tell you the product is right.

That distinction sounds obvious until you watch it play out in practice. The agent interprets a requirement, writes an implementation, then writes tests that confirm its own interpretation. Every assertion is checking whether the code matches what the agent thought you wanted, not whether the product does what users actually need. The build stays green. The product is wrong.

This is the central argument in Tim Stack's post on product verification for agent-written code, and it's one of the clearer statements of a problem we think is still underappreciated.

The pyramid fix is not enough

The instinctive response is to shift toward more end-to-end tests. Stack addresses this directly: inverting the pyramid doesn't solve the problem. Unit tests, integration tests, and E2E tests all operate at different levels of granularity, but they share the same flaw if they're generated by the same agent that wrote the feature. Moving up the stack does not make the expectations independent. An E2E test derived from the agent's interpretation of a requirement can still confirm the wrong thing convincingly.

The independence problem is not a property of test level. It's a property of where expectations come from.

What has actually changed

Stack makes a point worth sitting with: the economics that originally justified the testing pyramid have shifted. End-to-end tests were expensive not just to write but to maintain — brittle selectors, flaky environments, tests that broke whenever the UI changed. That maintenance burden is now largely something agents can handle. At the same time, the cost of catching a defect late has dropped for teams running agent-assisted repair cycles, where diagnosis and patching can happen in minutes.

This doesn't mean late detection is free. A defect that reaches users can still cause data corruption, broken trust, or downstream damage that no patch undoes. But it does mean the economic case for heavy investment in product-level verification is stronger than it was when the testing pyramid was formulated.

Where the work actually needs to happen

The shift Stack is advocating is not a new test framework. It's a change in where expectations come from. Product verification asks whether the product delivers the right capabilities to users. Code verification asks whether the implementation matches the developer's — or agent's — intent. Those are different questions, and only the first one matters at the release boundary.

Critically, the expectations driving product verification must come from outside the implementation loop: from a product definition, from user research, from domain expertise the coding agent didn't produce. And they have to be protected from modification by the agent. An agent that can edit a failing product test to make it pass has defeated the purpose.

Stack also notes that this protection extends to the acceptance criteria themselves. If the same agent that wrote the feature also writes the acceptance criteria, you've just moved the same problem one level up.

The practical discipline

For small teams, the implication is straightforward: someone has to own the product expectations, define them independently of any agent, and make them a hard release condition. Not a coverage threshold. Not a green badge on a PR. An explicit check, from outside the implementation loop, that the product does what users need it to do.

Unit tests still have a place — Stack is careful about this, and so are we. Code with externally defined correctness criteria (a permissions model, a pricing rule, a regulatory validation) still benefits from precise unit-level checks. But volume and coverage are not a substitute for independent product verification, especially when the code was written by an agent that also wrote the tests.

Sources

• Why Agent-Written Code Needs Product Verification (original)
• Found via the TLDR newsletter