2 min read
Treat your agent harness like production code, not like a config file
The instructions, tools, and sandboxes that run your AI agents are a software product. Most teams aren't treating them that way, and it shows.
By intermac
Most teams ship changes to their agent harness the way they used to ship SQL scripts before migrations existed: manually, optimistically, with no rollback plan. That's the gap this Thoughtworks piece names clearly, and it deserves to be taken seriously.
The harness — the prompts, the tool definitions, the sandbox configuration — drives agent behavior as directly as application code does. Change it carelessly and you've shipped a regression you may not detect for days. The article puts it plainly: a harness you cannot measure is a harness you cannot safely improve.
The measurement trap
The tempting proxies are tokens consumed, suggestions accepted, and tests passed. These are activity metrics. They tell you the system is busy; they don't tell you it's doing the right work. A harness could double the number of accepted suggestions while quietly degrading output quality on the tasks that actually matter.
The alternative the article proposes is a task-based golden test set drawn from real work, not public benchmarks. Twenty or thirty representative jobs the agent is expected to handle, each with a scoring mechanism, versioned alongside the harness itself. That way a change and its evidence travel together. Run a regression on every meaningful change, not just on releases.
This is not a novel idea. It's how good software teams already treat their test suites. The novelty is remembering that the harness is in scope.
Cost and latency are product metrics
One point in the article that often gets skipped in AI discussions: a harness that improves output quality while doubling inference spend has not improved the product. Cost and latency should be treated as first-class metrics alongside quality. For small teams especially, a 2× cost increase on a core agent workflow is a business problem, not a footnote.
Evaluation needs an owner
The article identifies something true and uncomfortable: evaluation sets decay. A test set that was representative in March could be misleading by June as real usage shifts. Without a named owner and a review cadence, the eval becomes what the article calls
Sources
- The harness is a product (original)
- Found via the TLDR newsletter