Skip to content
intermac
Journal
AI

4 min read

Engineering judgment when agents write the code

AWS Emulator lets coding agents test against a local copy of AWS. The harder question it raises is what engineers actually do when implementation is no longer their primary job.

By intermac

The interesting thing about AWS Emulator is not the emulator itself. It is what building it revealed about where engineering judgment now lives.

AWS CTO Werner Vogels and Swami Sivasubramanian describe a tool that gives coding agents a local copy of AWS to test against — 200-plus services, 400-plus API models, all written and tested by agents, released to the public domain. The motivation is straightforward: an agent working in a loop needs infrastructure that behaves like the real thing or it cannot check its own work. Put that infrastructure on the developer's machine and the agent can verify its output before anyone reviews it.

That part is a sensible engineering decision. The more consequential observation comes in one paragraph near the end: when Swami tried to use an agent to rebuild the Alf replication engine — a job that originally took four months — the agent spun for four nights and made no progress. Once it had a spec and a test environment, the rebuild took about two hours.

That ratio deserves to sit with you for a moment.

The bottleneck has moved

If an agent can compress a four-month implementation into two hours given the right inputs, then the bottleneck is no longer writing code. It is producing the inputs: a clear specification of what correct behavior looks like, and a test environment that can verify it without ambiguity.

This is not a new idea in software engineering. What is new is how sharp the feedback loop has become. A vague spec used to slow a human engineer down. It now stops an agent entirely, or worse, lets it run confidently in the wrong direction.

The AWS team built their scaffolding from public API models written in Smithy, plus fidelity tests that describe end-to-end customer scenarios and are verified against real AWS. The agents build the emulator until those tests pass, and a failing test points at a gap. The merge gate includes multiple security agents doing adversarial reviews alongside deterministic checks that prevent agents from modifying code outside their scope.

That is not a secondary concern. That is the engineering work.

What this means for small teams

For a small team, the shift is uncomfortable in a specific way. Writing code is legible work: you can see it happen, review it, point to it. Designing specs, test environments, and merge gates is less legible but increasingly where correctness is decided.

A few things follow from this:

  • Specifications become load-bearing. A spec that is good enough for a human to interpret loosely is not good enough for an agent to act on correctly. Precision that used to feel like over-engineering is now the minimum.
  • Test environments are infrastructure. The AWS team built the scaffolding for agents to run fidelity tests; an engineer did that work. That kind of setup is not overhead — it is what allows agents to operate without constant supervision.
  • Merge gates need to be designed, not assembled. The article describes multiple security agents each reviewing changes for different attack surfaces, alongside general code review and deterministic boundary checks. That architecture was designed by people. The agents work inside it.

The article puts it plainly: engineers decide what gets built and when the work is correct. That is a narrower mandate than it sounds. It rules out deferring correctness to production, it rules out informal review, and it rules out specs that exist mainly in someone's head.

We think this is roughly right. The judgment engineers exercise is shifting from implementation decisions to system design decisions — the shape of the guardrails, the precision of the spec, the fidelity of the test environment. Those are harder to delegate and harder to shortcut. That is where engineering attention should go.

Sources

Tags

  • #ai-agents
  • #software-engineering
  • #testing
  • #developer-tools