AI products rot differently: model drift is a quality problem, not just an ops problem

Traditional software breaks visibly. A button stops working. A form throws a 500. Users file tickets. You fix it. AI products don't work that way—and if you're building with AI components, Ravi Mehta's piece on product quality puts the core problem precisely: your feature can keep working long after your customers have given up on it.

That sentence should sit uncomfortably with anyone shipping AI-assisted workflows.

Silent degradation is the real risk

With a traditional feature, the failure modes are relatively bounded. You can enumerate the paths, write tests, and know when something breaks. A natural-language interface blows that up. Users ask the same thing in dozens of ways, combine requests no one anticipated, and bring contexts the team never modeled. The output varies across all of it.

Worse: the product can degrade without a single line of code changing. Models update. Retrieval indices go stale. User expectations shift as they see better alternatives. None of this triggers an alert. There's no stack trace. The feature returns results—just increasingly less useful ones.

What you get instead is behavioral drift on the user side. They start double-checking answers. They revert to the spreadsheet the feature was supposed to replace. They quietly route around it. No bug report, no angry email—just a slow withdrawal of trust that shows up eventually in retention numbers, long after the damage is done.

Mehta calls this "quietly quitting your feature." It's the right framing. And it means usage metrics alone won't tell you what's happening. A feature can show steady session counts while users increasingly treat it as a last resort rather than a primary tool.

What changes for teams building with AI

The traditional launch-and-monitor model assumes the product is static between releases. That assumption is wrong when an external model is part of the experience. You need ongoing evaluation—not just at deployment, but continuously—against a defined quality bar and across a representative range of real inputs.

This means a few things in practice:

• Define acceptable performance ranges, not pass/fail checks. For deterministic features, binary tests make sense. For AI features, you're managing distributions. Set the acceptable band and monitor for drift outside it.
• Look for workarounds, not just errors. If users are copy-pasting AI output into another tool before acting on it, that's a signal. Instrument for it.
• Treat model updates as releases. When a model or prompt changes—even if your code doesn't—run your eval suite. An upstream provider update is a deployment event for your product's quality.
• Build feedback loops before you need them. By the time retention drops measurably, you've already lost months. You want signal earlier: task completion rates, re-attempts, fallback patterns.

Quality as infrastructure

The deeper point in Mehta's piece is that quality is a system, not a checklist. For AI products, that system needs to be more robust than for traditional software, because the surface area is larger and the failure modes are quieter.

Small teams building AI-assisted tools are especially exposed here. The temptation is to ship fast and monitor later. But "later" in an AI product often arrives silently—in engagement numbers that look fine until they don't. Building the evaluation infrastructure early, even crudely, is worth more than most features on the roadmap.

Quality isn't what you ship. It's what users experience six months after you ship it.

Sources

• Product Quality. (original)
• Found via the TLDR newsletter