Skip to content
intermac
Journal
AI

3 min read

AI products need three feedback loops, not one

Spotify's engineering blog names something most small teams get wrong: research, model, and experience work each demand different definitions of quality, customer, and success—and conflating them creates invisible problems.

By intermac

Most teams building AI products treat evaluation as one problem. It isn't. Spotify's engineering post on AI product management lays this out clearly, with three PMs describing what their work actually looks like across research, foundation models, and user-facing experience. The language overlaps—everyone says "evals" and "quality"—but the meaning shifts at each layer.

That gap is where things go wrong on small teams.

The three layers are genuinely different jobs

At the research layer, Henok describes success as "technical leverage": the organization makes a better decision, or gains confidence it didn't have before. The feedback loop is long, measured in months, and the output often lands on another team's roadmap rather than in a shipped feature. Impact here is influence, not a metric.

At the model layer, Divita's customer is not a user—it's another product team. Quality means something different to each team that integrates the model. The two real levers are what you teach the model and how you measure what it learned, and success is defined by whether anyone actually ships with the model at all. As she puts it: "A good model is essential, but its value depends on whether another team can use it to solve a problem that matters to them."

At the experience layer, Humberto faces users directly. The input is open-ended. A/B testing still matters but can't cover every case. Evaluation becomes a continuous product of its own—custom judges, datasets built from real usage patterns, feedback loops designed to detect when the system misunderstood a request and route that signal back through the stack.

Three different customers. Three different definitions of quality. Three different cadences for knowing whether something worked.

What this means for small teams

Spotify can staff three principal PMs for three distinct problems. Most teams building AI products can't. That's not an excuse to ignore the distinction—it's an argument for naming which mode you're in at any given moment.

The failure pattern we see most often: a team runs their whole AI product through one feedback loop, usually the experience one, because that's where users are loudest. They optimize the interaction layer while the model quietly underperforms tasks they never bothered to define evaluation criteria for. Or they treat a research insight as a roadmap before they've agreed on what evidence would justify building it.

The Spotify post makes a point worth repeating: a promising model architecture is not a roadmap. The gap between a prototype that works in a demo and a system that holds up for real users at scale is real, and research evals and production evals are measuring different things.

The trade-off worth naming

Separating these layers adds overhead. If you're a two-person team, running three distinct evaluation frameworks and three feedback loops is probably too much. But the cost of conflating them is higher: you end up optimizing the wrong thing, with no way to know it, because your signal is coming from the wrong layer.

The minimum viable version: be explicit about which layer a given piece of work lives in before you decide how to measure it. Research work gets measured by organizational decisions it informs. Model work gets measured by whether partner teams can ship with it. Experience work gets measured by whether users find it useful and come back.

One vocabulary, three distinct jobs. The sooner small teams recognize that, the less time they spend debugging the wrong layer.

Sources

Tags

  • #ai
  • #product-management
  • #feedback-loops
  • #evaluation
  • #platform