Model name is not a capability guarantee

The model name on your API call is not a description of what your users receive. That is the blunt takeaway from Lon Lundgren's six-week analysis of 43,261 invocations of the same named model at the highest user-selectable effort settings. The median invocation delivered 123 thinking tokens. Published benchmarks for the same model at the same effort level use roughly 24,000 to 51,000. That is a gap of seven to eight doublings in reasoning compute. In 39.2% of invocations, the model performed no thinking at all.

For anyone building a product on top of a hosted model, this reframes the whole evaluation question. Benchmark scores are not a description of typical production behavior. They are a description of what the model can do under a favorable inference regime that your users may rarely or never receive. When you ship a feature and call it "powered by" a frontier model, you are making an implicit promise that the analysis suggests you cannot keep consistently.

The trade-off that was hidden

The usual framing of the model-selection decision is capability versus cost. Pick the strongest model your budget allows, check the benchmark scores, ship. What Lundgren's data adds is a third axis: inference stability. Not only did delivered reasoning vary sharply across invocations, it declined over the observation period. Median invocation thinking fell by more than 50% between July and August in the invocation-level comparison. The decline survived multiple reweighting approaches—by day, session, and project—ruling out simple workload shifts as the cause.

Instability of this kind is operationally worse than a flat reduction. A flat reduction is at least predictable; you can design around it. Instability means your evaluation results do not transfer reliably to production, your production behavior does not transfer reliably across time, and failures appear as intermittent regressions that are hard to diagnose.

What to do differently

First, stop treating benchmark scores as a proxy for production capability. Run your own evals, but run them continuously against live traffic, not just at release time. If the provider exposes reasoning-token counts, log them. Lundgren's analysis was possible because he built a proxy to capture wire-level data. That investment paid off. If your application depends on reasoning quality, you should be doing something similar.

Second, be skeptical of effort-level labels. "Always thinks deeply" is a behavioral description, not a contractual guarantee. The analysis shows that xhigh and max effort settings produced near-zero thinking in a large share of invocations. Labels are not specifications.

Third, account for this risk in your cost modeling. When reasoning depth drops below the threshold your task requires, the recovery cost falls on you: retries, parallel candidates, longer agent runs, or human review. What looks like cheap inference on the provider's pricing page can become expensive once you factor in the work required to compensate for shallow reasoning.

The broader point

Lundgren draws a distinction between the model frontier—what the underlying model can do—and the accessible frontier—what a particular user can reliably elicit. Historically these were close enough that model access was a reasonable proxy for capability access. The data suggests that gap is widening.

For small teams building on hosted models, the practical implication is uncomfortable: you are shipping a dependency you do not fully control, and the variance in that dependency may be larger than you assumed. The right response is not to stop building on these models. It is to instrument carefully, set user expectations accurately, and design for graceful degradation when reasoning depth falls short.

The model is only half the system. The inference regime is the other half, and right now most product builders are flying blind on it.

Sources

• Lon Lundgren (@Lon) on X (original)
• Found via the TLDR newsletter