Synthetic database benchmarks give you false confidence before migrations

Synthetic benchmarks measure the workload you imagined. Production traffic captures the workload you actually have. That distinction sounds obvious, but most teams don't act on it until something breaks in production.

Airbnb's writeup on their database traffic capture and replay system makes the cost of that gap concrete. During their MySQL 5.7 to 8.0 migration, replay testing exposed a query whose latency jumped from 0.03 seconds to 2.6 seconds — reading 273 MB per join instead of 5 MB — traced to a change in how MySQL 8.0.20+ reads rows during sorting. It also surfaced queries with no ORDER BY, or an ORDER BY without a unique tiebreaker, that silently returned different rows once a new engine applied a LIMIT. Neither of those would show up in sysbench.

The architecture Airbnb built is substantial: a ProxySQL sidecar captures queries to disk, a log processor partitions and rewrites them per cluster, and a fleet of replay workers executes them against target databases at configurable speed multiples. It handles auto-increment rewriting, concurrency preservation, and side-by-side result comparison. Most teams won't build this.

But the principle transfers at any scale.

What small teams can do

You don't need a distributed replay system to stop trusting synthetic benchmarks. A few cheaper steps move you in the right direction:

• Capture a real query sample before any migration. MySQL's slow query log and tools like pt-query-digest can give you the actual query mix — which tables, which join patterns, which query shapes dominate. Run those against a staging clone of your target version before you touch production.
• Test with a production-equivalent data shape. Airbnb restored both test databases from the same snapshot before running comparisons. Schema and data volume matter: a query that runs fine on a small dataset often behaves differently at production row counts, especially after an engine upgrade changes statistics or join order.
• Look explicitly for non-deterministic queries. Airbnb's replay surfaced queries relying on implicit row ordering — a pattern that breaks silently when you switch engines or versions. You can audit for this statically: search for LIMIT without a fully deterministic ORDER BY. It's tedious but not complex.
• Stress test with real throughput multiples, not peak estimates. Airbnb found that replaying 80% more write traffic than baseline pushed average commit latency from 6 ms to 34 ms on one large cluster. Guessing at peak load is how you discover the ceiling at the wrong moment.

The broader point

The deeper issue is that synthetic benchmarks create a specific kind of false confidence: they tell you the system is fine under conditions that never actually occur. Engineers pass them, feel safe, and ship. The failure mode isn't

Sources

• Beyond synthetic testing: Capturing and replaying real database workloads at Airbnb (original)
• Found via the TLDR newsletter