Fix the network boundary before tuning your worker pool

The lesson in WHOOP's pipeline writeup isn't about Kubernetes pod sizing or SQS. It's about finding the right layer to attack first.

Their starting pipeline was reasonable: a producer drops messages onto a queue, workers pull tasks and call an internal classification service over HTTP. That's a familiar pattern. At small scale, it works fine. At 15.8 million tasks, it would have taken more than two months. So they optimized—but they optimized in the right order.

Network first, parallelism second

The HTTP boundary between the worker and the classification service accounted for more than half of each task's 45-second runtime. Because WHOOP owned both sides, they pulled the model into the worker as a library, loaded it once per process, and ran inference in-process. That alone took per-task time from 45 seconds to 21—a bigger gain than anything else in the post.

This is worth sitting with. Many teams would have reached for more pods first. More workers, more parallelism, same fundamentals. That instinct is understandable—it feels like progress, and cloud infrastructure makes it easy to throw capacity at a problem. But doubling your worker count on a pipeline where the model loads per-task over HTTP roughly doubles your model-load time too. You're scaling the bottleneck, not eliminating it.

The rule WHOOP articulates at the end is blunt and correct: when you control both ends of a network call, removing it usually beats tuning around it.

The fork/spawn trap

Once the network was out of the way, running two worker processes per pod was a natural next step. They reached for Python's multiprocessing.Pool, which defaults to fork on Linux. The workers deadlocked immediately.

This is a real and underappreciated hazard. XGBoost, NumPy, and similar libraries initialize thread pools and internal locks when they're first loaded. Forking after that initialization copies the lock state into child processes, but not the threads holding those locks. The children wait forever.

The fix is one line: multiprocessing.set_start_method("spawn"). Spawn starts each worker as a fresh interpreter. Slower to boot, but no inherited state to untangle. If your pipeline loads any native ML library before forking workers, set this upfront. Don't wait for the deadlock to teach you.

Idempotency at the output, not the check

At 500 concurrent workers, SQS's at-least-once delivery guarantee stops being a theoretical footnote. Two workers will occasionally both pass an idempotency check and race to write the same output. WHOOP found that catching the "file already exists" exception from an atomic S3 write was the right place to resolve this—not at the pre-task check, and not by using overwrite=True. Overwriting silently succeeds and allows the worker to enqueue the next task in the chain, so duplicates compound. The atomic write failure is the reliable signal that another worker already won.

This is the kind of correctness detail that only surfaces at scale, and it's genuinely useful to see it documented. Idempotency has to extend to everything downstream that treats output existence as a signal.

What small teams should take away

You probably aren't running 15.8 million inference tasks. But the sequencing here applies at any scale: profile the per-task cost before scaling the worker count, know what your process isolation model does to native libraries, and design idempotency around the write, not the pre-check. Get those three things right and parallelism becomes a multiplier on a fast baseline—not a patch on a slow one.

Sources

• Scaling an ML Inference Pipeline for Batch Workloads | WHOOP Engineering (original)
• Found via the TLDR newsletter