Part of: this research — view all 4 rows
The baseline estimate centers on the severe model-FLOPs utilization (MFU) penalty of batch-one decoding for a sparse mixture-of-experts model. DeepSeek-V3 activates 8 of 256 routed experts plus one shared expert per token, giving a sparsity of 32 (roughly 37B active out of 671B total parameters) [arXiv 2412.19437]. On an H100, streaming 37GB of active weights per token against 3.35 TB/s of bandwidth caps batch-one MFU near 0.34%. Compared to an achievable 35% MFU at the critical batch size (roughly 9,600 sequences, driven by the H100's FLOPs-to-bandwidth ratio of 295), this yields a baseline per-token compute cost ratio of approximately 103x.
Hardware generation shifts introduce upward pressure on this ratio. While a B200 running FP4 maintains a similar FLOPs-to-bandwidth ratio (roughly 281) and keeps the baseline near 100x, late-2026 deployments like Vera Rubin NVL72 stretch the divergence. Rubin's advertised 35 PFLOP/s dense NVFP4 against 19.2 TB/s of memory bandwidth scales the theoretical batch-one to large-batch cost ratio closer to 140x–160x, assuming the 35% large-batch MFU anchor holds. Weighing these varying hardware profiles puts my median at 115x, slightly above the H100 baseline. Expected declines in overall serving and hardware rental costs do not materially shift that median: they lower absolute costs without necessarily narrowing the relative cost gap, whose core physical divergence remains governed by rigid hardware bottlenecks.
In empirical implementations, achieving theoretical throughput at either extreme is complex. Real-world batch-one serving often suffers from multi-GPU communication delays and latency overheads that further degrade memory bandwidth utilization, pushing the cost ratio higher. Conversely, techniques like multi-token prediction (speculative decoding) can significantly improve batch-one throughput and compress the ratio. DeepSeek-R1, for example, demonstrated a jump from roughly 14 to over 45 tokens per second per B200 GPU when using speculative decoding, highlighting the elasticity of the numerator. These competing effects support an interquartile range of 90x–176x.
The most significant uncertainty lies in the resolution convention, necessitating a heavy right tail. Reiner Pope originally estimated batch-one economics were "up to a thousand times worse," a heuristic derived from calculating total, rather than active, weight fetches. While physically unrealistic for sparse MoE decode steps that only fetch activated experts, a literal interpretation of this framework—or a direct equating of the cost ratio to the raw critical-batch size (roughly 9,600)—would yield an answer in the thousands. I place the 90th percentile at 358x, with those interpretations contributing risk farther into the right tail. The left tail accommodates a scenario where the reference evaluation assumes a much lower large-batch MFU (e.g., 20–25%), which would cut the ratio to the 60x–75x range. My 10th percentile is 63x.
Evaluating this alongside expected declines in overall serving and hardware rental costs kept our median estimate stable, as the core physical divergence between batch-one and large-batch throughput remains governed by rigid hardware bottlenecks.
Ask a followup
Sign in to run · $20 free credit, no card · every claim cited