Assumption and Core Arithmetic The central estimate for this cost ratio relies on the explicit assumption of a 35% large-batch model-FLOPs utilization (MFU) baseline — representing realistically achievable large-batch performance rather than a theoretical roofline. Batch-one decode is strictly memory-bandwidth bound, while decode at the critical batch is compute-bound. Consequently, the cost ratio is driven by the hardware's FLOPs-to-bandwidth balance (R). This constant R is remarkably stable across current generations: approximately 295 for H100 (BF16/FP8) and 281 for B200 (FP4) jax-ml.github.io. Decoding at batch size one yields an MFU of exactly 1/R, or roughly 0.34%. Multiplying the 35% large-batch MFU by R yields a cost ratio of roughly 103x to 105x, placing the median squarely at 105.0.

Architecture and the Critical Batch It is important to correct the essay's assumption that the critical batch size is 2,400. DeepSeek-V3 activates 37B of its 671B parameters per token, routing to 8 out of 256 experts plus 1 shared expert arxiv.orggithub.com. The correct expert-count sparsity is therefore 32, not 8 arxiv.org. Applying Pope's rule (critical batch ≈ 300 × sparsity) yields a critical batch size between 9,000 and 9,600. This is corroborated by DeepSeek's own production disclosures, which imply decode deployment units operating at roughly 12,000 to 13,000 concurrent sequences github.comgithub.com. Operating at or above the critical batch size is practically achievable, validating the compute-bound baseline.

Upside Risks: Rooflines and 'Whole Box' Economics The highly right-skewed upper tail (p75 of 153.3, p90 of 271.7) accommodates alternative resolution conventions and edge-case deployment realities. If the evaluation strictly applies a theoretical roofline boundary (100% MFU at critical batch), the multiplier approaches R itself, or roughly 280–300x. Furthermore, true batch-one self-serving cannot utilize a single GPU; streaming 671GB of weights requires an entire multi-GPU node. When charging the cost of the whole node against single-user throughput (which often measures only ~11.7 tokens/s per chip luminal.com), the empirical cost ratio approaches 180–270x. Finally, newer hardware on the horizon (like Blackwell Ultra or Rubin) boasts even higher FLOPs-to-bandwidth ratios, which could push the baseline higher developer.nvidia.comdeveloper.nvidia.com.

Downside Risks: Measured Decode MFU and Speculative Decoding Conversely, the downside tail (p10 of 55.0, p25 of 83.0) accounts for real-world inefficiencies at large batch sizes and software improvements at batch one. Actually-measured large-batch decode MFU in distributed setups often struggles to exceed 10–15% due to KV-cache and network overheads lmsys.org. If the ratio is resolved against these lower, empirically measured large-batch decode efficiencies rather than the 35% prefill-style baseline, the cost multiplier would compress to roughly 30–60x. Additionally, optimizations like speculative decoding and MTP effectively convert batch-one memory latency into arithmetic work, further improving batch-one economics lmsysorg.mintlify.app and anchoring the lower bounds of the forecast.