The production reversal
The faster empty road became the slower shared road.
We were tuning a live DeepSeek V4 service across eight H100 GPUs. Moving from two-way tensor parallelism with four data-parallel replicas to four-way tensor parallelism with two replicas improved serial latency. If the evaluation had ended there, the candidate would have looked like a win.
It did not survive concurrency. With 128 decode-heavy requests running together, aggregate output fell from approximately 10,039 to 1,197 tokens per second. Median time-to-first-token, or TTFT, rose from about 0.347 seconds to 22.450 seconds. The configuration that helped one request made the median loaded request wait 64.7 times longer.
It applies to this model, runtime, hardware, and concurrency profile. It does not establish that TP2 and DP4 are universally optimal. It establishes that serial latency alone selected the wrong configuration here.
What latency means
A production latency number needs a workload and a user-visible boundary.
TTFT measures when the first streamed token arrives. For a reasoning model, that first token may be hidden reasoning rather than visible answer text. We therefore timed first visible answer content separately. We also measured prompt and completion throughput across all requests, because a low per-request latency can hide a system that serves far less total work.
Those metrics answer different questions. TTFT shows whether the scheduler starts a request promptly. Aggregate token rate shows how effectively the fleet moves work. First-visible-answer time shows when the user receives content they can actually read. A candidate had to improve latency without materially reducing concurrency or visible-answer reliability.
The serving setup
Every candidate faced the same model and reproducible streaming workloads.
The sweep ran after a fixed service revision. Requests used the OpenAI-compatible streaming endpoint and a unique cache salt, which kept runs independent while exercising the real cache-isolation path. Short, long-prefill, decode-heavy, and mixed profiles exposed different pressure points.
- Model
- DeepSeek V4 Flash
- Runtime
- vLLM 0.25.0
- Hardware
- 8 × NVIDIA H100
- Context
- 131,072 tokens
- KV cache
- FP8
- Load profiles
- Short, 10K prefill, decode-heavy, and mixed
- Request timing
- First token, first visible answer, and wall time
- Acceptance rule
- Latency gain without material concurrency or answer regression
The retained service used TP2, DP4, expert parallelism across all eight GPUs, DSpark with seven speculative tokens and greedy draft sampling, automatic MoE backend selection, 0.81 GPU memory utilization, and a 16,384-token scheduler budget.
Why topology lost
More GPUs per replica left too few replicas for the queue.
Tensor parallelism divides one model replica across GPUs. Increasing it can shorten the path for an isolated request, but it also consumes more GPUs per replica. On this eight-GPU host, TP4 left two data-parallel replicas where TP2 supported four.
The serial test captured the first effect. The 128-way test exposed the second. Twice as many model replicas gave the scheduler more independent capacity to absorb concurrent decode work. Under load, that capacity mattered far more than the serial gain from widening each replica.
This is why topology cannot be selected from a single-request benchmark. The benchmark has to recreate the sharing pattern the deployment is expected to serve.
The other gates
Neutral token metrics were not enough, and unsupported paths were not benchmarks.
Changing speculative draft sampling from greedy to probabilistic with seven speculative tokens was approximately neutral at high concurrency. It reached 10,158.7 output tokens per second with 0.322-second median TTFT at 128-way decode. On the 64-way 10K profile, it reached 34,652 prompt tokens per second with 8.890-second median TTFT.
But it failed the fixed five-prompt max-reasoning answer gate. Three of five requests produced visible answer content, compared with four of five under greedy sampling. When missing answers were treated as right-censored infinity, median visible-answer time rose from 43.95 to 76.85 seconds. Five prompts are too few for a population reliability claim, but the result was enough to reject a production change that offered no clear load advantage.
The last two candidates never reached performance measurement. Mega MoE rejected the H100 architecture at startup, while expert-parallel load balancing could not initialize with this model and runtime. Reporting either as a latency loss would confuse compatibility with performance.
We also left GPU memory utilization at 0.81. Only about 0.9 to 1.1 GiB remained free per H100 after warmup, and the accepted 64-way long workload had zero preemptions. Raising the limit would have purchased more out-of-memory risk without evidence that memory headroom was the active bottleneck.
Recovery under load
Rollback ended with two clean workload suites.
After the rejected candidates, we restored the original service byte-for-byte and reran the high-load paths. All 128 short decode-heavy requests and all 64 requests with 10K-token prompts completed. The service reported no restart, preemption, or queued request.
The final recovery numbers are a validation of the restored configuration, not a paired comparison against every candidate. Their purpose is operational: the experiment ended with known service state and verified user paths.
The takeaway
A rollback can be the most useful output of an optimization sweep.
The retained configuration was not chosen because it won every microbenchmark. It stayed because every alternative failed a requirement the service actually had: concurrent capacity, visible answers, hardware compatibility, startup, or safe memory headroom.
The remaining long delay before visible answers in max-reasoning mode was dominated by variable hidden reasoning length, not prefill or queueing. Reducing it would require a reasoning budget or adaptive policy. That is a product and quality decision, not a no-risk kernel flag.
Production inference optimization is a constraint problem. Measure the shared workload, define the user-visible boundary, preserve a recovery path, and treat a rejected candidate as evidence rather than a failed experiment.