DFlash generated the same tokens in less than half the time.
Qwen3-8B DFlash predicted 16-token blocks in parallel, reached 2.154× geometric-mean speedup, and matched native token IDs in all 12 confirmation runs.
Read blogFeatured
An 8B model listened once. A 1B model answered, and got smarter doing it.
Ultravox 8B built the audio KV cache, a mapper plus rank-16 residual translated it into the frozen 1B decoder, and sealed-test accuracy nearly doubled (31.2% vs 17.2%, p=0.047) at 133-150 tokens per second.
Read blogSpeculative KV cache transfer matched 32B task accuracy at 19% higher throughput.
Qwen3-4B prefilled, transferred its KV cache, and drafted tokens for a repaired Qwen3-32B verifier. The system matched native-32B task accuracy while cutting long-context latency by 15.1%.
Read blogA 70B model built the context. An 8B model answered at near-8B speed.
Ultravox 70B built a richer audio KV cache and transferred it to an 8B model for near-native-speed generation. The handoff reached 81.7% accuracy at 59.6 tokens per second, recovering 12.7 points over native 8B.
Read blogA 32B read handed off to 8B-speed generation.
Qwen3-32B prefilling followed by mapped Qwen3-8B decoding reached 1.94 times native-32B decode throughput at 16K while using 14% less active GPU compute, with promising paired quality results.
Read blogThe small model already did the reading.
Before a model answers, it pays to read. We are making that prefill step cheaper by letting a smaller model read first, then carrying the work into Qwen and Llama targets.
Read blogA faster serial topology made 128-way TTFT 65× worse.
A topology that improved single-request latency collapsed under 128 concurrent decode requests, showing why production inference tuning has to measure load, visible answers, and recovery together.
Read blogA learned KV handoff cut 8K target time-to-first-token by 87%.
A frozen Minitron 4B to Llama 3.1 8B study made an 8K target handoff 7.9× faster than native prefill while improving held-out cache quality over direct reuse.
Read blogA rollout residual cut 30B sibling-transfer divergence by 6.5%.
A frozen Qwen3-30B-A3B sibling test improved 20 of 28 paired windows, while a matching-layer ridge map produced the lowest mean divergence.
Read blogAlignment first cut 32-step KV-transfer divergence by 11.1%.
Nine overnight Qwen3 sibling-transfer experiments showed that ridge alignment followed by rollout-aware correction produced our strongest 4B result.
Read blogPartial recomputation reached 100% next-token agreement.
Recomputing two target layers removed the nonlinear ceiling of an affine cache map across 480 held-out toy-model comparisons.
Read blogThe mapped cache retained 91.2% of task quality.
Our Qwen3 1.7B to 4B map retained 91.2% of chance-normalized task quality and reached 65% next-token agreement.
Read blogA ten-point sweep revealed three mapper tradeoffs.
A ten-point sweep picked k=24 on prefix loss while attention similarity peaked at k=8 and mapping time kept climbing.
Read blogMoving a KV cache from Qwen3 0.6B to 1.7B.
A learned map retained 85.1% of chance-normalized task quality and improved next-token agreement 30× over direct injection.
Read blog