The result

The quality stayed. The wait got shorter.

On a strict, disjoint 128-case multiple-choice set, mapped 4B→32B inference scored 96/128. Native Qwen3-4B scored 89/128 and native Qwen3-32B scored 95/128. On eight long-context prompts, the mapped system delivered 28.21 tokens/s against 23.70 for native 32B, while total latency fell from 10.80 to 9.08 seconds.

Three-panel results overview comparing native Qwen3 4B, mapped 4B to 32B, and native Qwen3 32B on task accuracy, time to first token, and throughput.
The promoted repair-32 operating point. Task quality and long-context performance use separate held-out datasets and are not pooled.

The method

Prefill small. Repair locally. Decode large.

Qwen3-4B reads the long prompt on the first H100. We transfer its key-value cache to Qwen3-32B on the second H100 by matching layers at the nearest normalized depth. This direct path uses no learned mapping coefficients.

The 32B model then recomputes only the latest 32 prompt tokens. That short causal repair restores the recent context most relevant to the next prediction without paying for a full 32B prefill. During generation, an adaptive speculative schedule proposes blocks between 4 and 24 tokens, handles mismatches early, and commits a bonus token when the verified block permits it.

  1. 01

    Draft prefill

    Qwen3-4B builds the prompt KV cache on GPU 0.

    Fast small-model TTFT
  2. 02

    Direct mapping

    Map 36 layers to 64 at the nearest normalized depth.

    No learned weights
  3. 03

    Target repair

    Qwen3-32B recomputes the latest 32 prompt tokens on GPU 1.

    Recover task quality
  4. 04

    Speculate

    Adaptive 4→24-token blocks use fast mismatch handling and bonus commits.

    84.1% mapped agreement

Quality retention

Thirty-two repaired tokens reached the balanced frontier.

Without repair, mapped accuracy was 78/128. Repairing 8, 16, 32, and 64 recent tokens raised it monotonically to 82, 87, 89, and 95 correct answers. The strict follow-up for the selected 32-token policy reached 96/128, one answer above the native 32B control on that sample.

Lollipop chart of target negative log-likelihood on eight disjoint 8K-token prompts: native 32B BF16 at 0.383, mapped 4B to 32B at 0.510, and native 4B at 0.589. Lower is better.
On held-out ~8K-token continuations the mapped path tracks native-32B behavior more closely than native 4B does. Repair-32 was selected as the speed-quality balance, not as proof that the mapped system is generally more capable than native 32B.

Latency and throughput

The first token arrived 2.64× sooner.

Against native 32B BF16 inference, mapped 4B→32B cut median time to first token from 1,139 ms to 399 ms, a 62.1% reduction. End-to-end latency improved by 15.1%, with a paired speed ratio of 1.177× and a 95% interval from 1.045× to 1.325×. Throughput increased from 23.70 to 28.21 generated tokens/s.

Native 4B remained faster at 51.37 tokens/s. The point of this operating mode is not to beat the small model's raw speed. It is to retain large-model-level task behavior while avoiding much of the large model's long-context prefill cost.

What this proves

A useful frontier point, not exact 32B inference.

The mapped verifier accepted 84.1% of proposals. This is agreement with the transferred-cache verifier and is the mechanism that raises generation throughput.

The claim boundary

This experiment establishes an audited speed-quality operating point for KV-transfer inference: native-32B-level accuracy on one strict task set, faster long-context response, and reproducible two-H100 execution.