Intuition

Capture the representational power of the large model without paying its cost on every generated token.

Prefill is where a model reads the prompt and turns it into an internal representation. Our intuition was to give that job to the 32B model, whose greater capacity can form a richer representation of the prefix, then translate the resulting KV cache into the 8B model's coordinate system.

If that handoff preserves the useful structure, the continuation gets the benefit of a large-model read while every new token is generated by the smaller model. In short: use 32B once to understand the context, then use 8B repeatedly to answer from it.

Potential applications

Where a large read and a small-model continuation could matter.

  • Long-context copilots: let a larger model process a long document set once, then use a smaller model for a detailed response.
  • Retrieval and search synthesis: build a richer representation of retrieved evidence before generating the answer at lower per-token cost.
  • Agent workflows: encode long tool histories, plans, and workspace state with a larger model, then run iterative actions with a smaller decoder.
  • Report and code generation: spend more compute understanding a large specification while keeping a long output economical.
  • Disaggregated inference systems: specialize prefill and decode pools around the model size best suited to each phase.

The result

One model read the prompt, and a smaller model answered at its own speed.

Ordinary inference asks one model to do two different jobs: read the full prefix, then generate the answer. We tested a split path instead. Qwen3-32B performed prefill, a frozen low-rank ridge map translated its key-value cache, and Qwen3-8B decoded from the mapped state.

The handoff worked as a systems primitive. Across 1K to 16K prefixes, mapped 8B generation sustained 46.8 to 48.0 tokens per second, effectively matching native Qwen3-8B and running 68% to 94% faster than native Qwen3-32B. At the same time, the measured path used 14% to 35% less active GPU compute than keeping the full request on 32B.

Three-card summary showing the mapped 32B-prefill to 8B-decode method improving paired HellaSwag and PIQA quality over native 8B, delivering 1.94 times native-32B decode throughput, and using 14% less active GPU compute at 16K.

Quality signal

The mapped cache recovered most of the 8B-to-32B multiple-choice gap.

In the unified residual-v2 scorer, every arm saw the same 1,000 HellaSwag and 1,000 PIQA examples. Native 8B scored 81.7% and 87.3%. The ridge-mapped 32B-to-8B path reached 86.2% HellaSwag and 90.6% PIQA, within 0.6 and 1.1 points of native 32B.

That corresponds to 88.2% of the HellaSwag gap and 75.0% of the PIQA gap recovered. The gains over native 8B were paired and positive: +4.5 points with a 95% interval of +2.2 to +6.8 on HellaSwag, and +3.3 points with a 95% interval of +1.1 to +5.6 on PIQA.

The overview below combines the requested native-8B reference from the original three-arm evaluation with the ridge and native-32B bars from the residual-v2 scorer. It is useful for visual orientation, but because the scorer changed, its cross-arm differences are illustrative rather than paired evidence.

Illustrative quality overview combining the original native-8B reference with ridge-mapped and native-32B values from the residual-v2 scorer. A note states that the cross-arm differences are not paired.
Illustrative overview using the two requested source charts. The native-8B bar uses the original scorer, while ridge and native 32B use the residual-v2 scorer; only the paired quality panel in the combined outcomes graphic above supports the quality-lift claim.

8B-speed decode

Once the handoff finished, generation ran at the small model's speed.

The mapped path decoded between 46.8 and 48.0 tokens per second at every measured prefix length. Native 8B occupied nearly the same range. Native 32B declined from 28.6 tokens per second at 1K to 24.1 at 16K.

This is the cleanest systems result in the study: the source model's large prefill did not drag large-model decode cost into the continuation. The answer phase behaved like an 8B decode because it was an 8B decode.

Line chart showing mapped 32B-prefill to 8B-decode throughput tracking native Qwen3-8B at roughly 47 tokens per second and exceeding native Qwen3-32B by 68% to 94%.
Median decode throughput after the first token. The fixed 64-token generation measured the answer phase separately from source prefill, transfer, mapping, and first-token work.

Active compute

The split path used less active GPU time than staying on 32B.

For the measured requests, the mapped path consumed 1.55 to 4.67 active GPU-seconds, while native 32B consumed 2.37 to 5.42. The relative saving was largest at 1K, where active compute fell 35%, and remained 14% at 16K.

The accounting is deliberately phase-aware. Source prefill and 8B decode are counted once. Mapping and transfer time are counted on both participating GPUs. This is the useful compute comparison for a shared serving pool, not a claim that the request ran on one physical GPU.

Line chart showing the mapped large-prefill small-decode path using 14% to 35% less active GPU time than native Qwen3-32B across 1K to 16K prefixes.
Estimated active GPU-seconds per request from measured phase times. The shaded region is the active-compute saving versus native Qwen3-32B.

What the map learned

The learned map found a strong shared coordinate system for keys.

Raw nearest-depth source keys were almost orthogonal to native 8B keys, with cosine similarity near zero and normalized RMSE of 1.259. Ridge mapping changed that dramatically: key cosine rose to 0.941 and key error fell 72.6% to 0.345 NRMSE.

Values improved too, moving from near-zero cosine to 0.504 and from 1.455 to 1.050 NRMSE. That remaining value mismatch is the clearest technical frontier. The transfer already learned where key information lives; the next step is preserving more of the target-compatible value state.

Two-panel tensor-distance chart comparing unmapped source, ridge-mapped, and ridge-plus-residual caches for keys and values using normalized RMSE and cosine similarity. Ridge mapping strongly improves keys, while values remain less aligned.
Mean tensor distances across eight untouched FineWeb-Edu prefixes totaling 4,826 tokens. Keys are compared before RoPE; values are compared in native value space.

The residual improved behavior rather than raw reconstruction.

A rank-16 residual network did not materially change final tensor NRMSE because it was trained against behavioral KL, not cache reconstruction error. On that intended objective it was consistent: source-teacher KL fell 21% to 26% on four frozen windows, including two 1,024-token confirmations with paired intervals entirely below zero.

Paired slope chart in which all four frozen evaluation windows move downward from ridge mapping to ridge plus a residual neural network, indicating 21% to 26% lower source-teacher KL.
Each line is the same frozen window before and after the selected rank-16 residual at scale 1.0. Every line moves downward, representing a 21% to 26% reduction in teacher divergence. This mechanistic gain did not beat ridge on HellaSwag, so ridge remains the quality headline.

What this establishes

Large-model reading and small-model answering can be separated, but the full product gate is still ahead.

The experiment establishes three things. First, mapped 8B decoding can preserve native-8B throughput after a 32B prefill. Second, active compute can be lower than keeping the request on 32B. Third, under one paired multiple-choice scorer, the mapped path can recover most of the quality gap from 8B toward 32B.

The boundaries matter. Cold time to first token was 9% to 15% slower than native 32B because the mapped path paid for the same 32B prefill plus transfer and mapping. Reserved-GPU accounting was also higher than native 32B because two H100s participated. In the original quality pipeline, ridge mapping did not beat native 8B, and later RULER retrieval screens showed that multiple-choice gains do not yet establish general long-context fidelity.

The product interpretation

The method worked as a new serving primitive: spend 32B compute on the read, then buy the continuation at 8B speed. The next gate is a mapper that retains this systems advantage while passing long-context retrieval and integrated end-to-end latency.