The handoff

Could 1.7B read the prompt for 4B?

Long prompts are expensive because every model asked to continue them has to build its own attention cache. We wanted to know whether Qwen3 1.7B could do that first pass, translate its keys and values, and let Qwen3 4B start decoding from the translated state.

Shape compatibility is the starting point. Our acceptance protocol then checks implementation controls, held-out task accuracy, token distribution fidelity, and latency in that order to verify that the target model preserves its intended behavior.

What we ran

One real model pair with disjoint fit, selection, and evaluation data.

Source model
Qwen3 1.7B
Target model
Qwen3 4B
Mapper fit
500 FineWeb-Edu prompts
Selection
100 disjoint prompts
Token fidelity
100 WikiText prefixes
Task checks
1,000 HellaSwag + 1,000 PIQA
Source-layer count
k=24 selected from 10 candidates
Runtime
BF16 model, FP32 solve, H100

For each target layer, the mapper selected source layers, removed source rotary position encoding from keys, learned per-head linear maps in content space, then applied target positions before cache injection. Target-cache reinjection and negative controls passed before we trusted the mapped result.

What improved

The map retained 91.2% of normalized task quality.

On HellaSwag, the mapped cache scored 62.4% against the target oracle's 66.5%. On PIQA it scored 74.1% against 76.1%. After accounting for each task's chance floor, mean retention was 91.2%, above our 80% gate.

The learned map lifted HellaSwag accuracy to 62.4% from 24.6% with direct injection, a 2.5× improvement. PIQA rose to 74.1% from 50.9%. Together, those gains show that the map recovered useful structure beyond compatible tensor dimensions.

Bar chart comparing target oracle, direct cache injection, mapped transfer, and reinjection accuracy on HellaSwag and PIQA. Mapped transfer approaches the oracle while direct injection remains near chance.
Held-out task accuracy and normalized retention. HellaSwag and PIQA use 1,000 examples each. Source: retained run qwen3-1p7b-to-4b-paper-scale-fp32-eval100-20260810T031000Z.

The fidelity signal

65% agreement defines the next optimization target.

The mapped cache matched a normal Qwen3 4B prefill on the next token 65% of the time. A 2,000-resample bootstrap placed the 95% confidence interval at 56% to 74%. Our next milestone is to move that distribution to the predeclared 80% fidelity threshold.

The mapped-to-oracle perplexity ratio reached 1.238, inside the 1.25 limit, while mean mapped attention-output cosine reached 0.743. These diagnostics help locate which representations to improve in the next replay.

Line chart of attention-output cosine across 36 target layers. Direct cache injection stays near zero while mapped transfer is much higher across most layers and drops in several late layers.
Attention-output cosine by target layer on held-out evaluation prefixes. The map substantially improves on direct injection and highlights later layers for focused experiments.
Metric refinement

A legacy rollout heuristic labeled mapped and normal oracle text alike, so the next replay will use corrected rollout metrics. That gives the follow-up study a cleaner measure of multistep behavior.

Next experiments

Locate the remaining token divergence, then measure speed.

First, we will replay multiple mapper depths on fresh, disjoint prompts and compare multistep token agreement. This tests whether additional source context or a narrower map best preserves the target model's behavior.

Next, causal interventions will replace mapped keys, values, or selected target layers with oracle state. That experiment can show where the remaining divergence concentrates and whether partial target recompute is enough. Once fidelity reaches the target range, a synchronized latency study will measure the savings against ordinary 4B prefill.