A larger decoder can use work done by a different model family.

A language model spends part of every request reading the input before it can answer. That reading builds a key-value cache: the attention state it consults while generating. Usually, the model that writes the answer also builds the entire cache.

We split those jobs between Llama 3.1 8B and Qwen2.5 72B-Instruct. Llama read the context, a learned mapper translated its cache, and Qwen generated the answer. At approximately 7,500 context tokens, median complete-request latency was 581 ms, compared with 1,058 ms for Qwen's native path. That is a 1.82× speedup, or about 45% less time.

Native Qwen2.5 72B1,058 ms
Llama 8B → Qwen 72B581 ms
Complete sequential HTTP request latency at approximately 7,500 context tokens; lower is better. Includes source work, cache handoff, and answer generation with warm resident models. Excludes model loading. This is not streamed time to first token.

Both modes answered all their measured requests correctly. The test contains twelve distinct synthetic cases: four fact sets at three context lengths, each repeated three times through each mode. The resulting 72 requests are repeated measurements, not 72 independent questions. Two warmups were excluded.

Translate the memory, then continue decoding.

The transferred object is model state, not a written summary. The mapper converts Llama's attention keys and values into a cache the Qwen receiver can import. The receiver then processes the remaining context tail and question before generating its answer.

The validated setup used three H100 80 GB GPUs: one for the source model and two for the BF16 target decoder. The receiver ran inside the actual vLLM 0.31 container, using same-host CUDA IPC to exchange the mapped state. These details matter because moving a cache into a real decoder is a different test from evaluating a mapper in isolation.

The potential benefit is simple: a large model can spend less time rebuilding context that a smaller model has already processed. Whether that helps depends on the price of the translation and handoff.

The handoff earns its keep as the context grows.

At roughly 512 tokens, native Qwen finished in 126 ms and the mapped path took 172 ms. Transfer was slower. At 2,048 tokens, the comparison reversed: 333 ms native versus 242 ms mapped, a 1.37× speedup. By 7,500 tokens, it reached 1.82×.

This crossover is the useful finding. Translation has overhead; a short prompt may not avoid enough target-model work to repay it. These three measured lengths demonstrate that tradeoff for this implementation. They do not locate a universal threshold for other models or workloads.

A working handoff also has to fail cleanly.

Eight tests passed against the actual decoder container. Two failure checks were especially important: a nonfinite value introduced only on the second decoder rank, and an attempt to reuse an already-consumed cache.

Both checks failed the affected request, kept the service healthy, and allowed the next native request to succeed. Both decoder ranks released their imported allocations. That is a concrete step toward a usable serving path: the receiver can reject bad state and recover.

A serving result with a deliberately narrow quality claim.

The questions test synthetic box-color recall. Perfect accuracy here does not establish general chat, coding, reasoning, or customer-workload quality. Requests ran sequentially with warm resident models; this is not a concurrency, throughput, cold-start, or billing benchmark.

The comparison is with this implementation's native target path on the same hardware, not another provider's engine. GPU validation is complete for this image pair; public registry publication and a live provider deployment were still pending in the October 10 report.

The result gives us a concrete next test: preserve the measured latency gain while expanding the quality evaluation to realistic documents and customer tasks.

Measurement record: October 10 model-image validation, Qwen72B final v1. Read the measurement summary and provenance.