The cost before the answer

Inference starts with a reading bill.

Long-context inference has two jobs. First, the model reads the prompt and builds an internal working memory. This is prefill. Then it generates the answer. The user sees the second job, but time and GPU cost start accumulating during the first.

Today, the model that answers usually does all of the reading too. That means a large model pays large-model rates before it has produced a single token. As prompts grow, this prefill step becomes one of the clearest places to make inference faster and cheaper.

Ordinary inference

Long promptLarge model readsLarge model answers

The most expensive model does both jobs.

Prefill-optimized inference

Long promptSmall model readsWorking memory movesLarge model answers

Use smaller compute for the first read, then preserve the work.

Our question was simple: can we separate who reads from who answers? If the small model's working memory can move, the large model does not need to start from a blank page.

The idea Make the first read cheaper without throwing the reading away.

The reading survived

The cache carried the substance of the prompt.

When a model reads, it builds a KV cache: the working state it uses to remember the prompt while generating. We transferred that state from a Qwen Instruct model to its Thinking sibling, then asked questions whose answers existed only in the original context.

What we learned

The target model had not read the text itself, but the important information was still there.

Prompt-memory comparison showing transferred Qwen caches retaining far more answer evidence than a model with no cached context.
Across 40 controlled cases, native prefill reached 99.4% mean answer evidence. The transferred caches reached 91.9%, versus 5.6% with no cached context. This was an exploratory memory diagnostic, not a final-answer quality benchmark.

After accounting for the no-context floor, the transferred state retained about 92% of the evidence available after native reading. That changed the problem. We no longer had to recreate every internal detail perfectly. We had to preserve the memory, then help the new model use it in its own way.

The reading survived. The next job was making the handoff feel native to the model receiving it.

Our best Llama transfer

On Llama, the long-context speedup became real.

We then tested a different family: Minitron 4B reading for Llama 3.1 8B. The best handoff was different from Qwen's, which reinforced an important product lesson: each model pair needs the correction that fits it.

This time the systems payoff was clear. At an 8K-token prompt, when the Minitron cache was already available, Llama reached its first token in 38.25 ms instead of 302.69 ms. That is a 7.91× faster target start. Even when we included the time Minitron spent reading from scratch, the complete path was still 1.25× faster.

Target start7.91×

Faster at 8K with the small-model cache already available

Full path1.25×

Faster at 8K including the small model's read

Continuation match82.52%

32-step top-1 agreement with native Llama behavior

Paired prompts54 / 64

Improved versus copying the cache directly

Llama handoff comparison showing the learned correction improving continuation agreement over direct cache reuse.
Our best Llama handoff improved 32-step agreement over copying the cache directly. The 82.52% figure is a scoped continuation metric, not general model quality.
Llama time-to-first-token by prompt length, showing native prefill getting slower as context grows while the learned handoff remains comparatively flat.
The prefill opportunity grows with context length. The 7.91× result assumes the small-model cache is already resident; the cold-context result at 8K was 1.25×.

The continuation check moved in the right direction too: the learned handoff reached 82.52% 32-step agreement and improved 54 of 64 paired prompts over copying the cache directly. This is a controlled continuation measure, not a claim about general task quality.

Llama made the economic reason concrete. The next question was how well the transferred behavior itself could be repaired.

Our best Qwen transfer

A small correction taught the second model how to continue.

Copying the Qwen cache directly preserved useful context, but the Thinking model did not use that state exactly as it used its own. We added a compact learned handoff with two intuitive jobs: translate the state into familiar coordinates, then correct the remaining difference in how the target model continued.

  1. 01
    Translate the memory

    Move the source cache into a form the target model recognizes.

  2. 02
    Smooth the continuation

    Correct the differences that still change the target model's next steps.

Best Qwen result

The learned handoff closed 41.4% of the gap left by copying the cache directly.

On the final untouched windows, the correction reduced Qwen's behavior difference by 11.5% beyond translation alone and closed 41.4% of the direct-to-native gap. The graph below comes from the earlier paired run, where the same idea helped 24 of 28 windows and helped most when the first translation still left the target furthest away.

Qwen transfer analysis showing the learned correction reducing target divergence and improving most paired evaluation windows.
Earlier paired Qwen evaluation: the correction improved 24 of 28 windows and helped most on the largest remaining mismatches. The later final confirmation measured an 11.5% reduction beyond translation alone.

The handoff made Qwen behave more like native prefill, but it did not make this pair faster. At 8K, the handoff took 256.7 ms versus 237.0 ms for native prefill. Qwen proved that the state could move and improve; Llama supplied the stronger cost result.

Together, the two families showed both halves of the opportunity: preserve the behavior and reduce the reading cost.

Where model routing fits

Prefill optimization is the core idea. Routing makes it adaptive.

The simplest version is a planned pipeline: a small model always reads, and a larger model answers. Model routing extends that idea. A small model can handle routine work, then pass its accumulated context to a stronger model only when the request becomes difficult.

That means the system is no longer choosing only who answers. It can also choose who reads, which saved state to reuse, and whether moving that state is worth it. For short prompts or weak model pairs, ordinary prefill remains the better path. For long prompts with a proven handoff, the router can avoid paying twice for the same reading.

01Prompt arrivesThe system sees the context before choosing the full path
02Small model readsBuild the working memory on lower-cost compute
03State movesUse the learned handoff validated for this pair
04Right model answersContinue from the work already completed
Lower-cost serving

Move long-context reading onto smaller compute while preserving the larger model for generation.

Model routing

Escalate difficult requests without making the stronger model reread the entire conversation.

Agent workflows

Carry accumulated context from routine turns into the model chosen for a difficult step.

Cache-aware scheduling

Prefer workers where the useful reading state is already available.

The takeaway

Spend large-model compute on the answer, not on rereading.

The story is bigger than routing. It is about breaking inference into stages and running each stage on the compute that fits it. Prefill does not have to be permanently attached to generation. If the reading can move, the system can make the first step cheaper and still bring the work forward.

Evidence boundary

The Qwen results measure controlled 32-step behavior and did not beat native prefill latency for that pair. The prompt-memory result is exploratory. The Llama 7.91× result assumes a resident source cache, while the full measured path was 1.25× faster at 8K. Its 82.52% figure is 32-step agreement, not general model quality, and broader task-level equivalence has not yet been established.

Let the small model do the reading. Preserve the work. Bring in the larger model when the answer needs it.