OneTriangle
Book a demo
← Research notes

Results updated on Oct 27

KV cache transfer benchmarks

Analysis
  • tl;dr: a translated KV cache carries most of the quality and almost none of the prefill cost. Qwen3.5-4B → Qwen3.8-27B scores 0.84 on QuALITY vs 0.94 native 27B, and GLM-4.7-Flash → GLM-5.3 gets 0.98 on ARC and 0.89 on ZebraLogic vs 0.99 and 0.93 native. On timing, Qwen2.5 3B → 72B turns a projected ~76-minute native 1M-token prefill into ~9 minutes, and Ministral-3B → Mistral-Medium-128B turns ~2 hours into ~11 minutes. The open weakness is exact string recall (Gemma 4 is the hardest case), and the details per pair are in the tabs above.
  • Task-specific mapping materially outperforms the generic mapper when the task distribution is known. On the Qwen3 1.7B to 8B transfer, the task-specific map reaches 0.79 on ZebraLogic versus 0.185 for the generic map, 0.78 versus 0.52 on LongHealth, and also improves QuALITY, ContractNLI, and BAMBOO PaperQA. The generic map remains useful as a zero-task-data baseline, while a small amount of task data closes much more of the receiver gap.
  • ZebraLogic is unusually sensitive to cache-transfer errors because each answer depends on preserving an exact chain of interlocking constraints. The generic mapper was trained on FineWeb-Edu rather than ZebraLogic, so small errors in the translated relational state can invalidate the whole solution. That explanation fits the result: generic transfer scores 0.185, while the ZebraLogic-specific map recovers to 0.79, matching the native 8B model.
  • Cache transfer can scale between models even when the source and receiver are not architecturally identical. Qwen3.5 to Qwen3.8 is a strong example: the mapped cache carries useful state across model generations and sizes. Gemma 4 is a harder case and shows where architecture differences still create large losses, especially on exact recall, but it also gives us a concrete target for better alignment methods.
  • The timing benefit becomes larger as context grows. At one million tokens, the current projections put Qwen2.5 72B native prefill at about 76 minutes versus about 9 minutes for transfer, and Mistral-Medium native prefill at about two hours versus about 11 minutes for transfer. The important scaling property is that mapping and cache movement grow much more gently than large-model prefill.
  • A partner could use this as a serving primitive for reading long contexts with a smaller model and handing the resulting cache to the model they already serve. With access to their GPUs, we would train and evaluate maps for their model pairs, measure long-context and concurrency behavior on their stack, optimize cache movement, and replace the remaining projections with production-scale measurements.
  • Exact-recall benchmarks are the clearest weakness: translated caches do worse when the score requires reproducing a hidden string exactly. This is a deliberately harsh test, and most real workloads care more about semantic correctness, reasoning, and downstream task performance than verbatim copying. We still show the result because improving exact recall would make the method useful for an even broader set of retrieval workloads.
  • We are exploring speculative decoding in the opposite direction: transferring state from a larger model into a smaller model, then letting the smaller model generate. The goal is to increase tokens per second by decoding with the cheaper model while giving it better output quality than it achieves from its own context processing alone. Cache transfer lets the large model contribute stronger internal representations without paying its full autoregressive decoding cost. This direction is especially relevant to real-time systems; see the Ultravox research blog.

Thank you to Lily Sun for all your help on these benchmarks.

How to read these benchmarks

We take the KV cache that a small model built while reading a long context, translate it with a learned map, and hand it to a bigger model. The bigger model then answers as if it had read the context itself, without paying for its own prefill. These pages measure how much quality survives that handoff, and how much time it saves.

The three bars

Every benchmark shows three numbers. The white bar is the small model on its own: the floor. The dark bar is the big model reading the context natively: the ceiling, and the thing we are trying to avoid paying for. The colored bar in the middle is the transfer. The closer the colored bar gets to the dark one, the less quality the translation loses. When the colored bar beats the white one, the big model is adding reasoning on top of the small model's reading.

The thin gray line is chance. The dashed line is the big model answering with no context at all. If a bar doesn't clear those, the context isn't doing anything.

Quality vs context length

Same idea, stretched over input length. Solid points are the transfer, with error bars and the number of items in each bucket. Dashed is the big model native, dotted is the small model native. The vertical line marks the longest context the map was trained on: past it, you're watching the map generalize to lengths it has never seen. Quality holding up past that line is the interesting part. Exact-recall tasks like MRCR are the hardest case: small reconstruction noise that barely moves a multiple-choice answer will destroy a verbatim copy. We show those anyway.

TTFT

Time to first token. The stacked bar is everything the transfer pays: small-model prefill, the map itself, moving the cache, and the first token out of the big model. The black tick is what the big model would have paid to prefill natively. The gap between them is the point of all this. Bars with dashed borders are projections from fitted curves, not measurements; we label which is which because the difference matters.

n is small on some buckets. Hover anything for exact numbers. If something looks off or you have a question, email medha@onetriangle.ai