What does a KV cache actually store?
Suppose a small model has already read a long document. We now want a larger model to answer a question about it. The usual approach is to give the larger model the document and let it read everything again. Cache transfer asks whether we can carry some of the first model's work into the second.
To understand what has to survive that handoff, it helps to unpack the cache itself.
At each attention layer, a Transformer stores a key and a value for each earlier token. When processing a new token, it forms a query. The query is compared with the stored keys, producing attention weights. Those weights determine how much each value contributes to the attention output.
Here, q is one query, K and V contain the cached keys and values, and d is the key dimension. The softmax turns query–key scores into weights a that sum to one. The attention output h is the weighted combination of the values. This is one attention head, with masking and the output projection omitted for clarity.
A useful intuition is that keys help determine where to look, while values determine what gets read out. Neither is a literal index or a sentence stored in memory. They are learned vectors, built for the particular model that will use them.
That last point is what makes transfer difficult. A vector that works with one model's queries may behave differently with another model's queries. Even matching cache shapes does not make the representations interchangeable.
Why can a close cache give a different answer?
A natural first approach is to train a translator that makes the source cache look like the receiver's native cache. If the receiver would have produced keys K and values V, we want the translated versions, K̂ and V̂, to be close.
For example, we could minimize a squared reconstruction loss:
This objective treats reconstruction errors uniformly across entries. Attention does not use every entry uniformly. A value at a position receiving almost no attention contributes little to the current output. The same-sized error at a heavily attended position can matter much more.
Keys introduce another dependency. Their effect is mediated by the receiver's query and the softmax. An error in a key changes the weights used to combine values. Its importance therefore depends on the query and on which values the changed weights emphasize.
We can make this explicit. For a fixed query, write the native output as h = aV and the translated output as ĥ = âV̂. Adding and subtracting âV gives:
The first term is the change from reading with different attention weights. The second is the change from reading different values under the translated weights. HeteroFold uses this decomposition, with the receiver's output projection included, to motivate separate calibration objectives. See its output-aware calibration section.
The identity is exact for this local calculation. It does not mean that key and value errors contribute independently to final answer accuracy. In a full model, changed attention outputs also change later hidden states and queries.
How HeteroFold and KV-Lingo approach this
Both papers take the receiver's behavior seriously, but supervise the translation at different points in its computation.
HeteroFold: preserve how attention uses the cache
HeteroFold handles cross-family differences by aligning token positions and layers, then initializing a map that aligns cache statistics. Its key calibration targets native attention weights and their effect on outputs; its value calibration targets the full attention output under the translated weights. The learned corrections are folded into affine maps for inference; both language models stay frozen.
The important shift is where the training signal comes from: the receiver's use of the translated state.
KV-Lingo: preserve what the receiver predicts
KV-Lingo initializes linear translators with a least-squares cache fit, then refines them through self-distillation. During that second stage, the target processes the same continuation using either its native cache or the translated cache. Training minimizes KL divergence between those next-token distributions. Again, the source and target models stay frozen.
This gives the translator a downstream objective: produce cache states from which the target makes similar predictions.
My reading of the two methods is that they address a shared weakness of plain reconstruction from different levels: HeteroFold supervises attention behavior inside the model; KV-Lingo supervises predictions at its output. This is a comparison of their training ideas, not a controlled ranking of their performance.
Which part of the cache is losing useful information?
This leaves a practical question. If a translated cache gives worse answers, how much would fixing the keys help? How much would fixing the values help?
We tested this in our Qwen model-size pilot. The experiment used ten already-trained adapters covering both directions of adjacent sizes from 0.6B through 32B. For each direction, we evaluated the same 128 held-out questions under four conditions: fully native cache, fully translated cache, translated keys with native values, and native keys with translated values.
Here, native means computed by the receiver itself from the original input. Substituting a native component lets us examine a source of error while holding the other component fixed. It requires receiver prefill, so these substitutions are diagnostic controls. They do not give us a cheaper inference method.
Starting from the fully translated cache, replacing values raised average accuracy from 43.67% to 56.02%, a 12.34-point gain. Replacing keys raised it to 49.14%, a 5.47-point gain. The difference between those gains was 6.88 points, with a paired 95% bootstrap interval of 3.98 to 9.77 points.
For these adapters, the value substitution helped more on average. It produced the higher mixed-cache accuracy in seven directions, while the key substitution did so in three. That numerical split does not establish a significant difference in every direction.
The exception helps explain the result
The 0.6B → 1.7B direction behaved differently. Replacing keys increased accuracy from 44.53% to 61.72%. Replacing values reduced it to 41.41%. For this pair, the diagnostic points toward the keys.
There are interactions too. In the reverse 1.7B → 0.6B direction, the fully translated cache outperformed both mixed conditions. A component can work well alongside its translated counterpart without becoming better when the other component is replaced. The attention identity above does not turn the final accuracy losses into independent, additive pieces.
So the aggregate result gives us a direction to investigate, while the individual pairs tell us where that direction may fail.
What this experiment can tell us
This is an ablation of our short-budget dense-transfer pilot, not a direct ablation of HeteroFold or KV-Lingo. The adapters received 256 reconstruction updates and 32 supervised updates. The set contains 32 questions each from ARC-Challenge, MMLU-Redux, MedQA, and OpenBookQA; scores use the summed log probability of complete answer-label continuations rather than free-form answer generation.
There are 5,120 scored condition records, but only 128 unique questions. The bootstrap resamples question IDs together across directions and preserves the task counts. Its interval describes question uncertainty for these fixed adapters, not variation across training seeds or unseen model pairs. Native and fully translated baseline predictions replayed the original pilot exactly.
The next experiment
The result motivates a concrete hypothesis: on the directions showing a value-side deficit, a translator trained to preserve value contributions to attention may recover more answer quality than one trained only to reconstruct value tensors.
We could test that by fixing the key translator and the training budget, then comparing value-reconstruction training with an attention-output objective. Both would be evaluated on a fresh held-out set, alongside fully translated and native baselines. An improvement would need to survive without native components at inference.
That experiment has not been run. The current finding is narrower: in this pilot, access to native values repairs more of the average accuracy loss than access to native keys, and the size of that advantage depends on the model pair. It gives us a specific place to look when the next translated cache looks close but the answer is wrong.
Measurements, per-direction results, and source provenance · Experiment completed October 9, 2026.