8B Listening Quality at 1B Decode Speed

The result

Nearly double the accuracy at the same decode speed.

The fastest commercial voice models live in the ~1B-parameter class because decode speed dominates perceived latency. The problem is that ~1B models are not very smart. Native Ultravox 1B answered 17.2% of sealed spoken MMSU questions correctly. Native Ultravox 8B answered 53.1%, but decodes at less than half the speed.

Our transfer path reached 31.2%, a +14.1 point gain over the native 1B at effectively the same decode speed. Nothing about the 1B decoder changed: no fine-tuning, no distillation, no extra decode-time compute. It simply starts from a better memory.

Native 1B: 17.2% accuracy · ~145 tok/s
Mapped cache, no correction: 20.3% accuracy · ~145 tok/s
OneTriangle KV transfer: 31.2% accuracy · 133-150 tok/s
Native 8B (ceiling): 53.1% accuracy · ~64 tok/s

Paired per-question against native 1B, the transfer fixed 16 questions and regressed 7 (McNemar exact one-sided p = 0.047). It also produced zero malformed answers across the sealed test; the native 1B produced 15.

Evaluation discipline

The test set was sealed before any model saw it.

Small-sample audio evaluations are easy to fool. We pre-registered a 64-row spoken MMSU final by SHA-256 hash rule before training the candidate, froze one candidate by checksum after selection, and opened the sealed set exactly once. The evaluation ran end to end from raw audio: no cached activations, no retries, no second look.

Training and selection rows were provably disjoint from the final (verified by identity audit, zero overlap).
The frozen candidate and cache mapper are pinned by SHA-256 in the promotion record, and the serving bridge refuses to start if either hash changes.
All headline numbers re-derive from the raw per-question records.

How it works

Listen with 8B once. Decode every token with 1B.

01
Listen with 8B.
Ultravox 8B (Llama 3.1 backbone, 32 layers) processes the spoken question once and produces its KV cache in a rotation-free content space.
02
Map the cache.
A frozen ridge mapper converts the 32-layer 8B cache into the 1B geometry (16 layers, smaller heads). This is a single linear pass over the prefix.
03
Correct the cache.
A rank-16 residual MLP, trained through the frozen 1B decoder on 320 audio questions, repairs what the linear map loses. This correction carries most of the quality gain.
04
Answer with 1B.
The unmodified Ultravox 1B decoder generates from the corrected cache at its native 133-150 tokens per second.

The mapper and residual run once per request at the handoff. Decode-time cost is exactly native 1B.

Negative result, kept

The first candidate failed out of domain. Mixed training fixed it.

Our first residual trained only on spoken OpenBookQA. It beat native 1B on held-out OpenBookQA (50.7% vs 36.6%), then failed a pre-registered MMSU test (25.0% vs 29.7%). The correction had overfit its training domain.

We diagnosed before retrying: the 8B teacher itself scored ~54% on fresh MMSU whether quantized or not, so the ceiling was fine and the residual was the problem. Retraining the residual on a mixed corpus, half OpenBookQA and half fresh MMSU rows, produced the candidate that passed the new sealed final reported above, and it still edges native 1B on the original OpenBookQA holdout (38.0% vs 36.6%).

The takeaway: the cache correction is tiny (rank 16) and cheap to retrain, and its training data, not its capacity, is what determines how far it generalizes.

TTFT and throughput

The handoff costs about 100 milliseconds.

After warmup, the full handoff, 8B audio prefill plus cache mapping plus residual correction, measured 68-100 ms per request on the serving bridge. Decode then proceeds at 133-150 tokens per second depending on run: 133.0 tok/s on the sealed final, 141.8 tok/s natural and 150.0 tok/s forced-128 on the holdout cross-check.

Cache handoff (prefill + map + residual): ~0.07-0.10 s after warmup
Transfer decode speed: 133-150 tok/s
Native 8B decode speed: ~64 tok/s
Speed ratio at decode: ~2.2× the 8B

For a 128-token spoken answer, that is roughly 1.0 second end to end on the transfer path versus roughly 2.1 seconds for the native 8B, with much better answers than the 1B could produce alone.

It runs live

Try it in the inference platform.

The exact frozen candidate from the sealed test now serves live on two H100s behind our inference platform. Signed-in workspace members can open the Audio lab in the Playground, pick a held-out spoken question, and race all three arms, native 1B, KV transfer, and native 8B, on the same audio. Every response reports the answer, the cache-handoff time, and the measured decode speed, and the serving bridge verifies the residual and mapper checksums against the promotion record at startup.

Next experiments

Close more of the gap to the 8B ceiling.

Train the residual on broad conversational audio rather than multiple-choice tasks.
Scale residual training data past 320 rows; capacity is not the bottleneck yet.
Overlap the 8B prefill with 1B decoder setup to shrink the handoff further.
Serve the 8B listener to many concurrent 1B decoders, amortizing the listen step.
Measure long-conversation behavior where the transferred prefix is reused across turns.

Measured scope: the original candidate was specific to spoken multiple-choice QA and failed a blind-judged free-form test (2.6/5 vs 3.9/5 native). A follow-up campaign fixed this: refitting the cache mapper on conversational audio and distilling the residual against the 8B's own continuations produced a candidate that matches native-1B free-form quality (4.11 vs 4.11 blind-judged, zero refusals) while widening the multiple-choice gap to +20.3 points (37.5% vs 17.2%, p=0.012) on a fresh sealed MMSU final. That candidate now serves the Audio lab. Quality used a 64-row spoken MMSU final pre-registered by hash before training and opened once, plus a 71-row OpenBookQA holdout cross-check. Speed used the live serving path on two H100 80GB GPUs after warmup, with 16-token natural decodes and 128-token forced decodes. Model loading was excluded. The p = 0.047 is a McNemar exact one-sided test on 16 fixes versus 7 regressions.