The result

A useful point between the small and large models.

Native 8B is fast, but it answered 69.0% of the held-out audio questions correctly. Native 70B reached 90.1%, but generated at only 14.63 tokens per second.

Our OneTriangle KV cache-aware inference path reached 81.7% accuracy and 59.64 tokens per second. It recovered much of the quality gap while retaining 93.5% of native-8B decode throughput.

Native 8B
69.0% accuracy · 63.82 tokens/s · 2.04s response
OneTriangle KV cache-aware inference
81.7% accuracy · 59.64 tokens/s · 2.38s response
Native 70B
90.1% accuracy · 14.63 tokens/s · 8.85s response

Try it locally

Listen to verified examples or ask your own question.

The first tab replays real held-out examples. The second tab connects to the localhost-only two-H100 comparison bridge.

Audio comparison

Send one spoken question through all three paths.

Checking GPU

Water levels may decrease on cloudless days because…

  1. A. water is warmer than the air
  2. B. air is warmer than water
  3. C. moisture is pulled upwards
  4. D. moisture always tries to rise
Reference answer: C

Native 8B

Small model

Different
B

OneTriangle KV cache-aware inference

70B read → 8B answer

Correct
C

Native 70B

Large model

Correct
C

These nine examples are selected from the untouched final set because the three methods answered differently. The sealed 71-example totals are reported below.

How it works

Use the large model once. Use the small model for every new token.

  1. 01
    Listen with 70B.

    Ultravox 70B turns the spoken prompt into a rich KV cache.

  2. 02
    Translate the cache.

    A quantization-aware mapper converts selected 70B layers into the 8B cache format.

  3. 03
    Answer with 8B.

    The frozen 8B model generates from that cache at close to its native speed.

The mapper runs once at the handoff. It does not run again for each generated token.

Latency

Most first-token time is spent in 70B audio prefill.

Horizontal bar chart of median transferred time-to-first-token components. The 70B audio prefill takes 179.7 milliseconds, KV cache transfer takes 5.2 milliseconds, KV mapping takes 22.9 milliseconds, and the first 8B token takes 17.0 milliseconds.
Figure 2. Median time for each component of the 229.9 ms transferred TTFT. The 70B audio prefill is the dominant component.

Cache transfer and mapping take 28.1 milliseconds together. The 70B audio prefill takes 179.7 milliseconds.

Cost

OneTriangle KV cache-aware inference costs 73% less than native 70B.

Horizontal bar chart of estimated cost per million audio plus 128-token responses. Native 8B costs 2,369 dollars, OneTriangle KV cache-aware inference costs 5,540 dollars, and native 70B costs 20,607 dollars.
Figure 3. Estimated busy-time cost per one million audio plus 128-token responses at $4.19 per H100-hour.

The measured busy-time estimate is $5,540 per million audio plus 128-token responses. That is about 73% below native 70B. The estimate uses $4.19 per H100-hour and excludes loading, idle capacity, storage, and networking.

Future applications

Better voice systems without full-size generation.

  • Voice agents: stronger answers with near-8B generation speed.
  • Customer support: better understanding of spoken questions before a long response.
  • Audio search: richer interpretation of retrieved or transcribed evidence.
  • Adaptive routing: use native 8B for simple requests and OneTriangle KV cache-aware inference when quality matters more.
  • Shared context: reuse an enriched prefix across a conversation or a batch of related requests.

Next experiments

Make the large-model read cheaper and easier to reuse.

  • Cache and reuse shared audio or system-prefix representations.
  • Overlap source prefill, mapping, and target setup.
  • Test sparse and smaller teacher models for a faster first token.
  • Train on broader conversational audio, not only multiple-choice tasks.
  • Evaluate the live bridge under concurrent requests and longer conversations.

Measured scope: quality used 71 untouched VoiceBench OpenBookQA examples. Latency used one fixed sample, one warmup, five runs, and 128 generated tokens on two H100 80GB GPUs. Model loading was excluded.