Issues / #2787

#2787 [Feature request] 16-bit KV cache option above 8K context (to A/B against int8)

open · @mbaaku · 0 Kommentare · Auf GitHub

Models & quantsDocumentation

Beschreibung


**What:** an opt-in `--kv f16` (or `bf16`) for contexts above 8K. Today the choices there are `int8` (default), `k8v4` and `q4_0`; full precision is used only up to 8K.

**Why:** with Qwen3.8-Flash-Next Coder IQ1_M (v0.1.44.1, SYCL, Arc Pro B60, `--kv int8`, `--max-context 131072`) as the model of a coding agent, tool-call formatting degrades with prompt length:
- the first broken calls appear around 15–25k prompt tokens;
- at ~79k we saw a streak of calls with an empty body (`<function=Bash></function>`, no parameter), 48 in 40 minutes against 19 well-formed ones;
- at ~100k a reply looped in its reasoning until max tokens.

The broken calls are already in the raw model output (`STRATA_DEBUG=1`), so this is generation, not parsing. It may be the weight quantization alone, but the KV cache is the other quantized part that grows with context, and there is currently no way to rule it out. `docs/DETAILS.md` measures q4_0 against int8 (perplexity +8–12% on long documents); a 16-bit option would allow the same comparison one step up.

**What we would measure:** the share of malformed tool calls per 10k prompt tokens over the same agent workload with `int8` vs `f16`, from the `STRATA_DEBUG` raw output. Happy to post the numbers here.

**Cost we understand:** twice the KV memory of int8, so fewer experts in VRAM or a shorter context. It would be an opt-in for diagnosis and for cards with room to spare, not a new default.

Mehr auf der Site

Links zu Install, Modellen, Releases.