#3352 [Engine] Feature request: FP8 (E4M3) KV storage (--kv fp8) — same 1 B/value as int8, better on heavy-tailed K/V, rides the FP4/FP8 tensor cores we already build for sm_120a

本文

The main path's KV menu today is `fp16 | int8 | q4_0 | k8v4` (`generate.cpp:2968`). We run `--kv int8` in production (RTX PRO 6000, sm_120, NVFP4 Flash-Next pack, 768K yarn3) and it works well; the ask is an FP8 (E4M3) option between int8 and k8v4:

1. **Precision shape.** int8 codes with an fp16 scale per 64 values put a fixed step on a heavy-tailed distribution — the outliers that matter most for attention scores are exactly what the clip and the uniform step hurt. E4M3's floating mantissa spends resolution where the values are, and llama.cpp's `f8_0` KV community experience is that it beats int8 at equal 1 B/value on outlier-heavy models.
2. **The hardware is already in the build.** The engine ships an FP4-MMA unit for sm_120a (`strata_mmq_w4a4`), so the card's narrow-float tensor path is not a new dependency the way it was for Arc.
3. **Honest note on the earlier rejection.** `docs/A770_SPEED_SPEC.md` lists fp8 among the KV formats "non-transferable from Infernix ... recorded so they are not retried". We read that as Arc-specific (no FP8 MMA there, and the QSA streaming layout is int8-first). If the reasoning was general rather than Arc-specific, we'd like to know; if it was Arc-specific, then on NVIDIA an FP8 QSA cell is ~1,056 B (same as int8) with a better error profile, and k8v4-style hybrids (`fp8 K + q4_0 V`) could go below 816 B/cell.

We can measure it: KL against fp16 KV on real long-document traffic (150K-600K prompts), decode/prefill tok/s, and the KV-streaming hit rate at `--kv-resident 65536`, on sm_120 — same protocol as our w4a8/w4a4 A/B. Happy to run a build behind an opt-in `--kv fp8` and post the numbers.