Issues / #2707
#2707 Metal: --kv fp16 is accepted and reported, but the QSA cache always stores int8
open · @Boundlessmolson · 0 comentários · No GitHub
Server & APINVIDIA / CUDAModels & quants
Descrição
**Summary** On the Metal engine, `--kv fp16` is accepted and the serve banner reports `kv=fp16`, but the Metal QSA kernel always stores K and V as int8 with an fp16 scale per 64 values. The flag is silently ignored for the QSA cache. int8 looks like a deliberate choice for this kernel, so this is mainly about the mismatch: the banner says fp16 while the cache is int8. **Steps to reproduce** Any Metal run with `--kv fp16`, for example: ```sh env STRATA_METAL_STREAM=1 STRATA_ROUTE_TAIL_SKIP=0 STRATA_HC_Q8=0 STRATA_PREFILL_CPU_SHARE=0 \ STRATA_METAL_PROMPT=window \ ./build-metal/strata --serve \ --pack "$HOME/Developer/Strata-data/packs/q2_0" \ --native "$HOME/Developer/Strata-data/models/Q2_0/Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf" \ --ple-gguf "$HOME/Developer/Strata-data/models/Q2_0/Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00002-of-00002.gguf" \ --expert-profile data/expert-profile.bin \ --expert-cache 64 --prefill auto --spec 2 --suffix-draft 0 \ --max-context 4096 --mmap-experts --kv fp16 --adapt-every 0 ``` The banner reports: ``` INFO ... context=4096 kv=fp16 kv_resident=0 ... engine=0.1.44.1-metal ``` **Where it comes from (at 296b59d)** - `generate.cpp:9596` prints `kv=` from the flag string (`o.kv`). - `metal/kernels/qsa.metal:135-147` (in `qsa_prep`) quantizes every K and V append to int8 with an fp16 scale per 64 values (`kq`/`vq` are `device char*`, `ksc`/`vsc` are `device half*`). Nothing in the kernel reads `--kv`. - The file's own contract comments say the same: lines 6-7 (int8 KV append, fp16 scale per 64) and line 48 (the `kq/vq ... int8, ks/vs ... half` cache layout). **Why it matters** It changes numerics compared with CUDA's fp16 KV, which matters when comparing the two engines or chasing drift over long contexts. Someone passing `--kv fp16` to rule out KV quantization would reasonably believe they had. **Suggestion** Either honor `--kv fp16` in the Metal QSA path, or print what the Metal cache actually uses (for example `kv=int8 (Metal QSA)`) and warn when another type was requested. **Environment** - Strata `296b59d2f73c8f04a805df29c2fa97655adace78` (engine `0.1.44.1-metal`) - Mac Studio (Mac17,14), Apple M5 Max, 36 GB unified memory - macOS 27.0.1 (26A434) - Model: Qwen3.8-Flash-Next-GSQ-RCO Q2_0, native IQ pack, streamed experts
No site
Links install, modelos, releases.