Issues / #2707

#2707 Metal: --kv fp16 is accepted and reported, but the QSA cache always stores int8

open · @Boundlessmolson · 0 commentaires · Sur GitHub

Server & APINVIDIA / CUDAModels & quants

Description

**Summary**
On the Metal engine, `--kv fp16` is accepted and the serve banner reports `kv=fp16`, but the Metal QSA
kernel always stores K and V as int8 with an fp16 scale per 64 values. The flag is silently ignored
for the QSA cache.

int8 looks like a deliberate choice for this kernel, so this is mainly about the mismatch: the banner
says fp16 while the cache is int8.

**Steps to reproduce**
Any Metal run with `--kv fp16`, for example:

```sh
env STRATA_METAL_STREAM=1 STRATA_ROUTE_TAIL_SKIP=0 STRATA_HC_Q8=0 STRATA_PREFILL_CPU_SHARE=0 \
  STRATA_METAL_PROMPT=window \
  ./build-metal/strata --serve \
  --pack "$HOME/Developer/Strata-data/packs/q2_0" \
  --native "$HOME/Developer/Strata-data/models/Q2_0/Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf" \
  --ple-gguf "$HOME/Developer/Strata-data/models/Q2_0/Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00002-of-00002.gguf" \
  --expert-profile data/expert-profile.bin \
  --expert-cache 64 --prefill auto --spec 2 --suffix-draft 0 \
  --max-context 4096 --mmap-experts --kv fp16 --adapt-every 0
```

The banner reports:
```
INFO ... context=4096 kv=fp16 kv_resident=0 ... engine=0.1.44.1-metal
```

**Where it comes from (at 296b59d)**
- `generate.cpp:9596` prints `kv=` from the flag string (`o.kv`).
- `metal/kernels/qsa.metal:135-147` (in `qsa_prep`) quantizes every K and V append to int8 with an
  fp16 scale per 64 values (`kq`/`vq` are `device char*`, `ksc`/`vsc` are `device half*`). Nothing in
  the kernel reads `--kv`.
- The file's own contract comments say the same: lines 6-7 (int8 KV append, fp16 scale per 64) and
  line 48 (the `kq/vq ... int8, ks/vs ... half` cache layout).

**Why it matters**
It changes numerics compared with CUDA's fp16 KV, which matters when comparing the two engines or
chasing drift over long contexts. Someone passing `--kv fp16` to rule out KV quantization would
reasonably believe they had.

**Suggestion**
Either honor `--kv fp16` in the Metal QSA path, or print what the Metal cache actually uses (for
example `kv=int8 (Metal QSA)`) and warn when another type was requested.

**Environment**
- Strata `296b59d2f73c8f04a805df29c2fa97655adace78` (engine `0.1.44.1-metal`)
- Mac Studio (Mac17,14), Apple M5 Max, 36 GB unified memory
- macOS 27.0.1 (26A434)
- Model: Qwen3.8-Flash-Next-GSQ-RCO Q2_0, native IQ pack, streamed experts

Sur le site

Liens install, modèles, releases.