#5918 [Engine] Metal: honour --kv fp16 with direct-half KV kernels

open · @catalinb13 · 0 评论 · 去 GitHub 看

BenchmarksSetup & installNVIDIA / CUDAModels & quants

说明

`--kv fp16` was silently ignored on the Mac. The window decided the cache layout from `st.k_pool != nullptr`, and Metal's `kv_plan` gives a session state **no pools at all** (`slots == 0`), so that test never held: the engine ran the 8-bit codes whatever the flag said. Only the generic CUDA path had an fp16 cache, and Metal never takes it. So a Mac user who asked for an unquantized cache got 8-bit, and nothing said so.

This adds the missing kernels and wiring so `--kv fp16` does what it says, and fixes four faults found while building it. Metal's default stays 8-bit: the released default path is unchanged and fp16 is the opt-in.

## Result

The cache can now hold the checkpoint's exact values, and it costs what that costs:

| | 8-bit (default) | `--kv fp16` |
|---|---|---|
| decode tok/s | 22.8 | 22.5 |
| prefill tok/s | 26.4 | 24.4 |
| resident | 51.4 GB | 53.9 GB |

M1 Max 64 GB, UD-IQ4_XS, arena 32 GB, 204800 context, 357-token prompt, 192 new tokens, `--spec 2` + MTP. **So this is an exactness option, not a speed option** — a half is two bytes where a code is one. At a 4K prompt the two are level again (a decode window is 60.9 ms against 60.3 ms).

Quality is unchanged on the repo's own token-forcing gate: 12.76 % of forced tokens score a different argmax, against 12.02 % and 12.26 % for two 8-bit stock arms — inside those arms' own spread, and no large-margin event (the event rule passes; the rate rule is INCONCLUSIVE at that sample size, which is 1215 forced tokens).

## Approach

Three kernel variants keep K/V as plain halves with no scales, `[cell][kv head][head_dim]`:

- `qsa_prep_fp16` — the append.
- `qsa_attn_fp16` — decode attention; it keeps the split-K partials, so `qsa_merge` is unchanged.
- `qsa_flash_fp16` — the prompt's matrix-unit path.

The window allocates twice the K/V bytes, binds the session state as the generic fp16 layout (`k_pool`/`v_pool` set, `kv_int8` false), and dispatches the variants in the decode path, the prompt path, the batch slot views and `slot_copy`. `bind_draft` stays 8-bit on purpose: the draft layer's dispatch is the literal `qsa_prep`/`qsa_attn` pair over its own buffers.

Four faults, all fixed here:

1. **Buffer binding is positional.** `mtl_rt.mm` binds `d.args[i]` at index `i`, so the fp16 kernels cannot keep the 8-bit scale-buffer holes at 14/15 and 5/6 — the argument after a hole is bound to `QsaArgs`. The fp16 kernels therefore number their buffers contiguously.
2. **`half*` arithmetic counts elements, not bytes.** The decode K load read at twice the offset, and its second `half4` advanced eight elements instead of four. This one is worth calling out because it produced coherent text at normal speed: only the token-forcing gate saw it, as a 36.5 % forced-token mismatch against 8.6–9.4 % for stock. (It also inflated an earlier speed measurement, which is why the numbers above are post-fix.)
3. **The format test itself.** Detection reads the state's format flags, not `k_pool`.
4. **The window now prints `QSA K/V cache in FP16 …`** when the mode is on, so a benchmark log says which layout ran. Without it, an "fp16" arm that silently stayed 8-bit looks identical to a real one.

## The launch path

`strata launch` already carries `--kv` and forwards an explicit value to the engine (`runner.py`: `if s.kv: args += ["--kv", s.kv]`), and it passes **no** `--kv` for a Mac, so the engine's default is what a plain `strata launch` runs — which is why the default change above is enough to keep that path as it was. Checked with `strata launch --plan`:

- `--plan --kv fp16` → the command line carries `--kv fp16`.
- `--plan` with no `--kv` → no `--kv` at all, so the engine default applies.

Two of the launch layer's own assumptions still described the old behaviour, and are fixed here:

- the `--kv` help said `int8 (default)`, which is only true on a Mac;
- `desc_matches` / `apply_choices` fell back to `"int8" if context > 8192 else "fp16"` when nobody named a format, so a Mac launch was sized and compared as if it would take fp16 at a small context. Both now ask the config's `backend` through a small `default_kv(backend, context)` helper, so a Mac resolves to int8 at any context and the card builds are unchanged.

`pytest tests/terminal` on the launch files: 46 passed, 3 failed. The three fail identically without these edits (this machine has no `macos-arm64` Terminal in the release folder, and the DeepSeek drafting line is `off` here), so they are pre-existing and not from this change.


## Validation

After the rebase onto `c77572806` (upstream main, 2026-11-18): one conflict, `strata_cli/launch/cmd.py`'s `--kv` help text, resolved to upstream's `--context` line plus this PR's `--kv` wording; `ninja -C build-metal strata` links clean.

Passed:

- Rebased tree (M1 Max 64 GB, macOS): `ninja -C build-metal strata` clean.
- Rebased tree: `python -m pytest tests/terminal/test_launch.py tests/terminal/test_launch_precedence_0213.py tests/terminal/launch_support.py` — 130 passed, 1 failed. The failure, `test_the_apps_own_installer_arguments` (it expects `--kv q4_0` in the app's installer arguments), reproduces identically in a clean `upstream/main` worktree, so it is pre-existing and not from this change.
- `build-metal/qsa_parity` — ALL PASS.
- `build-metal/qsa_prompt_parity` — ALL PASS.
- `ninja -C build-metal` — clean on this branch.
- An engine run with `--kv fp16` on this branch prints the mode line and decodes 48 tokens without a crash. (Run at arena 20 GB to keep it clear of the machine's memory ceiling, so its tok/s is not meaningful.)
- `tools/force_check.py` token forcing, as above: rate rule INCONCLUSIVE, **event rule PASS**, zero large-margin events.
- On the branch this was developed on (which carries the window-backed conversation park), `tools/conversation_cache_http_smoke.py` prints `PASS: HTTP reuse, slot eviction, cancellation/recovery`, and its records show `reused=1987/1994` and `reused=1085/1092` on return visits — park and restore under fp16.

Not run on this branch:

- A full-context (204800) speed and memory measurement. The numbers in the table come from the development branch, whose engine differs by 106 commits that are not in this PR; they should be re-taken here before the numbers are quoted.
- The conversation-park check: `window_pools` does not exist on `main`, so there is nothing to exercise. The park fix lives with the branch that adds it, not here.

## Limits

- `--kv q4_0` and `--kv k8v4` still have no Mac kernel and still run the 8-bit path.
- fp16 doubles the KV bytes, so it is the wrong choice when memory is the constraint — 8-bit stays the default for that reason.

Follows `CONTRIBUTING.md`: one problem, the default path unchanged, the measurements marked with the machine and settings they came from, and the failed/not-run checks kept separate from the passing ones.

本站相关内容

相关页面的快捷入口。