#4141 [Engine] metal: fix wrong prompt results with K-quant packs: the gate/up grid follows the tile (carries #2903)

closed · draft · @macintog · 0 commentaires · Sur GitHub

BenchmarksServer & APIModels & quantsSecurityDocumentationWindows

Description

Opened as a draft: validated on an M4 Max only. The M5 path needs a run I can't do (see Not run).

## Problem and result

This is a critical fix for output quality and parity. On an M4 Max with a K-quant pack, Strata's default prompt path
gives wrong results, with no error or warning. With the fix, on the positions measured below, it agrees with llama.cpp
as closely as Strata's own exact path does.

KL and top-1 are measured against llama.cpp (f16 KV) on the same token ids, over 511 positions after a 32K-token
context. Prompt throughput is measured separately, on 7K-42K-token prompts (the exact fallback on 7K and 15K only).

| prompt path | KL (top-64) | top-1 | prompt tok/s |
|---|---|---|---|
| prompt chain, `main` (the default) | 0.307 | 81.4% | 688-704 |
| exact fallback, `STRATA_METAL_PROMPT=window` | 0.0031 | 98.0% | 78 |
| **prompt chain, this PR** | **0.0033** | **98.4%** | **524-546** |

The exact fallback reads the prompt through the decode windows and doesn't use the code this PR changes; I measured
it on this branch's build. Without this fix, the accurate path the docs offer on such a pack is that fallback, which
`docs/MAC.md` lists for testing at "about 190 tok/s". On this Mac it reads 78. This PR makes the default chain as
accurate as the fallback at about 7× its speed (6.7× at 7K prompt tokens, 7.0× at 15K). The chain's prompt
throughput is about 23% lower than before, because it now does the work it had been skipping.

**What goes wrong.** When the pack's routed gate/up experts are K-quants (Q4_K, Q5_K, Q6_K), the prompt chain
computes only a quarter of each expert's gate/up output. The other three quarters are never written, and the down
projection reads them anyway. The prompt's hidden states come out wrong, and every token generated afterwards reads
the attention keys, values and recurrent state they produced.

**Why.** On a Mac without the M5 matrix-unit layout, `gu2_tile()` gives K-quants the tile `_16_16_128_2`, which
writes 16 output columns per threadgroup (`f0 = tg.x * TN` in `mm_gu2`). The host sets the gate/up grid once per
chunk to the constant `NFF / 64`, which is right for the 64-column tiles every other type uses. With `NFF` 640 that
launches 10 threadgroups where 40 are needed, so columns 0-159 are written and 160-639 are not.

This PR carries #2903 by @ctr-137, which fixed exactly this. That PR was closed in the 0.2.0-beta tracker reset. Its
author reported validating it on Windows only, with the host test. I rebased it onto `main` (`4b9e05a2`) and ran it on
an M4 Max with Unsloth's Qwen3.8-Flash-Next UD-Q4_K_XL.

## Related discussion

#2903: the same defect and fix, closed in the reset ("Closed here does not mean rejected"). I'm opening a new PR
rather than continuing that one because only its author or a maintainer can `/reopen` it, and its branch is the
author's, so the rebase can't go there. Its commit is kept with its original author, and a comment on #2903 links
here. If you'd rather reopen #2903 and take the rebase there, I'll close this one.

The 10-21 M4 Max comment on #2903 reports parity on the same tile from a separately written patch, not from #2903's
code.

## Approach

From #2903, unchanged apart from the rebase:

- `metal/src/moe_tiles.h` (new) returns each v2 tile's name together with the TN it writes, so the two can't drift
  apart, and holds the grid rule (`moe_grid_x`).
- `mm_scan` writes the gate/up and down grids (`meta[2]`, `meta[5]`) per MoE layer from the plan arguments, as it
  already writes that layer's tile count. A tile chosen per layer needs a grid computed per layer.
- `window.cpp` (streaming) and `prompt_chain.inc` (resident) pass the grid of the tile each layer dispatches. The
  per-chunk host writes are gone.
- `moe_mm_parity` and `prompt_chain` (the test harnesses) take their v2 tiles from the engine's `gu2_tile` and
  `dn2_tile`. Before, each had its own v2 tile rule (TN 64 for every type), so their default runs never used the
  K-quant tile the engine dispatches.

Small commits of mine on top:

- **The grid follows `b.v3e`.** Since 672fd99c the resident chain dispatches the v3 tiles only when `b.v3e` is set
  (the trellis types keep v2 on an M5). #2903's plan still chose the grid on `b.k4`, so on an M5 with a trellis pack
  it would size the grid for a v3 tile while the v2 tile ran. Both lines now test `b.v3e`, the condition the dispatch
  below them uses. The other `b.k4` tests in `pc_chunk` select attention kernels and stay as they are.
- **`moe_mm_parity --v2` keeps an explicit `--dtile`.** The new default overwrote it whenever `--tile` was absent.
- **`docs/MAC.md` gives the exact fallback's speed on this Mac** (78 tok/s) beside the existing "about 190", whose
  machine isn't recorded.

The rebase had one conflict: the host's grid writes in `pc_setup`, which this PR deletes.

Default path: the grids for the TN-64 tiles (10 gate/up, 40 down at this pack's sizes) and for the v3 arm (5 and 20)
come out as before. Only v2 K-quant gate/up layers change, from 10 to 40.

## Validation

Apple M4 Max (Mac16,5, 128 GB), macOS 27.2 (26B118), Apple clang, `-DCMAKE_BUILD_TYPE=Release`. Two builds: `main` at
`4b9e05a2`, and this branch at `39ca9d99`. The engine runs below used `430a8440`; the two later commits change only
`moe_mm_parity.cpp` and `docs/MAC.md`, so the engine binary is the same.

Model: Unsloth's Qwen3.8-Flash-Next UD-Q4_K_XL. Its routed gate/up experts are Q4_K in 47 layers and Q5_K in one, so
every layer takes the 16-column tile.

**Passed:**

1. `ctest` (host test `moe_tile_grid`): 39 checks pass. As a negative control I compiled the same test against a copy
   of `moe_tiles.h` with the K-quant TN set back to 64: 6 checks fail and it exits 1, as expected. (#2903 reported 4
   failing checks with MSVC.)
2. `moe_mm_parity --v2 --gu <type> --dn Q8_0` for Q4_K, Q5_K and Q6_K. The harness runs the engine's tile (TM 16,
   TN 16, TK 128, SG 2) and all three pass:

   ```
   Q4_K: parity: plan ok; y vs fp16-operand reference rel L2 2.61e-05, vs exact float reference 4.94e-04  PASS
   Q5_K: parity: plan ok; y vs fp16-operand reference rel L2 2.43e-05, vs exact float reference 4.89e-04  PASS
   Q6_K: parity: plan ok; y vs fp16-operand reference rel L2 2.65e-05, vs exact float reference 4.94e-04  PASS
   ```

   This checks the kernel's arithmetic on that tile. The harness sets its own grid, so it doesn't exercise the
   engine's indirect grid; item 4 does.
3. `prompt_chain <the pack's Metal GGUF> --prompt 2048` (a timing harness on the real weights, random tokens): runs,
   722.0 and 719.7 tok/s GPU time in two logged runs.
4. Accuracy of `strata` itself against llama.cpp on identical token ids, using upstream's `STRATA_LOGPOS` rows and
   the `GEN` line's `pin=` (both are in `main`). The model's arguments and the run's settings:

   ```
   STRATA_METAL_STREAM=0 STRATA_ROUTE_TAIL_SKIP=0 STRATA_LOGPOS=rows.tsv STRATA_LOGPOS_TOPK=64 \
   strata --serve --pack <pack dir> --native <pack>-metal.gguf --native-dense-gguf <pack>-metal.gguf \
       --ple-gguf <UD-Q4_K_XL shard 2> --expert-profile <expert-profile.bin> --expert-cache 64 --prefill auto \
       --spec 2 --mtp <mtp dir> --mmap-experts \
       --max-context <n + 512> --short-read 511 --prompt-cache 3 --turn-token -1 --adapt-every 100000 --suffix-draft 0
   GEN 1 pin=<n - 512> <n token ids, comma-separated>          (sent on stdin, one request per line)
   ```

   With `pin`, the prefix `[0, pin)` is read as one batched prompt (the prompt chain, the path this PR changes), and
   the last 511 positions go through the decode windows and write one row each. Any damage the prompt chain did to
   K/V and recurrent state shows up in those rows. The exact-fallback column adds `STRATA_METAL_PROMPT=window`, so
   the prefix goes through the decode windows too. There are two requests, which score the same 511 text positions:
   - **long:** the full 32,768 ids, `pin` 32,256;
   - **control:** the last 1,024 of those ids, `pin` 512. Its rows are numbered from its own start, so add 31,744 to
     match them to the long request's positions.

   The reference is llama.cpp (Homebrew libllama 0.7.0, ggml 0.27.0, Metal, f16 KV). It evaluates the same ids
   as one context per request (32,767 tokens for long, 1,023 for control) and keeps the full-vocabulary
   log-probabilities at those 511 positions. I used a small libllama program that keeps logits only at the chosen
   positions; `llama-perplexity --kl-divergence-base` gives the same rows at more memory. Both sides are cut to the
   reference's top 64 tokens and scored with `tools/quant_quality.py`'s `compare`. I can attach that program and
   the scoring scripts (about 750 lines together) if they'd help.

   KL / top-1 / PPL:

   | context before the scored positions | `main` | exact fallback (this branch) | this branch | llama.cpp PPL |
   |---|---|---|---|---|
   | 512-1,022 tokens | 0.253 / 79.6% / 3.849 | 0.0108 / 95.5% / 3.694 | 0.0118 / 95.7% / 3.692 | 3.695 |
   | 32,256-32,766 tokens | 0.307 / 81.4% / 2.168 | 0.0031 / 98.0% / 1.867 | 0.0033 / 98.4% / 1.876 | 1.875 |

   The exact fallback doesn't use the code this PR changes, so I ran it on this branch only. The text is a chat
   transcript (a code-review request) that I can't share; any long public text works with the commands above. For
   scale only: two fresh llama.cpp f16 runs of this text differ by KL 0.0034, but that was full-vocabulary KL over
   2,048 positions between 16K and 32K, so it isn't directly comparable to the top-64 figures.
5. Prompt speed: the OpenAI server (`serve/server.py --engine strata`, the same front end for both builds), starting
   `strata` with the model arguments and `STRATA_METAL_STREAM=0` from item 4, without the `STRATA_LOGPOS` and run
   settings, plus `--max-context 262144`. Each prompt is one chat request with `max_tokens` 1, after one warm-up
   request, timed by the server's `timings.prompt_ms`:

   ```
   POST /v1/chat/completions  {"model": "...", "messages": [{"role": "user", "content": "<prompt>"}],
                               "max_tokens": 1, "temperature": 0.6}
   ```

   Arms ran in the order main, branch, main, branch, then the exact fallback once on this branch:

   | prompt tokens | `main` (s) | this branch (s) | exact fallback, this branch (s) |
   |---|---|---|---|
   | 7,203 | 12.3 / 10.4 | 13.8 / 13.7 | 92.6 |
   | 15,387 | 22.1 / 22.1 | 28.9 / 28.2 | 198.3 |
   | 42,373 | 61.6 / 60.2 | 80.0 / 79.7 | not run |

   `main` reads 688-704 tok/s, leaving out its first measured request (588 tok/s, 7K). This branch reads 524-546
   tok/s, about 23% lower throughput. The exact fallback reads 77.6-77.8 tok/s. Token generation doesn't use this
   code.

**Failed:**

- On `main` only, the baseline: `moe_mm_parity --v2` for Q4_K, Q5_K and Q6_K gate/up exits 3 before running. For
  Q4_K it reports `mtl: no function mm_gu2_12_16_64_64_2` (Q5_K and Q6_K name types 13 and 14): its own TN-64 rule
  asks for a tile that doesn't exist. `prompt_chain` on this pack exits 3 the same way (`no function
  mm_gu2_12_16_64_16_2`).
- On both builds: `moe_mm_parity --v2 --gu Q4_K --dn Q6_K` crashes (exit 139). `N_FF` 640 isn't a multiple of the
  256-wide K-quant block, and `ref::random_blocks` writes past the buffer that `row_bytes` sizes. No real pack uses
  a K-quant down projection at this width. This PR doesn't change it.

**Not run:**

- **M5 (unconfirmed static analysis).** I have no M5, so the v3 arm and the `b.v3e` change on a trellis pack are
  checked only by reading the code. The `b.v3e` change needs item 4 run with a trellis pack on an M5. That path
  takes the v2 tiles, so the harnesses' `--v3` runs don't cover it. Item 4 with a non-trellis pack on an M5 would
  cover the v3 arm.
- **M1-M3, and other K-quant packs** (for example UD-Q6_K_XL, Q6_K gate/up in 47 of 48 layers). They take the same
  tile and code path on these Macs, but I haven't run them.
- **The streaming route** (`STRATA_METAL_STREAM=1`, the `win

Sur le site

Liens install, modèles, releases.