#6030 [Engine] CLI path: a native (IQ) pack's residual is never seeded, so output is pinned to token 0 (verified on sm_75 and sm_89; standard GGUF packs are unaffected)
open · @an94-code · 7 评论 · 去 GitHub 看
BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quantsWindows
说明
### What happened
On a Turing card (RTX 2080 Ti 22 GB, sm_75, eGPU over PCIe x4), a 120B MoE Q3_K pack loads and runs: the CUDA kernels are present (`GPU 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5`, no `no kernel image` / `invalid device function`), experts are read (`experts streamed 173 (91 by DMA), resident 307`), and prefill is healthy — `STRATA_DBG_NAN=1` reports
`prompt end: last residual row 0 non-finite (max |x| 4.71); PLE history max |x| 11.3`.
But every generation returns zeros: `output : 0 0 0 0 0 0 0 0`, argmax always token 0.
`STRATA_VERIFY_PROFILE=1` localises it:
```
verify top3: win=0 T=1 -> 0(0.000) 1(0.000) 2(0.000) rms=0.0000
verify top3: win=1 T=4 -> 0(0.000) 1(0.000) 2(0.000) rms=0.0000
```
Following `verify.cpp:2727-2747`, that rms is computed on `head_logits_`, produced by `native_mmvq(head_->type(), …)` with `head_->type() == Q5_K`. So **the native Q5_K head is the broken stage**; the trunk is fine.
`STRATA_LOGITS_DUMP` shows what the head actually produces — **not all zeros, but garbage in a very specific shape** (verbatim, 5 rows of 248320):
```
min: 0.0 max: 856028872704.0 mean: 1378899.1613 argmax: 0
top5: [(0, 856028872704.0), (248326, 856028872704.0), (2, 0.0), (248328, 0.0), (248327, 0.0)]
rows of 248320: 5
row 0: max=856028872704.000 nonzero=4/248320
row 1: max=856028872704.000 nonzero=5/248320
row 2: max=0.000 nonzero=0/248320
row 3: max=0.000 nonzero=0/248320
```
Reading it:
- **Only 9 of 1,241,600 entries are non-zero** (4 in row 0, 5 in row 1); rows 2–4 are entirely zero.
- The non-zero value is **856028872704.0** — identical in both rows. That is not "a large logit", it looks like a constant written into a few slots, with no plausible scale.
- argmax therefore lands on index 0 every time.
That is the signature of a write-back that covers only a couple of output rows: most of the `head_logits_` buffer is never written (stays 0), and the few slots that are written hold a constant. `rms=0.0000` above is consistent — it is the RMS over all entries, 99.999% of which are 0.
The head weights are loaded: `experimental native Q5_K head, 675430400 bytes, in 0.5 s` (644 MiB) — so this is a compute result, not a missing upload. `q5_q8_dot` in `native_mmvq.cu` uses `STRATA_DP4A` (`__dp4a`, native on sm_75) and I found no arch branch in that kernel.
### There is no CLI way to bypass the native head
- `generate.cpp:3948`: `if (native_head_gguf.empty()) native_head_gguf = native_preset;` — the head inherits `--native`.
- The native pack requires `--native`.
- Commenting out that line and building only `--target strata` succeeds, but the binary then stops with
`output.weight: this pack holds the tensor only in its GGUF form (run with --native SHARD1)`.
- `generate.cpp:4144` also requires `--stream-token`; dropping it exits immediately.
⇒ For this pack the native Q5_K head is the only path, and it is the broken one.
### Excluded (each changed one variable; output was still all zeros)
`STRATA_PREFILL_MMQ=0`, `STRATA_Q8_PACKED=0`, `STRATA_SM75_IL=1`, `STRATA_SM75_PERF=1`, `STRATA_MMVQ_IL=0`, `STRATA_OLD_IQ_MMVQ=1`, `STRATA_GDN_TC=0`, `STRATA_GDN_TC=2`, `STRATA_LAT_FUSED_Q3K=0`, `STRATA_LATENT_FUSED=0`, `STRATA_LATENT_FUSED_DU=0`, `STRATA_LATENT_U_GPU=0` (path clearly changed: 10.5 tok/s), `STRATA_HEAD_MIX_MULTI=0`, `STRATA_FUSE_HEAD_GR=0`, `--kv fp16`, `--prefill 64`, no MTP, `--spec 2` (`--spec 1` is refused: `native (IQ) pack: it needs --spec T (T >= 2)`), `--expert-cache-cpu-order`.
Also excluded: GDN (both paths), routing (`--dump-routing` returns valid expert ids, e.g. layer 0 = `[434,350,308,91,105,238,161,2,201,167]`), expert streaming, and the MoE kernels generally — `STRATA_VERIFY_DEBUG=1` shows every layer ringing and being served with no error.
`STRATA_GR_V3` is not in play: `fused_gr.cu:2419` — `unset = gr_v3_default_on (a 12.0 card with > 13 GiB)`, i.e. sm_75 defaults it off.
### Smallest steps to reproduce
```sh
set STRATA_LATENT_A=<pack>/experts.bin
set STRATA_LATENT_LABELS=<pack>/labels.bin
set STRATA_LATENT_DENSE=<pack>/latent_dense_q8.bin
set STRATA_LATENT_DENSE_FMT=8
set STRATA_LAT_FUSED_Q3K=1
set STRATA_EMBED_VRAM=1
set STRATA_LATENT_FUSED=1
set STRATA_LATENT_FUSED_DU=1
set STRATA_LATENT_U_GPU=1
strata --pack <pack> --native <pack>/model-skeleton.gguf \
--native-dense-gguf <pack>/model-skeleton.gguf \
--ple-gguf <pack>/ple-fp8.gguf --ple-io direct \
--expert-profile <pack>/profile.bin --expert-cache auto \
--prefill 64 --spec 4 --max-context 4096 --kv fp16 \
--tokens "39,92003" --max-new 8 --gpu 1
# -> output : 0 0 0 0 0 0 0 0
```
Add `STRATA_VERIFY_PROFILE=1` to see `rms=0.0000`; add `STRATA_DBG_NAN=1` to see the healthy prefill residual.
(Two footnotes for anyone re-running this on Windows: there is no `strata generate` subcommand, and paths must use forward slashes — a backslash breaks the `labels.bin` lookup.)
### Questions
1. Has the **plain (`non-interleaved`) `native_mmvq` Q5_K path** ever been validated for *numerical* correctness on sm_75? (#5137 notes that `kIl58Arch` covers sm_80 / sm_120 only, so Turing runs the plain kernels.)
2. Have you seen **zero-valued logits rather than NaN** from the native head on Turing? A clean 0 with `nonfinite=0` and a finite trunk residual looks like a dot-product that never accumulates, rather than a bad scale.
3. Is `--dump-logits` writing a 0-byte file a known issue? It prints `logits dumped` but the file is empty.
### Environment
- RTX 2080 Ti 22 GB (modified VRAM, eGPU over PCIe x4) · 32 GB RAM · Windows · CUDA 13.x
- Build: `setup.py` package, `CMAKE_CUDA_ARCHITECTURES=75`; `strata_kernels.lib` 62 MB
- Model: 120B MoE, Q3_K, 85.94 GB pack, expert arena 27.98 GiB; `low-ram` / `file-tier` never appeared
- decode 14.7–30.5 tok/s depending on switches; prefill 2.1–2.8 tok/s (streaming experts from disk, 32 GB RAM)
### Area
[Engine]
### Desktop app, Terminal or source?
Terminal (the `strata` command), built from source (our tree is a fork of `c62d5e3`).
### Reproduction status
Reproduced locally (both GPU runs; the two commands are in the comments above).
### Strata version
Strata 0.1.55 · engine 0.1.55 (our fork of `c62d5e3`). The two code facts (`session_zero` call sites, `if (native_pack) { spec_pos = pos; break; }`) also hold in the `v0.2.16` source tree; we have not yet run the 0.2.16 binary.
### Graphics card, memory and system
RTX 2080 Ti 22 GB (modded VRAM, sm_75, eGPU over PCIe x4) · 32 GB RAM · Windows. Cross-check on RTX 4080 Laptop 12 GB (sm_89) · Windows.
本站相关内容
相关页面的快捷入口。