#6584 [Engine] 0.2.19 sm_120 defaults break the FP4-Local (nvfp4_local_pack) Qwen3.8-Flash-Next pack: EXP_MODE=8 gives garbled output, PREFILL_NVFP4=w4a4 gives non-finite logits (#879) on long prompts
open · @MagneticApp · 4 コメント · GitHub で見る
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsLinux
本文
### Area [Engine] (speed, kernels, models, the server and API) ### Desktop app, Terminal or source? Source build, Terminal server (`terminal/serve/server.py`), Linux. ### What happened The engine prints at start: `sm_120 (Blackwell): 13 exact speed switches on by default (...): HC_UPMIX L2_PF Q80_ROWS1 EXP_MODE SIDE_MIXER DEQUANT_SIDE QFUSE LFUSE PLE_COMPACT PLE_COPY_STREAM PREFILL_CPU_SHARE_NT PREFILL_NVFP4 VERIFY_DEVICE_HITS` Two of these break this pack. **1. `STRATA_EXP_MODE=8` (sm_120 default): every reply is garbled from the first token.** Greedy, thinking off, e.g. a one-word question returns `' RATHIRaud轻便toThrow抛出 arch6的一道'`; tool-call requests return no tool call. Bisect, each row a fresh engine start, 2 short prompts at temperature 0: | env | result | |---|---| | defaults (13 on) | garbled | | `STRATA_SM120_PERF=0` | correct | | `STRATA_PREFILL_NVFP4=0` (12 on) | garbled | | `STRATA_EXP_MODE=-1`, the other 12 on | **correct** | | `SIDE_MIXER, DEQUANT_SIDE, PLE_COMPACT, PLE_COPY_STREAM, HC_UPMIX, PREFILL_CPU_SHARE_NT` = 0 | garbled | | `L2_PF, Q80_ROWS1, VERIFY_DEVICE_HITS, LFUSE, QFUSE` = 0 | garbled | **2. `STRATA_PREFILL_NVFP4=w4a4` (sm_120 default), with `EXP_MODE=-1`: short prompts are fine, but a 44,427-token prompt fails** with non-finite logits (#879) right after the prompt read, and the client gets HTTP 503. 2 of 2 runs. The prompt read itself was fast: 44,427 tokens in 25,728 ms (1726.8 tok/s). With `STRATA_PREFILL_NVFP4=0`: 0 non-finite in 2 of 2 runs, correct answer. ### Smallest steps or command that reproduce it 1. Start the engine with the settings below on sm_120 with defaults; send any short chat request at temperature 0: the answer is garbled (`EXP_MODE=8`). 2. 2. Add `STRATA_EXP_MODE=-1`: short requests are correct. Send a ~44K-token prompt: the request ends with non-finite logits and HTTP 503 (`PREFILL_NVFP4=w4a4`). 3. 3. Add `STRATA_PREFILL_NVFP4=0` as well: all requests are correct. ### Reproduction status Reproduced locally, every run (see the counts above). Happy to run more env combinations or a debug build on request. ### Strata version v0.2.19 (tag `c6f6427d9`). Engine built from source in `nvidia/cuda:13.0.2-devel-ubuntu24.04` (cmake 3.28.3): `cmake -G Ninja -S engine -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF -DSTRATA_EXPERIMENTAL_SM60=OFF -DSTRATA_NATIVE_EXPERTS=ON -DSTRATA_Q6K_EXPERTS=ON -DCMAKE_CUDA_ARCHITECTURES=120a` CMake reports: `Blackwell mxf4nvf4 + mxf8f6f4 prompt & hardware FP4 decode kernels enabled (120a-real)`. Baseline for comparison: v0.2.13 built from source on the same machine. ### Graphics card, memory and system NVIDIA RTX PRO 4000 Blackwell SFF Edition, 24 GiB (sm_120) · AMD Ryzen 9 9955HX · 91 GiB RAM · Ubuntu 24.04.5, kernel 7.0.0-42 · driver 595.99.02 (CUDA 13.2). ### Model, size and settings `nvidia/Qwen3.8-Flash-Next-FP4-Local`, packed with `nvfp4_local_pack.py` (pack built 2026-10-25; the 0.2.19 version of the tool only changes imports, so the pack format should be unchanged). Server: `--parallel 4 --lazy`. Engine args: `--pack <pack> --native dense.gguf --native-dense-gguf dense.gguf --ple-gguf ple-nvfp4.gguf --expert-profile terminal/data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <rt> --max-context 262144 --kv k8v4 --kv-resident 32768 --conversation-cache-mib 512 --vision --resident-budget-gib 54` ### Logs ``` strata serve: non-finite logits (#879) in a first decode window at position 44426 (row 0): its tokens are not emitted, the request ends (DONE nonfinite) ``` ### Workaround and numbers `STRATA_EXP_MODE=-1` + `STRATA_PREFILL_NVFP4=0` passes all our checks (text, tool call, fenced code, conversation isolation, vision, a 44K-token needle). Same machine, same session, same check script: | | v0.2.13 | v0.2.19 + workaround (2 runs) | |---|---|---| | 44K-token needle, end to end | 30.6 s | 32.7 s / 32.8 s | | decode, 1 stream | 38.4 tok/s | 36.8 / 34.8 tok/s | | decode, 4 streams total | 56.8 tok/s | 57.2 / 57.9 tok/s | So on this pack the 0.2.19 Blackwell speed-up does not show once these two switches are off, and we stay on 0.2.13 for now. **Possibly related:** #6503 (garbled output on sm_120, IQ3_S_G8 pack). There `EXP_MODE=0` and `SM120_PERF=0` did not help; here `SM120_PERF=0` and `EXP_MODE=-1` (automatic) both do, so `-1` might be worth trying there too. (All numbers come from our own runs on this host.)
関連リンク
インストール・モデル・リリースへのサイト内リンク。