#6503 [Engine] Swift IQ3_S_G8 (GDN Q8_0) returns garbled token soup with engine 0.1.63 on sm_120, while the same files decode correctly in another engine
open · @jie3000 · 0 コメント · GitHub で見る
BenchmarksNVIDIA / CUDAModels & quantsWindows
本文
**Area:** [Engine] (speed, kernels, models, the server and API) **Desktop app, Terminal or source?** Desktop app (Windows) — the Terminal server with a saved arm **Strata version:** Strata 0.2.17 · control 0.1.63 (commit `12d491da4a6904eb49aaa6f2f63ec4b1a17582e3`) · engine 0.1.63 (release build, `01d81206d420`, CUDA 13, archs 86/89/120) **Graphics card, memory and system:** NVIDIA GeForce RTX 5090 Laptop GPU, 24 GiB VRAM (sm_120, 82 SM) · Intel Core Ultra 9 275HX (24c/24t, no AVX-512) · 255.4 GiB RAM · Windows 11 (10.0.26300) · driver 617.42 · CUDA 13.4 **Model, size and settings:** Swift `Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S_G8` (Unsloth "G8" variant, 2 shards + BF16 mmproj), local pack and model files. Arm: `--pack swift-iq3_s_g8 --native Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S_G8-00001-of-00002.gguf --expert-profile expert-profile.bin --expert-cache auto --prefill auto --spec 8 --spec-min-p 0.5 --mtp <rt> --max-context 262144 --kv int8 --kv-resident 49152 --resident-experts --pool-workers 15 --vram-reserve-mib 1012` **What happened:** Every completion on this pack returns garbled text — a stream of unrelated token fragments in mixed scripts, not a decoding of the prompt and not a plausible answer: ``` " TENTonatevinavicinclus スポンサーサイト twiceloch_numerOVOielding recap páginasprintStatsDMIielding Dyn亿欧元关闭窗口ramaieldingysize一代azionemporühappelleantyumpingloch▷不退alyti" "ce 「 finishinganziENUM rabatöffHeaderCode **■eeperInstantiationExceptionadle_yield Nicholson Overnight明け denialorzaszyanonapus/up仅提供信息onatибо因人egosapus/tty也不再 ス" ``` The same prompt on the same machine, the same engine build and the same day returns clean text on **UD-Q4_K_XL**, so the failure is specific to this pack. The same G8 files decode correctly in another engine (llama.cpp/Basalt), where they pass a quality gate and generate a playable Mario game at 177.8 tok/s — so the model files themselves are good. **Smallest steps or command that reproduce it:** Start the engine with the arm above and send any short chat request; the answer comes back as the token soup shown above. It does not depend on the prompt, the temperature or the output length: the very first characters are already garbage. **Reproduction status:** Reproduced locally. (The model files have since been removed from this machine, so the run cannot be repeated here right now — the investigation logs and the patches are listed below.) **Evidence collected** - The engine log of the G8 arm is the only one that prints the Blackwell switch block: `strata: sm_120 (Blackwell): 4 exact speed switches on by default (STRATA_SM120_PERF=0 turns them off, as does STRATA_DEFAULTS=026; a switch you set is kept): HC_UPMIX L2_PF Q80_ROWS1 EXP_MODE` The UD-Q4_K_XL arm on the same machine (engine 0.1.62) prints no such line. 0.1.63 turns these four on by default on sm_120. - Pack integrity was verified, so "bad download" is ruled out: shard 1 SHA-256 `010d0c6e…c8889fe` (identical to the copy another machine holds), the pack structure matches (dense.bin 1,538,035,200 B, 1079 tensors, 96/96 tensor conversions exact), and the engine is the official 0.1.63 release build. - The GDN Q8_0 blocks of G8 and of the working UD-Q4_K_XL pack are nearly isomorphic, which rules out a "Q8_0 bytes are wrong" explanation. - The G8 arm runs the **experimental pinned CUDA GDN path** (`--native-gdn`: "experimental pinned CUDA GDN norms/gates/recurrence", with the head, dense and PLE key taken from GGUF), unlike the UD arm, which reads its prompt on Strata's own K-quant kernels. **What was tried and did not help** (all on engine 0.1.63, G8 arm): | attempt | result | |---|---| | `STRATA_GDN_CHUNKED=1` | still garbled | | `Q80_ROWS1=0` | still garbled | | `OLD_Q80_SCALED=1` | still garbled | | `HC_UPMIX=0` | still garbled (the engine log confirms the switch was applied: "3 exact speed switches on by default … L2_PF Q80_ROWS1 EXP_MODE") | | `EXP_MODE=0` | still garbled | | `STRATA_SM120_PERF=0` (all four off) | still garbled, or the engine exited with rc=2 | | `--native-gdn` not used / no drafting | still garbled | **What would help** A pointer on where the G8 path diverges from the K-quant path would be much appreciated — the working UD pack and the broken G8 pack differ most in the pinned GDN path and in the expert/attention kernel selection, and the output looks like a logits or token-mapping fault rather than a sampling fault (the fragments are valid tokens, just unrelated ones). If G8 is not supported on this engine, saying so explicitly in the model list would also save people the debugging time I spent on it. --- *(Measurement notes: all timings and the garbled samples come from the engine's own output on this host; no third-party numbers are quoted.)*
関連リンク
インストール・モデル・リリースへのサイト内リンク。