#6121 [Engine] Non-finite logits (#879) with Unsloth UD-Q4_K_XL native on RTX 4090 — deterministic at 267 tokens; IQ4_XS and Q5_K_XL clean on same engine
open · @DarkSkar · 10 comentarios · En GitHub
Server & APINVIDIA / CUDAModels & quantsDocumentationWindows
Descripción
## Summary On an RTX 4090 (CUDA), the **UD-Q4_K_XL** GGUF of `unsloth/Qwen3.8-Flash-Next-GGUF` read in place (`--native`) degenerates into non-finite logits (#879) **deterministically**, even on a 267-token prompt — while **UD-IQ4_XS v31** of the same model, same engine, same config shape, on the same box, is completely clean (zero non-finite all day). We ran a one-variable-per-cold-boot probe ladder on our box; the signature is **identical in every arm**, which is why we initially suspected something specific to K-quant experts × CUDA rather than our config. **Update below (2026-11-17): UD-Q5_K_XL — also K-quant, same engine, same expert-profile, clone-exact config — is clean**, which points at the content of this specific UD-Q4_K_XL shard set rather than the K-quant family. ## Environment - GPU: RTX 4090 24 GB (sm_89), Windows 11, CUDA 13 prebuilt engine - Engine: 0.1.60 (build_id 09b603886925); also verified on 0.1.59 (same result) - Model: `unsloth/Qwen3.8-Flash-Next-GGUF`, `UD-Q4_K_XL`, 4 shards, re-downloaded fresh and **SHA-256 verified against the LFS oids** (revision `f5d175de`) - Launch (abridged): `--serve --pack <pack> --native <shard1> --expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.50 --mtp <rt> --max-context 262144 --kv int8 --kv-resident 32768 --vision --vram-elastic --vram-reserve-mib 700 --pcie-frac 0.30` - `STRATA_NAN_GUARD=1` ## Symptom - `/health` OK, model loads fine, chat returns nothing. - A **267-token** prompt aborts 100 % of the time with `non-finite logits (#879) in a first decode window at position 262/266`, 4 log lines, `0 generated`, HTTP 503 (`DONE nonfinite`). - Size-independent: also fails deterministically at ~22.8k tokens (same positions 6/6), while nearby bursts of the same size pass clean. - The chronic `only N MiB of VRAM free` warning is present but benign on this box (the IQ4_XS control shows it too and never NaNs). Reproduction status: **reproduced locally**, deterministic — one cold boot per arm (~2 min). ## What we ruled out (one variable per cold boot, same 267-token probe) | Arm | Result | |---|---| | Engine 0.1.60 → 0.1.59 (backup exe, witness `session is up (engine 0.1.59)` in log) | identical NaN | | Pack: rebuilt vs the original debut pack (A/B) | NaN in **both** arms | | Shards: abliterated (one shard found bad by SHA) → **official raw, 4/4 SHA-verified byte-exact** | identical NaN | | `STRATA_HC_Q8=0` (exact BF16 hyper-connection read) | identical NaN | | `--spec 2` | identical NaN | | `--mtp` removed entirely | identical NaN | | `STRATA_MMQ_KQ=0` — K_XL then logs the *fused prompt-kernels* path | identical NaN | | Control: UD-IQ4_XS v31, same engine, same config shape | **clean all day** | | Control: **UD-Q5_K_XL `--native`**, clone-exact config of the failing K_XL — only the GGUF content varies | **clean** (see update below) | Note on the last probe: with `STRATA_MMQ_KQ=0` the K_XL startup line changes to the same `prompt kernels: ... fused kernels` line the healthy IQ4_XS logs — so the fused prompt kernels themselves also can't be the root cause, since the clean model uses that same path. Surviving suspects on our side were: shared `--expert-profile`, arena/VRAM state — the UD-Q5_K_XL control below shares both and runs clean, so we now doubt either is the trigger. ## Update 2026-11-17 — UD-Q5_K_XL isolates the content, not the family Same box, same engine 0.1.60, same `data/expert-profile.bin`, config **cloned exactly** from the failing K_XL (`--spec 4` + MTP, `STRATA_HC_Q8=1`, `--kv int8 --kv-resident 32768`, expert-profile, `STRATA_NAN_GUARD=1`) — only the GGUF content differs: - **UD-Q5_K_XL** (also Unsloth Dynamic **v3.1**, like the failing UD-Q4_K_XL; per the repo README both quants are v3.1, updated Nov-2 — so v3.1 itself is not the differentiator). - All 6 shards **SHA-256 verified** against the LFS oids (all OK). - Result: `/health` OK in 64 s; the same 267-token prompt returns in **7.0 s with 0 non-finite values in the entire engine log** — where the failing UD-Q4_K_XL aborts 6/6 on that very probe. This survives the whole ladder without touching a single knob, so engine version, K-quant family, hyper-connection path, speculative decoding, MTP and our config all stay excluded. We now suspect **content-specific to these UD-Q4_K_XL shards** — e.g. expert-tensor values that the CUDA path amplifies into non-finite outputs where the Metal path (Nov-12 clamp on #879) would have absorbed them. ## Questions 1. Is the Metal fix (clamp of fp16 MoE activations, Nov-12 comment on #879) still absent on the CUDA path? The poison signature you describe there — garbage bits in MoE output rows, repeated token, NaN logits — matches ours exactly. 2. Is there any opt-in CUDA clamp/knob we could try for this while a fix lands? 3. Was it expected that `STRATA_MMQ_KQ=0` switches a K-quant model onto the fused prompt-kernels path (rather than back to llama.cpp MMQ)? 4. Given UD-Q5_K_XL is clean with the same expert-profile and config, does a content-dependent amplification (rather than a family-level CUDA bug) match what you see in #879? If it helps, we can dump the MoE output rows at the failing decode window for both quants and compare. Related: #879 (umbrella), #5259 (NaN reached the router on RTX 40 cards while reading prompts). Happy to run any instrumented build or extra knob on this box — the repro is deterministic and takes one cold boot (~2 min).
En el sitio
Enlaces a install, modelos, releases.