#6019 [Engine] Rare non-finite logits (#879) with native IQ2_XS and IQ3_XXS packs on sm_86, engine 0.1.59.1 (none with Q2_0 on the same flags)
open · @emoubarak · 1 评论 · 去 GitHub 看
BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsSecurityWindows
说明
### Area [Engine] (speed, kernels, models, the server and API) ### Desktop app, Terminal or source? Source (START-HERE.bat, setup.sh or a build) ### What happened On engine 0.1.59.1, the native IQ2_XS and IQ3_XXS packs of Qwen3.8-Flash-Next ended 3 requests with `non-finite logits (#879)` during a night of benchmarks. The Q2_0 GGUF on the same build, with the same flags, had none. All arms were cold starts of the same binary with the same command line, except `--pack`, `--native` and `--ple-gguf`: | Pack (engine 0.1.59.1, MTP, `--spec-min-p 0.5`) | Starts | Requests | Non-finite | |---|---|---|---| | IQ2_XS, native pack | 17 | 1,410 | 2 (decode) | | IQ3_XXS, native pack | 5 | 262 | 1 (prompt) | | Q2_0, native pack | 10 | 574 | 0 | | Q2_0, older non-native pack | 11 | 568 | 0 | Another 9 Q2_0 starts with other drafting settings (DFlash2, `--spec-min-p 0` or unset) served 700 requests with 0 events. The previous build, Strata 0.2.9 / engine 0.1.55, serves the same Q2_0 GGUF on this machine and has 0 events in its last 1,197 requests. The three events: 1. IQ2_XS, `STRATA_ADAPT_FETCH=2`: `decode window at position 204 (row 1)`. A 47-token prompt stopped after 157 of 700 tokens. This start also scored 29/40 and 32/40 on a fixed 40-question arithmetic check, while the 12 other IQ2_XS starts where I ran this check scored 39 or 40/40. Just before it, a llama.cpp-based server on the same card had been killed in the middle of a request. 2. IQ2_XS, no `STRATA_ADAPT_FETCH`: `decode window at position 176 (row 0)`. A 49-token prompt stopped after 128 of 700 tokens, and 4 parked conversations were dropped. Just before this start, another llama.cpp server (a different model) had used the card. 3. IQ3_XXS, `STRATA_ADAPT_FETCH=2`: `prompt window at position 36046 (row 0)` of a 36,053-token prompt, which followed a 14,688-token prompt in the same conversation. The retry through the batched path completed normally (101 tokens). Each time, the next request went through. The same 47/49-token request produced its full 700 tokens 21 other times on IQ2_XS starts. Expected: no non-finite logits with IQ packs, as with Q2_0. Checked so far: - The sha256 of both IQ shard 1 files matches the Hugging Face LFS oid at revision `ed59f92`. I recomputed it after the events: IQ2_XS `92cee27a…`, IQ3_XXS `219ea929…`. That rules out the corrupted download from #5263. Shard 2 has the same oid (`316b46f3…`) for Q2_0, IQ2_XS and IQ3_XXS, so all arms load the same file. - A control vector is on in every arm, IQ and Q2_0 alike (`--control-vector-scaled … --cvec-mode project`, layers 4-44). I have not run without it. - The card is an eGPU over OCuLink. The engine's PCIe share line measures the link at 6.7 GB/s. - One separate crash happened the same night, at 04:01, on the older Q2_0 pack with `STRATA_ADAPT_FETCH=2`: `prefill gemm: cublasGemmEx f16: cuBLAS status 14`, with an NVIDIA `Xid 13` in the kernel log. It is not counted in the table. These may be related to #5055 (sm_120, IQ3_S/IQ3_XXS), #5093 (CUDA DMA staging) or #5701 (engine 0.1.59, long sessions). I have not confirmed a link to any of them. ### Smallest steps or command that reproduce it I cannot trigger it on demand: 3 events in 1,672 IQ requests, at different positions, and the failing request completes when sent again. What I ran: cold starts of the command below, each followed by a few dozen to about 90 requests. The requests were arithmetic questions without thinking, three 700-token generations (prose, code, rewrite), and prompts of 2k to 147k tokens. ``` STRATA_PREFILL_CPU_SHARE=1 STRATA_KV_PREFETCH=1 STRATA_PREFILL_MMQ=0 STRATA_ROUTE_TAIL_SKIP=4 [STRATA_ADAPT_FETCH=2] \ ~/repos/tools/strata-0213/engine/strata --serve --expert-profile ~/repos/tools/strata-0213/data/expert-profile.bin --expert-cache auto --prefill 16384 --spec 4 --mtp ~/.local/share/strata/mtp/rt --max-context 262144 --kv int8 --control-vector-scaled ~/.local/share/strata/cvec/Huihui-Qwen3.8-Flash-Next-refusal-direction-r.gguf:1.0 --control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer --prompt-cache-every 2048 --vram-reserve-mib 960 --spec-min-p 0.5 --kv-resident 32768 --conversation-cache-mib 8192 --conversation-cache-slots 4 --vision --pack ~/.local/share/strata/packs/iq2_xs-native --native ~/.local/share/strata/models/IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf --ple-gguf ~/.local/share/strata/models/IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf ``` Pack built with `setup.py --prepare-pack ~/.local/share/strata/packs/iq2_xs-native --family qwen --model IQ2_XS --gguf-dir ~/.local/share/strata/models/IQ2_XS`. My next run will be a few hundred requests of IQ2_XS native against Q2_0 native, with `STRATA_DBG_NAN=1`. I will post the result here. Tell me if another switch would help more. ### Reproduction status Reproduced locally ### Strata version Strata 0.2.13.1 · engine 0.1.59.1 · commit 628f74955a56fcba092495b800dd02af75373a0f, built from source for sm_86 with CUDA 13.4 ### Graphics card, memory and system RTX 3080 20 GB (sm_86, driver 615.78.08) as an OCuLink eGPU · Ryzen 7 8845HS · 60 GB RAM · Fedora 44, kernel 7.3.3 ### Model, size and settings ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF (revision ed59f92), IQ2_XS and IQ3_XXS native packs; Q2_0 for the comparison · 256K max context, kv int8 · vision on · MTP drafting, spec 4 ### Logs Event 1 (IQ2_XS, `STRATA_ADAPT_FETCH=2`): ``` strata serve: prompt 940 tokens = 0 reused + 940 read in 1629 ms (577.1 tok/s), 700 generated in 3284 ms (213.2 tok/s), drafts accepted 635 of 640, 1 checkpoints strata serve: suffix drafts: 55 windows, 603 of 605 drafts accepted strata serve: non-finite logits (#879) in a decode window at position 204 (row 1): its tokens are not emitted, the request ends (DONE nonfinite) and the next request reads its prompt from token 0 strata serve: prompt 47 tokens = 0 reused + 47 read in 341 ms (137.8 tok/s), 157 generated in 1828 ms (85.9 tok/s), drafts accepted 69 of 142, 1 checkpoints strata serve: after non-finite logits (#879): dropped the live session, 1 checkpoint, 4 parked conversations and 0 idle slots (0 running slots not kept after their reply); reading this prompt from token 0 strata serve: prompt 63 tokens = 0 reused + 63 read in 357 ms (176.5 tok/s), 700 generated in 4213 ms (166.1 tok/s), drafts accepted 485 of 592, 1 checkpoints ``` Event 2 (IQ2_XS, no `STRATA_ADAPT_FETCH`): ``` strata serve: prompt 940 tokens = 0 reused + 940 read in 1707 ms (550.6 tok/s), 700 generated in 3962 ms (176.7 tok/s), drafts accepted 524 of 548, 1 checkpoints strata serve: suffix drafts: 5 windows, 39 of 39 drafts accepted strata serve: non-finite logits (#879) in a decode window at position 176 (row 0): its tokens are not emitted, the request ends (DONE nonfinite) and the next request reads its prompt from token 0 strata serve: prompt 49 tokens = 0 reused + 49 read in 389 ms (126.1 tok/s), 128 generated in 1500 ms (85.4 tok/s), drafts accepted 67 of 106, 1 checkpoints strata serve: after non-finite logits (#879): dropped the live session, 1 checkpoint, 4 parked conversations and 0 idle slots (0 running slots not kept after their reply); reading this prompt from token 0 strata serve: prompt 62 tokens = 0 reused + 62 read in 405 ms (153.0 tok/s), 700 generated in 4316 ms (162.2 tok/s), drafts accepted 485 of 590, 1 checkpoints ``` Event 3 (IQ3_XXS, `STRATA_ADAPT_FETCH=2`): ``` strata serve: PCIe share: the CPU pool takes 67 us per missed expert (26.0 GB/s of expert bytes), the link 6.7 GB/s -> pcie_frac 0.15 (was 0.10; --pcie-frac N fixes it, STRATA_PCIE_FRAC_DEFAULT=old keeps the start-up rule) strata serve: prompt 14688 tokens = 0 reused + 14688 read in 7277 ms (2018.5 tok/s), 80 generated in 749 ms (106.7 tok/s), drafts accepted 41 of 71, 1 checkpoints strata KV prefetch: 24 layer prefixes, single existing staging pool strata serve: non-finite logits (#879) in a prompt window at position 36046 (row 0): its tokens are not emitted, the request ends (DONE nonfinite) and the next request reads its prompt from token 0 strata serve: prompt 36053 tokens = 0 reused + 36046 of 36053 read in 20410 ms (1766.1 tok/s), 0 generated in 0 ms (0.0 tok/s), drafts accepted 0 of 0, 3 checkpoints (cancelled) strata serve: after non-finite logits (#879): dropped the live session, 3 checkpoints, 2 parked conversations and 0 idle slots (0 running slots not kept after their reply); reading this prompt from token 0 strata serve: after non-finite logits in a prompt window (#3465): this request reads its whole prompt through the batched path, none of it through the verify windows strata serve: prompt 36053 tokens = 0 reused + 36053 read in 20224 ms (1782.7 tok/s), 101 generated in 927 ms (109.0 tok/s), drafts accepted 54 of 77, 3 checkpoints ``` I can attach the full logs of the three starts if that helps.
本站相关内容
相关页面的快捷入口。