Issues / #2666
#2666 Peer-expert tier (--peer-device 1) crashes on first prompt: "iq_dequant_gu_f16: an illegal memory access"
open · @CaptRauti · 0 comentarios · En GitHub
Server & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows
Descripción
## Bug: two-GPU peer-expert tier (`--peer-device 1`) crashes on the first prompt — "iq_dequant_gu_f16: an illegal memory access" **Engine:** 0.1.41 (locally built for this CPU) **OS:** Windows 11 (WDDM) **Hardware:** 2x NVIDIA RTX 3060 12 GB, Intel Xeon E5-2697 v2 (no AVX2), 128 GB RAM **Model:** swift-1.5-iq2_xs (Qwen3.8-Flash-Next MoE), context 128000, KV q4_0, `--kv-resident 32768`, `STRATA_ARENA_PIN_GIB=26` ### Reproduction Run on two GPUs (config `gpu: [0, 1]`) with `--peer-device 1`, so CUDA1 becomes an adaptive expert-cache tier instead of a layer split. The engine starts cleanly and fills the peer tier, but the **first** ~5.3k-token prompt crashes it deterministically: ``` strata generate: --trim-stage-weights needs a layer split across GPUs (ignored) strata generate: expert cache 3353 slots, 4.49 GiB of VRAM; policy is strata generate: peer GPU 1: 7723 experts, 10.36 GiB (filled in 4.5 s); with the primary's 3353 that is 11076 of 24576 on the GPUs strata generate: session is up (engine 0.1.41) strata serve: prompt chunk auto: 7936 tokens, a 96-slot ring strata prefill: peer GPU 1 computes its experts' rows of each prompt chunk (up to 79360 rows per layer, no P2P: through mapped host memory in FP16, one weighted sum per token back, compact group buffers); 0 MiB left free on it strata serve: 443 MiB of VRAM free with everything loaded iq_dequant_gu_f16: an illegal memory access was encountered ``` Crash site: `src/prefill/prefill.cpp:3679` (`iq_dequant_gu_f16` on the P6 native-pack dequant path). ### Not a VRAM-shortage (already ruled out) All three crash identically on the same first prompt: - `--vram-reserve-mib 700` (baseline): 443 MiB free on CUDA0, peer card 0 MiB free - `--vram-reserve-mib 1800` (primary reserve raised): 1497 MiB free on CUDA0 — same crash - `--prefill auto:8192` (smaller prompt chunks, per the engine's own startup hint) + `--peer-reserve-mib 1200`: same crash, peer card 679 MiB free With no peer tier the same model+prompts run fine on the same two cards as a layer split. ### Why this matters The peer tier is a big win for small cards: primary 3353 + peer 7723 = **11076 of 24576 experts on the GPUs**, versus ~5,700 on the two-card layer split for the same model. If the peer prefill path could run without the illegal access, small-VRAM dual-GPU boxes would get a substantial decode lift. Repro args: `--serve --pack ... --expert-cache auto --prefill auto:32768 --spec 4 --spec-min-p 0.5 --mtp ... --max-context 128000 --kv q4_0 --kv-resident 32768 --peer-device 1` Happy to attach the full engine logs or run further diagnostics.
En el sitio
Enlaces a install, modelos, releases.