Issues / #2666

#2666 Peer-expert tier (--peer-device 1) crashes on first prompt: "iq_dequant_gu_f16: an illegal memory access"

open · @CaptRauti · 0 commentaires · Sur GitHub

Server & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows

Description

## Bug: two-GPU peer-expert tier (`--peer-device 1`) crashes on the first prompt — "iq_dequant_gu_f16: an illegal memory access"

**Engine:** 0.1.41 (locally built for this CPU)
**OS:** Windows 11 (WDDM)
**Hardware:** 2x NVIDIA RTX 3060 12 GB, Intel Xeon E5-2697 v2 (no AVX2), 128 GB RAM
**Model:** swift-1.5-iq2_xs (Qwen3.8-Flash-Next MoE), context 128000, KV q4_0, `--kv-resident 32768`, `STRATA_ARENA_PIN_GIB=26`

### Reproduction

Run on two GPUs (config `gpu: [0, 1]`) with `--peer-device 1`, so CUDA1 becomes an adaptive expert-cache tier instead of a layer split. The engine starts cleanly and fills the peer tier, but the **first** ~5.3k-token prompt crashes it deterministically:

```
strata generate: --trim-stage-weights needs a layer split across GPUs (ignored)
strata generate: expert cache 3353 slots, 4.49 GiB of VRAM; policy is
strata generate: peer GPU 1: 7723 experts, 10.36 GiB (filled in 4.5 s); with the primary's 3353 that is 11076 of 24576 on the GPUs
strata generate: session is up (engine 0.1.41)
strata serve: prompt chunk auto: 7936 tokens, a 96-slot ring
strata prefill: peer GPU 1 computes its experts' rows of each prompt chunk (up to 79360 rows per layer, no P2P: through mapped host memory in FP16, one weighted sum per token back, compact group buffers); 0 MiB left free on it
strata serve: 443 MiB of VRAM free with everything loaded
iq_dequant_gu_f16: an illegal memory access was encountered
```

Crash site: `src/prefill/prefill.cpp:3679` (`iq_dequant_gu_f16` on the P6 native-pack dequant path).

### Not a VRAM-shortage (already ruled out)

All three crash identically on the same first prompt:
- `--vram-reserve-mib 700` (baseline): 443 MiB free on CUDA0, peer card 0 MiB free
- `--vram-reserve-mib 1800` (primary reserve raised): 1497 MiB free on CUDA0 — same crash
- `--prefill auto:8192` (smaller prompt chunks, per the engine's own startup hint) + `--peer-reserve-mib 1200`: same crash, peer card 679 MiB free

With no peer tier the same model+prompts run fine on the same two cards as a layer split.

### Why this matters

The peer tier is a big win for small cards: primary 3353 + peer 7723 = **11076 of 24576 experts on the GPUs**, versus ~5,700 on the two-card layer split for the same model. If the peer prefill path could run without the illegal access, small-VRAM dual-GPU boxes would get a substantial decode lift.

Repro args: `--serve --pack ... --expert-cache auto --prefill auto:32768 --spec 4 --spec-min-p 0.5 --mtp ... --max-context 128000 --kv q4_0 --kv-resident 32768 --peer-device 1`

Happy to attach the full engine logs or run further diagnostics.

Sur le site

Liens install, modèles, releases.