Issues / #3626
#3626 [Engine] Coder IQ1_M: SIGSEGV in kernels::cpu::native_gu_rows (ExpertPool), deterministic across 0.2.0-beta … 0.2.2 (RTX 4070)
open · @saschamarco · 0 Kommentare · Auf GitHub
Server & APIAMD / HIPNVIDIA / CUDAModels & quantsSecurityLinux
Beschreibung
## What happens
Serving `Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M` (coder pack) segfaults the engine. It
crashes reliably on the first generation requests of a session, never at load time
(load always reports `session is up`, `slot 0 verified`). The same machine serves
IQ3_S all day without a single crash.
## Environment
- Linux (Manjaro, kernel 7.2.11), AMD Ryzen 7 5700G (Zen 3, 8 physical / 16 logical), 62 GiB RAM, 127 GiB swap
- GPU 0: RTX 4070 12 GB (`gpu_roles: 0=main, 1=off`), GPU 1: RTX 3050 (unused by this config)
- CPU ISA as exposed by ggml-cpu autotune: AVX2 + BMI2 available, no AVX-512
- Engine 0.1.47 (Strata 0.2.2); also reproduced on 0.1.x from 0.2.0-beta (19.10) and 0.2.1 (20.10)
- Model: `Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-0000{1,2}-of-00002.gguf`, pack `coder-iq1_m`,
`expert-profile-coder.bin` (12288 pairs)
## Reproduce
```
engine/strata --serve --pack packs/coder-iq1_m \
--native models/coder-IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf \
--ple-gguf models/coder-IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf \
--expert-profile data/expert-profile-coder.bin --expert-cache auto --prefill auto \
--spec 4 --spec-min-p 0.5 --mtp mtp/rt --max-context 204800 --kv q4_0 --kv-resident 32768
```
Send any chat request. Today (21.10.) four sessions all SIGSEGV'd within the first
request each (16:36, 16:38, 16:48, 16:53). Historical coredumps show the same
signature on 0.2.0-beta (`--max-context 262144 --kv int8`) and 0.2.1 (with and
without cvec flags), 19.–20.10.
## Crash signature (identical in all dumps)
Main thread (verifier path), one of the pool threads:
```
#0 strata::kernels::cpu::native_gu_rows(NativeFmt const&, unsigned char const*, unsigned char const**, int, float const*, int, int) (strata + 0x80df47)
#1 strata::kernels::cpu::ExpertPool::drain(int, ExpertScratch&, unsigned) (strata + 0x7ec7df)
#2 strata::kernels::cpu::ExpertPool::run_phase(int, int) (strata + 0x7edab6)
#3 strata::kernels::cpu::ExpertPool::run_split_multi_native(NativeFmt const&, ExpertJobMulti*, int) (strata + 0x7edbc1)
#4 strata::core::expert_pool_dispatch_multi(...) (strata + 0x1c1035)
#5 (anonymous)::drive_pool_multi(...) (strata + 0xcbc6c)
#6 strata::core::Verifier::run(...) (strata + 0x2237bf)
```
The seven pool workers sit in `ExpertPool::worker → drain → native_gu_rows` at the
same instruction. The log ends right after
`strata verify: captured the 4-token window (upload no error, sync no error)`.
## What I ruled out
- **Not VRAM pressure**: the run logs the <512 MiB warning (#2065), but the config
crashes the same way with more free VRAM, and Coder IQ1_M is benchmarked stable on
an RTX 4070 12 GB (#3452, #3249).
- **Not a corrupt/truncated pack or GGUF**: byte sizes match upstream exactly; I
parsed all 48 `native_experts.txt` rows programmatically — every `gate_off/up_off/down_off +
blob_bytes` lies inside shard 1 and matches the IQ1_M block size.
- **Not quant-width per se**: the same engine args on IQ3_S (same card, same host)
never crash — the difference is the IQ1_M quant / coder pack.
## Coredump
A full dump of the last crash (25.6 GB, zstd) is kept:
`core.strata-host.1000.f8eff642fa2243969edabf71852762a0.342032.1792594314000000.zst`.
Happy to upload it or run `gdb` probes (faulting address, `$pc`) if you tell me
what is most useful.
## Isolation I can run on request
- `STRATA_PREFILL_CPU_SHARE=0` (the crash is in the CPU expert path)
- `STRATA_ROUTE_TAIL_SKIP=0`
- `--spec 0` to confirm the verify path is the trigger
The stack points into the native (IQ1_M) row reader — my guess is an out-of-range
row pointer / a row one superblock shorter than the size the pack table promises,
which load-time `slot 0 verified` would not catch.
Mehr auf der Site
Links zu Install, Modellen, Releases.