Issues / #3489
#3489 [Engine] 0.1.45: prefill crash at ~165K tokens (0xC0000005 in gather_native_group) takes the whole GPU with it; 0.1.43.1 clean, 0.1.46 unchanged
open · @litao9417 · 0 comentários · No GitHub
BenchmarksServer & APINVIDIA / CUDAModels & quantsWindows
Descrição
### What happens Reading a prompt of ~165K tokens kills the engine. It exits with `0xC0000005` (access violation) during prefill — the frames we could capture were in `gather_native_group` — and after that the **GPU is gone**: `nvidia-smi` no longer lists it and only a reboot brings it back. Three launches, three crashes, same place. ### System - RTX 3080 Ti 12 GB (sm_86), driver 617.42, 12288 MiB - i7-12700 (12 cores / 20 threads), 96 GB RAM, Windows 11 26300 - Model: Qwen3.8-Flash-Next IQ3_XXS (GSQ-RCO), 47.0 + 28.8 GB shards, PLE table on an SSD - Config at the time: `--max-context 262144 --kv int8 --kv-resident 32768 --prefill auto --vram-reserve-mib 1728`, no `--vision` - Started through `serve/server.py` (source build, tag `v0.1.45`) ### Reproducer One user message of roughly 220K characters of Chinese (~165K tokens) plus a one-line question, `max_tokens` small. The crash lands inside the first prefill, before any token is generated. ### Evidence that it is the engine and not the machine - **0.1.43.1 on the same box, same prompt: 6/6 passes, zero crashes, the GPU never dropped.** - 0.1.45.3 and 0.1.46 at `prompt chunk auto: 5632`: a 168,814-token prefill completes twice (87 s and 79 s, ~1.9K tok/s). So a long prompt alone is not enough. ### What we think the trigger is At crash time the engine printed `prompt chunk auto: 6400` (0.1.45 raises this above the 5632 of 0.1.43.1 on the same config). Today I cannot get 6400 back: `strata serve: prompt chunk 6400 -> 3200 tokens so its buffers fit in every expert cache`, so the exact condition is not reachable on my current VRAM settings. That is why the report is "crashed at 6400 three times, passes at 5632 twice" rather than a clean A/B. ### Code note `src/prefill/moe_mmq.cu` between tag `v0.1.45` and `a7bdbfdabc9a` (engine 0.1.46) differs by 63 lines, all of them about the new Gyro trellis types (`tq2_t` / `tqk6` / `tqk7`). `gather_native_group` and its `tail_off` / `zero_bytes` parameters are byte-identical in both. ### Ask 1. Is the 6400-token chunk path a known-broken one? 2. What should I set so the chunk really reaches 6400 on a 12 GB card, so I can confirm or clear it? 3. I kept `strata-<model>.log` from all three crashes — say the word and I will attach them. Happy to run anything you want on this machine.
No site
Links install, modelos, releases.