Issues / #3133
#3133 Crash (Xid 31 MMU Fault ACCESS_TYPE_VIRT_WRITE) during 22K+ token generation with 51K context
open · @dehuiyede · 0 Kommentare · Auf GitHub
BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quants
Beschreibung
## Environment
- **Strata Version**: 0.1.43.1 (Built from source with `STRATA_ENABLE_CUDA=ON`)
- **GPU**: 2× RTX 2080 Ti (22GB, sm_75), Driver 595.99.02 (CUDA 13.2), PCIe `03:00.0` & `82:00.0`
- **OS**: Ubuntu 22.04 (Kernel 6.8.0)
- **CPU/RAM**: Dual Xeon E5-2696 v3 / 126GB DDR4
- **Model**: Unsloth UD-IQ4_XS (Qwen3.8-Flash-Next)
- **Configuration**: `--max-context 262144`, `--kv int8`, `--kv-resident 32768`, `--prefill auto:32768`, `--pcie-frac 0.45`, `--mtp-q4 all`, `--mtp-window 6144`
## Problem Description
During an extremely long single-turn generation (approx. 22,446 generated tokens with a 51K prompt context), the engine crashed. The server reported:
`verify: an illegal memory access was encountered`
The kernel log shows a GPU MMU Fault:
```text
NVRM: Xid (PCI:0000:03:00): 31, pid=7979, name=strata, ... MMU Fault: ENGINE GRAPHICS GPC5 GPCCLIENT_T1_1 faulted @ 0xfffffa89_c4fcc000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_WRITE
Evidence from Engine Log
As the generation progressed and the context grew, the amount of KV cache streamed from RAM increased linearly:
Early in the conversation: KV streaming: 99.63% of 217372490 block reads hit VRAM, **3231.4 MiB read from RAM**
Just before crash: KV streaming: 99.69% of 398434695 block reads hit VRAM, **5015.8 MiB read from RAM**
It seems the engine attempted to write a virtual address out of bounds (ACCESS_TYPE_VIRT_WRITE) when migrating or streaming a large volume of accumulated KV cache back to VRAM during a very long generation.
Steps to Reproduce
Load Qwen3.8-Flash-Next with 256K context and KV streaming enabled (--kv-resident 32768).
Feed a prompt of around 50K tokens.
Instruct the model to generate a very long response (e.g., >10,000 tokens continuously).
Crash occurs around the 20K+ generated token mark.
Additional Context
Stable under normal load: This exact setup runs perfectly stable at ~57 tok/s for normal conversations (1-5K tokens) with 98-99% expert cache hit rate.
Trigger condition: The crash is highly correlated with extreme context length accumulation (>50K) combined with massive single-turn generation.
Hardware tuning: We have extensively tuned parameters for this Turing (sm_75) architecture, using STRATA_BF16_TC=1 for prefill acceleration and --pcie-frac 0.45 for optimal CPU expert pool scheduling. The server ran out of VRAM or crashed on older 0.1.40.4 under similar extreme conditions, but 0.1.43.1 consistently triggers this specific Xid 31 ACCESS_TYPE_VIRT_WRITE.
Workaround: Limiting the client max_tokens to <8192 avoids the issue, but long-form generation is part of the intended use case.
Mehr auf der Site
Links zu Install, Modellen, Releases.