Issues / #3071
#3071 [Windows/NVIDIA 0.1.45] engine 0xC0000005 at strata.exe+0x535c60 (Q4_K CPU dot product reads past a weight buffer), UD-Q4_K_XL with --batch-mtp on a RAM budget
open · @dnnspaul · 1 commentaires · Sur GitHub
Setup & installNVIDIA / CUDAModels & quantsWindows
Description
### Summary `strata.exe` 0.1.45 (ready-made Windows release, CUDA 13.0) crashed twice within three minutes with an access violation (`0xC0000005`, server: "the engine stopped unexpectedly (exit code 3221225477)"). The faulting offset was the same both times. The fault is a **read past the end of an expert weight buffer inside the CPU Q4_K x Q8_K dot product**, while serving Unsloth `UD-Q4_K_XL` with `--batch-mtp` on a RAM budget. ### Setup - Windows 11 Pro, RTX 5090 32 GB (driver 617.14), Ryzen 7 9800X3D (AVX-512), 96 GB DDR5 - Engine 0.1.45 release (`BUILD.json`: cuda 13.0, archs 75/86/89/120, vision gpu), repo at tag `v0.1.45` (477ac13) - Model: Unsloth Qwen3.8-Flash-Next `UD-Q4_K_XL` (rev 38bb39e), `--compat-bf16` pack, MTP draft layer, vision on - Engine args: `--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp ... --max-context 262144 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 1212 --resident-budget-gib 57 --batch-mtp --batch 3` (`"parallel": 3`), env `STRATA_HC_Q8=1 STRATA_ROUTE_TAIL_SKIP=0` (written by setup) ### What happened - Load log with `--batch-mtp`: `strata verify: window up to 8 tokens` (6 without it), and `FileExpertSource: WARNING: 91 of 1276 lendable slots (7%) will read their experts from the pack during a long prompt` (39 of 1276 without it), `the mapped view of the experts is closed (3 GGUF shards unmapped; every expert outside RAM and VRAM is read unbuffered)`, `only 673 MiB of VRAM free at the start of this request (below 1024 MiB)`. - About 4 h of single-user agent traffic, all on the solo path (no `strata batch` lines at all): sessions of ~110-127k tokens with cached follow-ups and resumed prompts of ~50k new tokens. - Crash 1: the engine log ends after a normal request (`prompt 121481 tokens = 121386 reused + 95 read ... 4125 generated`) with no error line; the server restarted the engine on the next request. - Crash 2: about 2 minutes after that restart, again silent, again after a ~127k-token request. - Windows Application log, both crashes: `Application Error` 1000, faulting module `strata.exe` (timestamp `0x6ad2b51e`), exception `0xc0000005`, **offset `0x0000000000535c60`**. ### Dump analysis (two WER minidumps, available on request) - Exception record: **read** access (`ExceptionInformation[0] = 0`) at `0xe308e0000` (crash 1) and `0xca7168000` (crash 2), both page-aligned, i.e. the read ran from a valid page into an unmapped one. - Disassembly at RVA `0x535c60` is the Q4_K x Q8_K dot product (ggml-cpu style: `and 0x3f3f3f3f`, `0x0f0f0f0f`, `0x30303030` scale unpacking). The faulting load is `vmovdqu ymm5, [rbx+0x20]`, where `rbx` walks the **weight** block (`d`/`dmin`/`scales` at `rbx-0x4e..-0x44`). So a CPU expert kernel read a Q4_K expert through a buffer that ends early, or through a stale pointer. ### What did not reproduce it - A scripted repro of the same pattern on the same `--batch-mtp` config: 10 rounds and 49 requests (cold 105-118k prefills, cached follow-ups, +50-85k resumed prefills up to 246k). No crash. - About 3 h of heavier benchmarks without `--batch-mtp` that morning (257k cold prefills, 3-4 concurrent ~240k sessions) also had no crash. ### Memory situation at the time (may matter) The Windows commit limit was 125.6 GiB: 94 GiB RAM plus a 33 GB system-managed page file that never grew. `strata.exe` held about 106 GiB of private bytes. The engine logged `Windows commit: 4.3 GiB left` at start, and during our repro requests commit free was **only 0.3-0.8 GiB**. We cannot tell from the dumps whether the faulting page was part of a mapping, because the WER minidumps have no MemoryInfoList. So a failed allocation or file-view map whose NULL/short result is not checked is also possible, as distinct from the `0xC0000409` commit-limit exit (#1607). We have since set a fixed 64-96 GB page file (commit free now ~34 GiB while serving). ### Guess Something on the solo path that `--batch-mtp` changes, either the 8-token verify window or the larger uncovered lend region (91 instead of 39 slots read from the file), leaves a CPU expert pointer or length that does not match the blob actually staged. We have reverted `--batch-mtp` for now (10 rounds of the repro pattern on the reverted config: no crash). A short `--batch-mtp` re-check with the larger page file (1-4 x 2k, 2-3 x 32k) also ran clean. If more logging would help (`STRATA_TRACE`, `--stats`), tell us which switches to run with.
Sur le site
Liens install, modèles, releases.