#6145 [Engine] Deterministic #879 with --ple-io direct on UD-Q6_K_XL (mmap passes; expert blob bytes certified clean; boot-state dependent)
open · @IvanAndreychikov · 4 commentaires · Sur GitHub
Multi-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows
Description
## Summary
Follow-up to #3602 (closed), continuing that thread's phenomenon on a different quant and a
different machine: two captured real requests fail with `non-finite logits (#879)`
deterministically on `--ple-io direct` within one engine boot, pass on `--ple-io mmap`, and then
pass on direct again in later boots. A byte-level check of the direct expert path finds no
corrupted bytes anywhere it can verify: a patched engine (opt-in `STRATA_BLOB_VERIFY=1`, the
default path stays byte-identical) re-reads every blob `read_direct` delivers through an
independent channel and compares it. The expert blobs are byte-correct in every sample so far.
What did fail, twice and verifiably, is a *buffered* reader under the engine's mixed I/O: it
returned, twice consistently, pages of the same file from a different offset. The captured bodies
are private traffic, shareable privately on request.
## Environment
- Engine: local rebuild of 0.1.56 (`f-0291-rc1` / `06698c674673` source, Release, CUDA 13.1,
`STRATA_Q6K_EXPERTS=ON`, `STRATA_MMQ_KQUANTS=ON`), default-path bytes kept identical to the
release; the release exe itself was also exercised (one note below).
- Model: Qwen3.8-Flash-Next **UD-Q6_K_XL** (157.6 GiB, six SHA-256-verified shards), read in place
(`--native`), 24,576 experts, all 48 layers; local exact-FP32 pack (460 conversions, 0 rounded),
INT8 KV, `--kv-resident 32768`, 262144 context, dflash2 draft, `--layer-split 9,24`,
`--resident-budget-gib 63`, `--expert-cache auto` (4,030 GPU slots, 63 GiB in RAM, the rest on
the file tier).
- GPUs: RTX 2080 Ti (sm_75) layers 0-8 + vision, RTX 5060 Ti (sm_120) layers 9-23, RTX 4080
(sm_89) layers 24-47 — three generations on one box.
- Windows 11, AMD Ryzen, 96 GiB RAM, NVMe Gen5.
## The arm split, measured
Two captured real request bodies that die with `#879` on `--ple-io direct` (verbatim replay, both
bodies 3 tries each, the automatic retry included):
| boot | direct | mmap |
|---|---|---|
| 2026-11-16 evening | **6 of 6 fail** | **23 of 23 pass** (the same bodies plus live traffic) |
| 2026-11-17 boots (x3) | passes (6/6; `min-wiki.json` from #3465 passes too, like the public controls in #3602) | passes |
Within a failing boot the replay is deterministic down to the engine's own counters: the failing
window reports identical numbers every try (`GU 58 H 58 Dm 1276` at T=2677, `2/2/44/44` at T=136,
`7/7/154/154` at T=399), and the layer trace puts the first non-finite values in the layer-18 MoE
— a layer whose experts live on the file tier of shard 4, on the sm_120 stage. Across boots the
outcome varies: the same bodies that failed 6/6 in one boot pass 6/6 in another, while mmap has
never failed in any boot tested. This matches what the #879 reporter described ("deterministic
per-seed reproduction does not survive an engine restart"), quantified per boot.
## What the byte check found
The verifier arms after startup (`STRATA_BLOB_VERIFY_DELAY_SEC`) and, after every `read_direct`
delivery, re-reads the same byte ranges twice through an independent buffered channel (both
re-reads must agree with each other before any verdict) and arbitrates confirmed mismatches
against a third, memory-mapped view of the file.
- **No direct-read corruption was ever caught.** Every blob delivered through `read_direct` during
the replays and live traffic (direct arm, file tier active) was byte-correct. Honest limit: no
`#879` event occurred while the verifier was armed (the failing boot predates the instrumented
build), so this certifies passing deliveries, not the moment of a NaN — consistent, though, with
#879's finding that the poisoned expert's cache slot was byte-identical to the pack file.
- **The buffered reference channel itself failed twice, verifiably.** Under the engine's mixed
buffered + unbuffered I/O on the same 46 GiB file, the buffered re-read returned, twice in a row
and consistently, bytes that belong to a different offset of the same file (offset
4,032,351,328+159,750: file, mmap and `read_direct` all say `0x66`; the buffered channel said
`0x69` twice, and its dump matches the file at +11,139,624,054 — 11.1 GiB away; a second case
similar). Both were post-verified with two independent channels (memory-mapped view + a plain
external read). The engine itself never reads buffered — this is diagnostic-channel evidence,
but it shows wrong pages can be served on this box when one file is opened through several
channels at once under load.
Standalone probes outside the engine, all clean: concurrent unbuffered re-read verification of the
failing span (~154 GiB, 0 mismatches); a GPU-DMA probe on all three GPUs (pinned H2D -> VRAM
kernels -> D2H, bit-exact, ~118 GiB disk re-reads alongside ~45 GiB of DMA); a mixed-channel probe
running buffered-overlapped and unbuffered-overlapped reads of the same offsets of one shard
(~95,000 verified 4 MiB windows, ~370 GiB, 0 mismatches). The wrong-page behavior needs the
engine's process context to appear.
## Side observations (flagged unresolved, not attributed)
- The instrumented build crashed twice during startup (`0xc0000005` in WER, faulting module
`ntdll.dll` / unknown) while the verifier was active during the startup cache fill; with the
verifier held off until after load, no crash. My instrumentation is not ruled out; minidumps
kept.
- The release 0.1.56 exe exits silently during startup on `--ple-io direct` with this pack,
reproducibly (2 of 2), right after the `blk.1.ple_conv1d.weight is not stored as F16` warning;
the local rebuild of the same source loads the same pack fine on the same arm. Possibly
pack-specific (the pack is a local exact conversion).
## What this narrows down
Expert blob bytes are certified clean wherever verifiable, and the failing bodies pass on other
boots — so on this box the poison is made after the blob lands (as #879 concluded) and depends on
process/boot state, not on what the disk returned. Combined with #3602's trace (first non-finite
already in the gathered PLE embedding, the residual clean), the PLE input
acquisition/decoding/upload path stays the strongest suspect here too.
## Next steps I can run on this machine
1. The same byte-check aimed at the PLE row path (gathered rows vs disk vs uploaded GPU floats) —
the check suggested in #3602.
2. A CPU-only row fixture (#3602's step 2).
3. Fixed-chunk cold A/B (direct vs mmap) with per-boot results; the failing-boot recipe is
"evening-state boots fail, morning boots pass" so far, and I can capture whatever state you
want instrumented.
4. The blob verifier itself (~120 lines, Windows paths only, opt-in, default path byte-identical)
— worth a PR if you want it upstream.
Related: #3602 (the same phenomenon, IQ3_S), #3465, #879 (umbrella), #5055 (sm_120 — the poison
here is born on the sm_120 stage), #6121 (UD-K_XL family — the arm dependence there refines the
content-only hypothesis).Sur le site
Liens install, modèles, releases.