#6166 [Engine] opt-in blob verifier for the direct expert path: STRATA_BLOB_VERIFY
closed · draft · @IvanAndreychikov · 0 Kommentare · Auf GitHub
Setup & installAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Beschreibung
Problem and result: when the engine dies with `non-finite logits` (#879), the first question is
whether the expert bytes the direct reader just delivered were correct. Nothing in the engine
answers that today: #879 could only compare the poisoned expert's cache slot against the pack
file after the fact. This PR adds an opt-in diagnostic that certifies every blob at the moment
it is delivered: after each `read_direct` batch, the same byte ranges are re-read twice through
an independent buffered channel and compared against what the unbuffered path wrote. With it
armed on the machine from #6145 (direct arm, file tier active, UD-Q6_K_XL pack), every
expert blob delivered during replays was byte-correct, and the verification channel itself
was twice caught returning pages of the same file from a wrong offset under the engine's
mixed I/O - exactly the case the double re-read plus mmap arbitration exists to separate.
Result: a run of the direct arm with this verifier armed now carries a positive, per-blob
statement that the disk I/O was not the poison, instead of an absence of evidence.
Related discussion: #6145 (measured data, boot-state dependence; this verifier was offered
there as next step 4), #3602, #879 (umbrella), #5055 (the poison there is born on the sm_120
stage, which is also where #6145's layer-18 file-tier experts are read).
Approach: `STRATA_BLOB_VERIFY=1` arms, at mapped-GGUF open time, a per-blob re-check: after
every `read_direct` delivery the same byte ranges (the same offset arithmetic the direct path
uses) are re-read in 4 MiB chunks through an independent buffered channel - a synchronous,
thread-owned handle per shard file, opened on first use (`CreateFileW`, no overlapped state,
no shared file pointer between pool workers). Each range is read twice into separate zeroed
buffers; a verdict is only possible when both re-reads agree with each other, so a flaky
verification channel cannot frame the direct path. A read error on the verification channel is
logged loudly with its `GetLastError` code and counted as a failed verification (no verdict).
A confirmed difference is logged loudly (shard, offsets, role, first-diff hexdump of all three
views) and immediately arbitrated against a fourth view: the file mapped read-only through the
memory manager for the duration of the check. The arbitration prints one of `DIRECT CORRUPT`
(mmap agrees with the buffered re-read, not with `read_direct`), `CHANNEL ANOMALY` (mmap
agrees with `read_direct`; the verification channel returned foreign bytes), or that mmap
differs from both. Arbitration is capped at 50 events per process and detailed mismatch
logging at 200, both counted.
`STRATA_BLOB_VERIFY_DELAY_SEC=<n>` holds verification off for the first n seconds after open,
so the startup cache fill is not verified. The gate exists because of that history: the first
two iterations, which verified the startup fill, crashed the engine twice at load on the
#6145 box (0xc0000005, WER dumps archived); a later iteration armed behind the gate still hit
two access violations during armed replays (both times the same wild-address signature in
CRT stdio, dumps and link map archived). The verification channel was then rewritten from a
shared `OVERLAPPED` handle table to the synchronous per-thread handles described above, with
error codes logged - the code those crashes occurred on is gone. Across seven fresh engine
boots and 125 armed requests since the rewrite, no crash and no verification-channel failure
was observed on this machine, and one direct-engine run that verified the startup fill (the
gate disabled) completed clean with the exit summary
`total blobs compared 1592, mismatches 0, unstable channel reads 0, mmap arbitrations 0`;
whether the rewrite removed the cause of the armed crashes or they were boot-state dependent
(#6145) is not established. Verifying the fill is not offered by this patch; every dump is
kept in the validator's lab.
Default path: with `STRATA_BLOB_VERIFY` unset the only change on the hot path is one relaxed
atomic bool load per blob (false, then return), no handles are opened, and no counters are
touched. Windows paths only (`#if defined(_WIN32)` throughout; the non-Windows build keeps
the same empty stubs it compiles today). No defaults change; no precision, scheduling, or I/O
behavior changes on any path the user does not opt into.
Validation:
Passed:
- Build: v0.2.10 + this patch (branch nan879-blob-verify, commits 955198e42 and
417611ae0), local Release build, CUDA 13.1, MSVC 14.44.35207, `STRATA_ENABLE_CUDA=ON`,
`STRATA_Q6K_EXPERTS=ON`, `STRATA_MMQ_KQUANTS=ON`, `CMAKE_CUDA_ARCHITECTURES=75-real;
89-real;120` (the PC's three GPU generations - the repo default without the option is
sm_120-only and the binary then cannot start on older cards), external ggml per
docs/SOURCE_INSTALL.md; ninja `strata` target. The two changed files add no warnings
beyond the `getenv` deprecation notices (C4996) the file already carries for its other
`std::getenv` uses.
- Verifier off (default path): 7/7 replays HTTP 200, zero `nan879-verify` lines in the
engine log; exact commands and results in the first comment below.
- Verifier on: 125 armed requests across 7 fresh engine boots (the two captured #6145
bodies, a 40-try soak of the triggering body, `min-wiki.json`), 0 mismatches, 0 channel
anomalies, 0 failed verifications, 0 crashes; exact commands and results in the first
comment below.
Failed: none observed.
Not run:
- HIP and SYCL builds: no AMD/Intel toolchain or hardware on this machine. The changed
guards are Windows-only and sit inside code paths HIP/SYCL builds do not compile today
(the file's direct/`OVERLAPPED` sections are already `#if defined(_WIN32)`), but per
CONTRIBUTING this needs someone with the toolchain to confirm; keeping the PR a draft
until that.
- Linux build and run (the file's Linux O_DIRECT path is untouched; compile check wanted).
- Metal (the mmap arm there never uses `read_direct`).
Limits: verifies only deliveries that happen while it is armed - it certifies passing
deliveries and gives a three-way verdict at the moment of a real event, but it cannot rewind
an event that happened before the engine started (the #6145 failing boot predated this build).
While armed it doubles the file-tier read volume for verified blobs, so it is a diagnostic,
not a mode to leave on for benchmarks. On this machine the engine's teardown outlasts the
server's stock 20 s QUIT window, so the exit summary line (printed by the source's
destructor) is only observable when the engine is stopped directly; smaller setups may see
it in normal stops. Packs that are not mapped-GGUF expert sources are skipped (no handles,
counters stay zero). Windows only.
Test notes for reviewers: the two captured request bodies behind the A/B table in #6145 are
private traffic; I can share them privately on request. Logs with the verifier's banner,
error-code lines and the validation runs are attached below.
Mehr auf der Site
Links zu Install, Modellen, Releases.