#5748 [Engine] verify: STRATA_TAP_FILE works on a layer split
closed · @botBehavior · 0 comentários · No GitHub
Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Descrição
## Problem and result
`STRATA_TAP_FILE` (the residual-stream research dump) is switched off without any message when the model runs on a layer split. `Verifier::init` drops the path unless one verifier holds every layer (`tap_whole`), so on two cards the serve starts, answers normally and never creates the file. Before this change you could only read a model's residual stream if it fit on one card.
After this change, a two-card split writes the same "STAP" v1 file as one card: the same header, the same layer list and the same `[pos, token, ntaps × n_embd]` rows, with every requested layer filled in from whichever stage holds it. If the dump can't be done, the engine now says so instead of skipping it silently.
## Related discussion
none. This is a focused fix (two files, +53/−5) with no change to defaults, so it comes straight to a PR as CONTRIBUTING allows. I searched issues and PRs for "TAP_FILE" and "taps split" and found nothing that covers it.
## Approach
It reuses the mechanism the DFlash2 drafter already has for split taps, rather than adding a new one:
- On a split, each stage parses `STRATA_TAP_LAYERS` into `tap_req_` (sorted, unique, last layer excluded, the same rules as the one-card path). It then keeps the layers inside its own range, exactly as `set_taps()` does for the drafter.
- The first stage (`lb_ == 0`) opens the file and writes the header with the full layer list.
- After the chain runs, the first stage calls `gather_taps_host()`, which already walks `next_` and copies each stage's rows into the right slot. It then writes the window. Both return paths are covered: the overlapped split, after the chain sync, and the sequential split, after `next_->run()` returns. The last stage never writes, because its `tap_file_` stays null, so no row is written twice.
- `record_window` skips the one-card mapped copy when the split path is active.
- Two new messages, both printed once by the first stage: `STRATA_TAP_FILE ignored: STRATA_TAP_LAYERS names no layer in 0..N-2`, and `STRATA_TAP_FILE ignored: the drafter owns this layer split's taps` (when `set_taps()` already claimed the taps, the drafter's layers are left alone).
Compatibility: with `STRATA_TAP_FILE` unset nothing new runs, because `tap_split_` stays false. The one-card path is unchanged.
## Validation
Environment: Windows 11, 2× RTX 5070 Ti 16 GB (sm_120), Ryzen 7 3800X, 128 GB RAM. CUDA 13.0.2 and MSVC 14.44, `cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120`. Baseline = `dbb2d16` built the same way. Model: a Qwen3.8-Flash-Next IQ3_S GGUF (48 layers, n_embd 2560) with the MTP head (`--spec 4`). `--layer-split auto` put layers 0–24 on CUDA0 and 25–47 on CUDA1. Serve with `--max-context 65536 --kv int8`, in the deterministic lane from docs/DETAILS.md ("Reproducible output"): `STRATA_IQ_MT_MIN=1 STRATA_ROUTE_TAIL_SKIP=0`, `--prompt-cache 0 --adapt-swaps 0 --pcie-frac 0`. Three short English prompts (22–25 tokens), `temperature: 0`, `max_tokens: 96`, thinking off. Fresh serve for every run.
**Passed:**
- **No-op when unset.** Baseline split (two runs after a cold first load) and patched split with the tap unset gave byte-identical replies on 3/3 prompts.
- **The dump does not change the output.** Patched split with `STRATA_TAP_FILE` set and `STRATA_TAP_LAYERS=5,15,25,35,45` gave byte-identical replies to the above on 3/3 prompts. The log says `STRATA_TAP_FILE …: 5 taps of 2560 floats per row (layer split: gathered from every stage)`.
- **Rows match one card.** The same binary on one card (no `--layer-split`) with the same tap settings was the reference. I compared rows by position, last row per position (the accepted token), up to each request's first token divergence: 297 positions. Per-layer cosine, median (min): L5 0.99998 (0.9990), L15 0.99991 (0.9982), L25 0.99974 (0.9904), L35 0.99938 (0.9437), L45 0.99967 (0.9929). Layers from both stages match. Positions from 0 are present in both files (423 vs 418 rows over the three requests).
- **Slots are in the right order.** Wrong-layer control: split layer k against one-card layer k+1 at the same position gives median cosine 0.68, against ≥0.9994 for the matching layer.
- **No empty slots.** 0 of 2115 (row, tap) slots in the split file are all-zero, and every value is finite.
- **Warning.** `STRATA_TAP_LAYERS=47` on a split prints `STRATA_TAP_FILE ignored: STRATA_TAP_LAYERS names no layer in 0..46`, creates no file, and the replies are byte-identical to the no-op run.
**Failed:**
none observed. One note: the first baseline run after a cold load differed from its own rerun on 2/3 prompts. That is the unpatched `dbb2d16`, so it isn't caused by this change, and the reruns were identical to each other and to every patched split run.
**Not run:**
- Linux, HIP, SYCL and Metal builds (CUDA on Windows only). The Metal tap path (`metal_engine::window_taps`) is untouched.
- The "drafter owns the taps" message: it needs DFlash2 on a split, which I didn't have.
- Three or more stages: I have two GPUs. `gather_taps_host()` walks the whole `next_` chain, but this is untested beyond two.
- Prompts long enough to take the batched prompt path, and `--batch-groups` serving (the dump is skipped under `batch_rec_` on both paths, as before).
## Limits
- Split-vs-one-card rows are close, not bit-identical. The two configurations place experts differently (VRAM vs CPU), and the IQ CPU kernels round differently. The gap grows with depth, so the worst position was at L35 (cosine 0.944). Bit-identical rows are only expected against the same placement.
- The first stage gathers through host memory synchronously each window. That's fine for a research dump, which is already ~n_taps × n_embd × 4 bytes per row to disk, but it isn't meant for production serving.
No site
Links install, modelos, releases.