Pull requests / #2458
#2458 engine: --dump-kv-select, the decode windows' KV block selections as a policy-bench trace (#2415)
open · @MrKop-git · 0 comentarios · En GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows
Descripción
## Summary The engine-side half of #2415 (the bench is #2434): `--dump-kv-select PATH` records what the KV streaming page pool is actually asked for - **one record per resolve call** of the decode windows, `int32 layer, int32 k, k int32 block ids, k float32 weights`, the shape `tools/replay_kv_stream_policy.py` reads. - The resolve wrapper copies each selection into per-QSA-layer **pinned staging at fixed addresses**, so the captured graphs bake the copies in as memcpy nodes; a flush after the window's `cudaStreamSynchronize` appends the records. Weights are 1.0 (the indexer's scores are not copied). - The staging is allocated at `kv_select_dump_open`, **before any window is captured** - the first build died on exactly that (an allocation inside a capture poisons its stream). ## What it changes and what it leaves alone The resolve wrapper gains a `dump_slot` argument (the two verify call sites pass the QSA ordinal; the prompt path passes nothing), a small dump block in layer.cpp, a flush in verify.cpp after the window sync, and `--dump-kv-select` plumbing in generate.cpp. **Without the flag nothing runs and every answer is unchanged** - the copies are the only new work, and they exist only when the flag is given. ## How it was checked - `kv_stream_parity` **PASS** (q4_0 and k8v4: 0 failures, ring restore identical) on a GTX 1080 Ti (sm_61, `STRATA_EXPERIMENTAL_SM60=ON`), CUDA build with g++-14. - **Live captures** from this box's IQ3_S pack (the deployment's own args + `--dump-kv-select`): a 127-token prompt → a 115 KB trace (348 calls), a 4095-token prompt → a 3.5 MB trace (684 calls, 441,848 block lookups, 12,586 distinct blocks). The engine ran unchanged otherwise (prefill 276 tok/s, decode 20.7 tok/s on this card). - Both traces replay through #2434's bench. On the 4K trace, at a per-layer pool between one call's width (646 blocks) and the working set (~1,050 blocks) the **real selections rank the policies**: | pool/layer | compulsory | clock (today) | lrfu | lru | oracle | |---:|---:|---:|---:|---:|---:| | 768 | 12,586 | 41,878 | **40,046** | 69,929 | 31,552 | | 896 | 12,586 | 21,741 | **21,014** | 23,646 | 16,114 | (misses; lower is better). So on real QSA selections LRFU beats the kernel's CLOCK by 3-4% of misses, LRU is worse than both, and Belady says more is on the table - the numbers #2415 asked for before anyone changes the default. ## Not tested HIP (no toolchain on this box; the .cu sources are shared with the HIP build) and SYCL (a separate engine tree with no dump). The `--batch` window path keeps each layer's last call per window (documented in DETAILS.md). No wall-time claim from the miss counts - that is the live A/B #2415's next step, using the engine's own `KV streaming:` line. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
En el sitio
Enlaces a install, modelos, releases.