#3919 [Engine] kv: size the resident ring from the run's depth (opt-in, STRATA_KV_RING_ADAPT)

open · @xyzzing · 0 评论 · 去 GitHub 看

BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

说明

Supersedes #3168 (closed in the 0.2.0-beta tracker reset). The /reopen check on #3168 **passed** but GitHub itself would not reopen it (the branch was force-pushed while closed; the bot's reply names this case and points at a linked new PR). The 0.2.0-beta measurements and the discussion stay in #3168; this PR's tables were measured on v0.2.3.1 (build 0.1.48.1) and fold in one real finding (the 64K regression, see Why commit 3).

**2026-11-21:** rebased onto current main (3142b66b, v0.2.17-era, engine 0.1.63) — the branch had gone CONFLICTING because main moved `src/`/`include/` to `engine/src/`/`engine/include/` (the 0.2.16 layout move) and then took the 0.2.17 release. All 5 commits replayed with 0 conflicts, paths re-anchored to the new layout; per-commit diffstat identical to the previous head; the settings-register commit regenerated on the new layout (`terminal/data/settings-register.json`, read-site line numbers re-scanned, register tests 8/8 + docs check green); 723-target gfx1100 HIP build rc=0 on the new tree. The tables below stay as labeled (the v0.2.3.1 measurements; the 0.2.9-era **Passed** block).

The branch is now rebased onto main (c77572806, v0.2.15-era). **2026-11-17:** all 5 commits replayed with 0 conflicts; per-commit diffstat identical to the previous head; `layer.hpp`/`layer.cpp` byte-identical at the tip (upstream touched neither since the 11-06 base); 668-target gfx1100 HIP build rc=0 on the new tree. (Build-config note: the Fedora 45 upgrade removed the system ROCm's hip CMake config, so the local build configures against a vendored nightly-SDK prefix — CI's own HIP job is the independent check.) The tables below are from v0.2.3.1 (build 0.1.48.1); the new base carries its own verification, see **Passed**.

## Problem and result

The resident K/V ring is one global value (`--kv-resident`), and the QSA KV pool is sized by it, so a ring chosen for depth is paid by every run. On this fork's card (RX 7900 XTX, gfx1100, IQ3_S pack) a flat raise is a measured loss at shallow depth: ring 32,768 -> 49,152 at 1K cost 40 % of the MTP path (acceptance 207/207 -> 149-156/215) while being worth +11.7 % @32K / +21.6 % @128K on 0.2.0.

This PR sizes the ring from the run's depth: `STRATA_KV_RING_ADAPT=1` makes the configured `--kv-resident` the cap; a shallow run keeps the live 32,768, a deep one grows toward the cap. Unset (or =0) restores today's behavior exactly. On v0.2.3.1 the adaptive ring is worth **+23.9 % decode at 128K** and is a verified no-op at every tier it does not grow on:

| tier | static 32768 decode | adaptive decode | delta | acceptance static / adaptive | ring the adaptive arm ran |
| --- | --- | --- | --- | --- | --- |
| 128K x3 | 53.28 tok/s | **66.03 tok/s** | **+23.9 %** | 21/29 vs 32/40 | 49,152 (grew) |
| 64K x2 | 74.44 tok/s | 74.34 tok/s | -0.1 % (no-op) | 39/50 both | 32,768 (kept floor) |
| 1K control | 129.81 tok/s | 140.76 tok/s | same-config acceptance-variant noise (197/197 vs 207/207) | | 32,768 both |

Against a static 49,152 (what a user who raised `--kv-resident` by hand would run) the flag is worth **+22.8 % at 1K** (drafter acceptance back to 197/197, 207/207 from 167/173 and 147/204), -2.1 % at 32K (n=2, acceptance mix differs), and a no-op at 128K (the rule grows there, so both arms run the cap).

## Related discussion

#3168 (this PR's predecessor: the 0.2.0-beta tables, the discussion, and the /reopen attempt). #2450/#2630/#3166 (the resident-KV setup rule, the page-pool eviction policy, and the shallow-depth drafter interaction, carried over from #3168's body).

## Approach

Five commits:

1. `qsa_kv_ring_adapt(g, cells_needed, cap)` (+ declaration in `layer.hpp`): opt-in sizing, `--kv-resident` becomes the cap. The hook in `generate.cpp` runs before the session arena is sized and the graphs are captured, because `qsa_state_bytes` -> `kv_pool_bytes` sizes the pool by slots.
2. The drafter-aware growth rule: a shallow run keeps the floor (the engine's historical default 32,768 - with the allocator minimum 20,480 an earlier version measured -7.3 % at 1K), a deep one grows.
3. **The growth threshold is 2x the cap.** The re-measure for this reopen found the original `needed >= cap` form **regresses 64K by -16.5 %** on that base (74.8 -> 62.5 tok/s, n=2 interleaved, acceptance 39/50 vs 27/41): at 64K the grown ring's draft windows cost more than they return. 128K in the same window still gained +23.3 %. The crossover sits between 64K and 128K, so the rule grows only when `needed >= 2 * cap` - every measured tier below the crossover keeps the live 32,768 (verified per-cell: identical acceptance both arms), and a context of twice the cap or deeper still receives the cap. The 0.2.0-era table in #3168 predates this amendment and stands as history, not as the current claim.
4. `a6458d82` fixes two things found after the measurements above. `STRATA_KV_RING_ADAPT=0` used to turn the option **on** (the code only checked that the variable was set), while this body said unset or =0 restores today's behavior; it now reads the variable once and treats unset, empty or 0 as off, any other non-zero value as on. Second, serve mode sized the ring from the CLI prompt plus `--max-new`: under `--serve` the CLI prompt is empty and sessions live for the whole process, so a serve deployment with a cap above the 32,768 floor kept the floor at every depth instead of getting the measured deep-context behavior. `--serve` now sizes from `--context`; one-shot `generate` keeps sizing from prompt plus requested tokens. The engine's stderr line still starts with `strata generate: KV ring adaptive:` and now names where the depth came from, `serve context <n>` or `prompt <n> + <m> new`.
5. `d202fb5c` adds a row for `STRATA_KV_RING_ADAPT` to the KV cache table in docs/DETAILS.md: opt-in, `--kv-resident` becomes the cap, the 32,768 floor, the 2x-cap rule, generate versus `--serve` sizing, unset or 0 as off, and the RX 7900 XTX measurements.

The branch is rebased onto current main (0.2.8, engine 0.1.54); the first three commits replayed with no conflicts. Files touched: `include/strata/core/layer.hpp` (+3), `src/core/layer.cpp` (+21), `src/program/generate.cpp` (+21), `docs/DETAILS.md` (+1).

## Validation

**Passed:**
- Same-binary interleaved cells on v0.2.3.1 + these 3 commits (build 0.1.48.1), RX 7900 XTX, TheRock 10.2 nightly, production-shaped args (`--spec 5 --kv int8 --lookup-chain 2` etc.), greedy one-shot through the engine's own prefill/decode counters; the ring the engine actually ran is recorded per cell and engagement prints only when the ring changes, so a silent fallback cannot masquerade: tables above; raw logs `r32-cell-*`, `r32b-cell-*`, `r32c-cell-*`.
- On-card parity on the PR binary: `iq_parity`, `iq_multi_parity`, `s2_expert_grouped_parity` PASS.
- Cross-binary confirmation of commit 3 on the fork's production line (old binary = needed>=cap vs new = 2x cap, interleaved 64K x3): 61.32 -> 75.52 tok/s median, acceptance 27/41 vs 39/50 - the same direction and magnitude as on the PR binary.
- The pre-amendment functional ring code is byte-identical to what has been running in this fork's production since 2026-10-19.
- Metal link (CI): passed after the rebase (1m47s, run [41064393814](https://github.com/Niko1221/Strata/actions/runs/41064393814)); the check that was red before this push ran the old `macos-14` job this branch predated.

**Failed:**
- The original `needed >= cap` threshold at 64K on v0.2.3.1: -16.5 % (fixed by commit 3; kept here as the finding).

**Passed (on the new base, for the record):**
- The chain applies clean on the v0.2.9 tag, and a 32-cell paired one-shot window on gfx1100 (IQ3_S GSQ-RCO, same-window ABBA with an identical-env control arm, real-text workloads at 32K/128K, threshold max(2%, 2x SD of the control)) held the -5% no-regression guardrail on decode / prefill / ms-per-round at both depths with this chain in. The window's gains are the upstream base's own (decode +7-8%, ms-per-round -7-10% vs our 0.1.48.1 production build); the point is composition and no-regression, not attribution to this PR.
- The rebased tree builds on main (9f7d977d2): 549 targets, rc=0, gfx1100 HIP, pinned-SDK toolchain.
- Upstream moved qsa_kv_resident_min() from 20,480 to 4,096 since the tables were measured; this PR's floor is max(qsa_kv_resident_min(), 32,768), so the floor stays 32,768 and explicit --kv-resident deployments are unaffected.

**Not run:**
- Re-measurement of this PR's isolated effect on main or 0.2.9 (the chain was validated composed, per the refresh). **Staged, not yet run**: the A/B campaign is frozen (adaptive vs flat ring, 5 depths x 2 tracks x 2 reps ABBA with an identical-env control, TTFT captured per cell; plan fingerprint `907376df`); it needs one >=65 GB free-memory window on the measurement box and lands in this body as a table when it runs.
- CUDA and SYCL builds on the new head: the sycl/ tree is a separate Intel port; docs/INTEL.md says no shared upstream file is changed.
- Metal execution: no hardware for it here; the change is host-side sizing on the QSA/KV path, and Metal is covered by CI.
- Contexts beyond 262K, other models/packs, other AMD cards: the rule's constants (floor 32,768) are this card's measurements; the mechanism is shape-generic.


## Current-base figures (2026-11-06, the refresh line this PR now rides)

Measured on the composed base this PR rebases onto (upstream v0.2.9, engine 0.1.55 + the replayed fork chain, RX 7900 XTX 24 GB, gfx1100, IQ3_S GSQ-RCO, production environment, Linux): the public same-protocol replay (exact-token synthetic-record prompts, three secret codes, retrieval-validated, 256 output tokens, temperature 0, seed 42, no prompt cache, median of 3; 21/21 requests passed; `TTFT` = time to first output token, which at these lengths is the prompt read + ~1 s):

| input tokens | TTFT (s) | decode (tok/s) | prefill (tok/s) |
| --- | --- | --- | --- |
| 1,024 | 1.6 | 90.4 | 645 |
| 2,048 | 2.0 | 94.5 | 1,020 |
| 4,096 | 2.5 | 84.0 | 1,647 |
| 8,192 | 3.6 | 94.3 | 2,307 |
| 16,384 | 6.9 | 89.7 | 2,401 |
| 32,768 | 13.6 | 90.2 | 2,419 |
| 65,536 | 27.1 | 83.3 | 2,421 |

Decode is flat with context (83-95 tok/s from 1K to 64K; the paired 32-cell gate window on the same base measured decode +7.1 % at 32K and +8.4 % at 128K against the 0.1.48.1 production build, ms-per-round -6.7/-10.3 %, prefill +1.3/+0.6 %, control noise +/-1.3 %).

### Community context (IQ3_S rows from this repository's community table)

| card | CPU | engine | prefill | decode |
| --- | --- | --- | --- | --- |
| RTX 4090 (this table, #624) | Ryzen 9 7950X | 0.1.38 | 2,059-3,078 | 76.1-102.5 |
| RTX 5070 Ti (#832) | Ryzen 7 9800X3D, Win | 0.1.39 | 3,012 (5K) / 3,487 (41K) / 3,523 (131K) | 75.8 / 74.7 / 82.8 / 79.6 |
| RTX 5090 (#811) | Ryzen 7 9700X, Win | 0.1.39 | 3,072 | 164.3 |
| RX 7900 XTX, ROCm nightly + the gfx1100 hipBLASLt table (#745 lineage) | - | 0.1.38 | 926 -> 1,687 (128K) | 62.3 |
| this PR's measurement box (the table above) | Ryzen 9 7900X | 0.1.55 + this chain | 645-2,421 | 83.3-94.5 |

Not like-for-like: the community rows are 0.1.38/0.1.39-era engines (five-plus releases behind this base), mixed Windows/Linux, and different CPU/RAM tiers; they are cited for context only. On decode, the fresh 0.1.55 figures sit at or above the same-quant 4090 rows at every measured length; on prefill at depth this box remains behind the 5070 Ti/5090 rows (a discrete-VRAM streaming boundary this PR does not address - the ring sizing here is about the decode path's cost at shallow depth, not prefill throughput).

## Limits

The cap a user picks must still be one their card can hold (here 49,152 int8 cells on 24 GB beside the other allocations). The 2x-cap threshold is calibrated by measurement between 64K and 128K on v0.2.3.1; a deeper re-measure on a future base may move it.



**2026-11-17, later the same day:** upstream's new settings register (data/settings-register.json + tools/test_settings_re

本站相关内容

相关页面的快捷入口。