#1614 [Engine] shared KV pool for all lanes (--kv-pool-tokens): at 2 x 256k the first token comes 2-5x sooner and lane moves ~10x faster; off = identical to main

open · @noon-at-cgn · 0 コメント · GitHub で見る

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationWindowsLinux

本文

Branch `pr/kv-pool-141`, head `1e1cd762`: 20 commits on upstream main `41cf05a0` (v0.2.5-6, engine 0.1.50). Opt-in: without `--kv-pool-tokens` the output is byte-identical to main (identity gate below). Scope issue: #3351.

## Problem and result

With KV streaming (`--kv-resident`), the main session and every `--batch` slot pin the K/V of the whole `--max-context` in RAM, however short the conversation: 3.09 GiB each at 262,144 cells with `--kv int8`. A conversation that moves between the main session and a slot is copied, which at 256k takes seconds, and two long lanes push a RAM-limited host to its memory guard.

`--kv-pool-tokens N` pins N cells once. Sessions hold only the 4,096-cell chunks their conversation has reached, and a move between lanes swaps chunk tables instead of copying.

**Two lanes of about 256k tokens** (2 x RTX 3080 20 GB, UD-Q4_K_XL, same binary in both arms, only the flag differs; measured on 0.2.0-beta, engine 0.1.45.3):

| | with the pool | without the pool |
|---|---|---|
| First token of an appended short question (both lanes) | **1.1-1.2 s** | 2.3-6.7 s |
| Both lanes decoding an appended turn, aggregate | **59.1 and 61.5 tok/s** | 25.5 and 37.7 tok/s |
| Moving a ~256k conversation to or from a slot | **0.36-0.5 s** (`moved`) | 3.5-5.2 s (`copied`) |
| Host memory guard (stops the engine below 1.5 GiB `MemAvailable`) | never fired | **stopped the engine in 2 of 3 starts** |
| Idle `MemAvailable` (mean of 3 starts each) | 14.5 GiB | 11.4 GiB |

At about 133k per lane the move took 0.1-0.25 s with the pool and 0.9-2.0 s without. With the pool a third lane (three conversations of about 104k) fits, every needle answered.

**Short context is unchanged:** one stream 84.5 vs 81.9 tok/s, two streams 93.7 vs 89.8 (means of three starts each, within the round-to-round spread). The gains are at long context with more than one lane.

On this PR's head (0.2.5) the pool runs the same workload end to end (2 x 256k, numbers under Validation). I did not repeat the no-pool arm on 0.2.5, so the comparison above is the 0.2.0-beta one.

## Related discussion

- Scope issue: #3351.
- #1011 (xdbxdbx): the original pool. His commits are first in this branch with his authorship.
- #1962 (share identical KV prefix blocks): a different mechanism, complementary.
- #2526 (park an idle slot before reuse): the same idea as our checkpoint-chain commit, which exists only with the pool. It will conflict textually; I will rebase on whichever lands first.
- #3251 (restore a session file without its K/V in RAM): needs the same chunk translation in `conversation_snapshot.cpp`.
- #2518 (turn-start rule in `pick_slot`): our slot-choice change is not in this PR.

## Approach

A session keeps a chunk table, a device copy for the kernels and a host copy for DMA and copies. Unreached chunks point at one spare "trash" chunk. Without the flag the table is the identity layout. A lane reserves chunks as it writes. When the pool is full, an idle slot's cached conversation gives way first, then a new request gets a retryable 503 (`kv_pool_full`), and a decoding lane that cannot grow ends with `length` and `"truncated": true`. It works across a layer split: each card pins its own layers' share and a conversation uses the same chunks on every card. The MTP drafter ring is not pooled.

- Every host-side K/V reader and writer goes through the chunk table (`KvHostPools::block`, `block_host`, `for_each_piece`): append kernels, resolve and stage copies, verify, MTP, prefill, state hash, `conversation_kv_copy`, `conversation_kv_restore_prefix`, session files, K8V4 halves. I checked the 967 upstream commits since the last base for new host K/V sites and found none.
- With the pool, the batch slots borrow the main session's RoPE table (64 MiB per slot at 262k). Without it each slot keeps its own, as on main (`1e1cd762`; see Failed).
- The checkpoint chain moves with its chunks (own commit), so a client that edits its last question still finds a checkpoint below the edit.
- The pool's host allocation goes through `kv_host_alloc` (WSL2 fallback, pin budget).
- Docs: `docs/BATCHING.md`, `docs/DETAILS.md`, `docs/MULTI_GPU.md`.

## Validation

Head `1e1cd762` unless a line says otherwise. `ebb61958` is the same series before the last two changes (the RoPE gate, and `(void) cudaFree` in `kv_pool.cpp`; that file compiles to identical machine code either way).

**Passed:**

- **Flag off = main, byte for byte.** Main `41cf05a0` vs this head without `--kv-pool-tokens`, production config otherwise (2 x RTX 3080 20 GB, layer split 23, `parallel 2`, `--spec 4 --batch-mtp`, `--max-context 262144 --kv int8 --kv-resident`). Greedy (temperature 0, top_k 1, seed 1234), 256 tokens, prompts of 989, 31,280 and 128,209 tokens, same bytes for both. The stock config is not reproducible even main against main (adaptive experts), so both arms ran the deterministic settings from `docs/DETAILS.md` (`--adapt-every 100000 --adapt-async 0 --pcie-frac 0`) with `STRATA_STATE_HASH=1`. Two starts of each: all three outputs byte-identical to main's, the STATE_HASH fields equal, the expert cache at 4893 / 4161 slots like main's. At `ebb61958` the same held with `--spec 2`.
- **Pool on, 2 x 256k, end to end** (production config with `--kv-pool-tokens 524288`, one start): `KV pool: 524288 cells shared by the sessions and the stages of the layer split, 6.24 GiB of pinned RAM`. Two concurrent 256k prompts, then appended turns: 4/4 needles correct, slot moves 444-842 ms, appended-turn decode 53.0 tok/s aggregate, single-lane questions 43.5 and 50.5 tok/s, no `kv_pool_full`, host `MemAvailable` never below 17.2 GiB.
- **CUDA build** (CUDA 13.3, sm_86, `-DSTRATA_ENABLE_CUDA=ON -DSTRATA_NATIVE_EXPERTS=ON -DSTRATA_MMQ_KQUANTS=ON`): rc 0, same warning count as main.
- **CPU tests** (`CUDA_VISIBLE_DEVICES=-1`): kv_chunks_test; conversation_cache_test (4791); conversation_memory_test (31); conversation_file_test (201); conversation_validation_test (972); conversation_transfer_test (1548 host-only); session_save_reclaim_test (119); memory_guard_test (134).
- **GPU tests:** `ctest -j1` at `ebb61958`: 161 of 168 passed, 4 skipped (CPU without AVX-512, ggml without trellis types), 3 failed, all pre-existing (Failed). At `1e1cd762` the pool-related subset again: 17/17 (conversation_snapshot_test with the pooled variant, kv_chunks, the kv_* parity tests, batch_rows, verify_batch_parity).
- **Serve tests** at `ebb61958`: `python -m unittest discover -s serve -t .`: 1050 tests, 15 skipped, OK.
- **HIP build** at `ebb61958` (`rocm/dev-ubuntu-24.04:7.2.4`, AMD clang 22 / HIP 7.2, flags of `packaging/build_hip_linux.sh`): `strata` for gfx1100 and for gfx1151;gfx1201, and all targets with tests on, rc 0 on the branch and on main. The branch adds one test (`kv_chunks_test`). The only new warning, `cudaFree` result unused in `kv_pool.cpp`, is fixed on this head. `kv_chunks_test` and `conversation_cache_test` (4791) pass on the host.
- **SYCL build** at `ebb61958` (oneAPI DPC++ 2026.1.1, flags of `packaging/build_sycl_linux.sh`): `strata` and all targets, rc 0, warnings identical to main. The SYCL tree compiles its own migrated copies and does not build the pool: of the touched files only `layer.hpp`, `kv_stream.hpp` and `conversation_cache.hpp` are compiled there, and they build cleanly.

**Failed:**

- At `ebb61958`, flag off was **not** identical to main: the batch slots borrowed the RoPE table without the pool too, so each slot was 0.07 GiB smaller, `--expert-cache auto` cached 43 more experts per card, and greedy output differed. With a VRAM reserve that equalised the expert count the outputs matched, which located the cause. `1e1cd762` borrows only with the pool; the gate above is from that head.
- Pre-existing, also on main: `ple_parity` (its Q2_0 GGUF is not on this host), `ggml_tq_overlay` (cmake script "Unknown arguments", same on main), `isa_guard_check` (`iq512x_rows` violation, same on main's library). This branch does not touch `src/kernels/cpu`, `cmake` or `tests/cmake`.
- CI `metal-link` fails in `metal/src/mtl_rt.mm` (`MTLGPUFamilyMetal4`, `MTLCompileOptions.mathMode`: macOS 26 API, in main since `ad9eb890`) on the `macos-14` runner's SDK. Other PRs on current main fail the same job; this branch does not touch `metal/`.
- CI `Linux / runtime requirements only` at `1e1cd762`: 1051 serve tests, 1 failure, `test_lifeline.test_the_portable_watcher_ends_the_child_when_the_server_is_killed` ("the child outlived a killed server (watcher)"). It depends on how the tests are started: on the lm-server box it fails on main `41cf05a0` too, 5 of 5 runs from an ssh shell (`python -m unittest serve.test_lifeline`), while the whole serve suite above passed when run from a systemd unit. This branch does not touch the lifeline code or its test.

**Not run:**

- HIP and SYCL hardware (no AMD or Intel GPU here); HIP and SYCL builds of `1e1cd762` (the change since `ebb61958` is 3 lines of host C++ in `generate.cpp`, which SYCL does not compile, plus the `(void)` cast); Metal; Windows and the WSL2 `cudaHostRegister` path.
- The full ctest and the serve tests at `1e1cd762` (subset and CPU tests only); the no-pool 256k arm on 0.2.5.
- Identity of two *concurrent* lanes: main against main already differs there under the deterministic settings, so there is nothing to compare against.
- `STRATA_SLOT_CKPT_MOVE=1` with the pool; `STRATA_SLOT_COPY_DIRECT` and `borrow` end to end; session-file `SAVE`/`RESTORE` with a pool end to end (unit tests only); pool-full paths under load; more than 3 lanes; a single 512k conversation.

All raw outputs (prompts, per-request JSON, engine logs, binaries' sha256) are kept and can be attached.

## Limits

- The pool is for long conversations on more than one lane. At short context nothing measurable changes.
- Measured on one box (2 x RTX 3080 20 GB, one model and quantization). The no-pool 256k timings on 0.2.0-beta are one full run and two partial ones, because the host guard stopped two of three no-pool starts.
- N is the operator's choice; too small a pool means refused requests or shortened answers. Setup and launch do not know `--kv-pool-tokens`; pass it in `args`.
- The MTP drafter ring is not pooled. CUDA only was run on hardware.

関連リンク

インストール・モデル・リリースへのサイト内リンク。