#4928 [Engine] long --lookup-max-t windows over a layer split (opt-in, single conversation)
closed · @lyesrock · 0 comentários · No GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows
Descrição
### The problem
On a two-GPU layer split, `--lookup-max-t` long prompt-lookup windows never run: three places clamp a
split to `kVerifyNarrowT` rows - the argument clamp at serve start, the draft policy's size mask in the
request loop, and a verifier guard ("windows past 8 tokens need one stage"). On 2x Tesla P40 (IQ3_S,
ctx 262144) that caps recopies at ~7 accepted tokens a window (20K-token code recopy: 60.3 tok/s),
while the same feature on one card - and on this box's older engine with its `STRATA_LOOKUP=1` - ran
~14 tokens a window at 67-75 tok/s.
### The approach
One opt-in knob, `STRATA_SPLIT_LONG_LOOKUP=1` (single conversation: `--batch <= 1`, no
`--pipeline-windows`). Off, every path stays byte-identical to upstream. Four places, one per clamp:
1. the argument clamp: `o.spec` may stay past `kVerifyNarrowT` when the knob is on;
2. the split's hand-off buffers size to `kVerifyLongT` rows (and `set_handoff_rows` follows);
3. the draft policy keeps its 9..16 window sizes on a single-conversation split;
4. the verifier guard at `src/core/verify.cpp:3257` yields: the two-token-group layout a long window
runs as needs nothing else across stages - the hand-off is read and published whole-window.
### Validation (2x Tesla P40, sm_61, CUDA 12.9, engine built from this branch)
Passed: start logs `window sizes 1,2,3,4,5,8,12,16` and "long lookup windows ON over the layer split";
code recopy 1,744 tokens: 70.7-73.0 tok/s at **14.4-15.0 accepted a window vs 7.0** (102-105 windows
instead of ~195); 157,395-token recopy: 64.1 vs 61.1 (**+5 %**) at 14.4 vs 6.75; agent session decode
+6.5 %; the standard suite (chat/4K/16K) within run-to-run noise; greedy verbatim copies byte-identical
(6,014 chars in/out, every round); knob off reproduces the upstream numbers.
Live traffic since (same box, this build in serve, 100 real requests after start): 971 suffix-draft
windows at **10.5 offered tokens a window, 96 % accepted** - past the upstream cap of 8 in everyday
use, not only in probes; copy-heavy requests decoded 60-72 tok/s against 40-45 for non-repeating
content, with the policy correctly keeping short windows there. One operational note: a request right
after a start reads short while the draft policy learns the long sizes; judge steady state.
Not run: correctness suites (speed only), HIP/SYCL/Metal builds, `--batch` or `--pipeline-windows`
paths (the gate refuses them). A first probe right after a start reads short while the policy learns -
judge the steady state. On one RTX 5090 the same feature (upstream, no this branch) is worth +19 % on
recopies: the win scales with how cheap a 16-row pass is on the card.
No site
Links install, modelos, releases.