改动 / #3615
#3615 [Engine] conversation cache: --conversation-cache-min-tokens WRITE[,READ] gates parked reads
open · draft · @G0-0000 · 0 评论 · 去 GitHub 看
Server & APIAMD / HIPNVIDIA / CUDA
说明
## Problem `--conversation-cache-min-tokens N` (WRITE, #1597, now in main) stops short side requests from parking themselves and evicting the long conversation they are interleaved with. The read side has no equivalent: every request calls `ConversationCache::best()`, and a short request can restore a short parked copy that it rewrites at once. ## Change `--conversation-cache-min-tokens WRITE[,READ]`: - WRITE keeps today's meaning: the shortest outgoing conversation worth a slot (`ConversationCache::wants`). - READ is the shortest complete prompt, cached history included, worth searching the parked conversations for. Below it the engine calls no `best()` and restores nothing. The active session's prefix, its checkpoints and the batch slots are chosen before the gate and are unaffected. - A single value is `WRITE,0`, the default is `0,0`, and `READ=0` searches every prompt as before. A startup with READ=0 prints the line it printed before. This extends #1597 rather than replacing it; that flag is already in main and keeps its meaning. ## Validation Passed: - CUDA build on a Tesla V100 (sm_70): `cmake -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=70 ... && cmake --build build-pr -j8` - `conv_cache_test`: PASS (4792 checks); the same source also passed under `-fsanitize=address,undefined` - `strata --serve --help` shows the new `WRITE[,READ]` syntax Not run: - An end-to-end GPU run of this exact branch. The gate wiring is static analysis plus unit tests on the parser, the boundary and the full-prompt counting. - HIP and SYCL builds: no such hardware here. The change is in shared code and looks backend-neutral; help welcome. Failed: none. ## Limits - READ counts the complete prompt, cached history included, so it only affects short requests; a long conversation's every turn is above any useful READ. - No disk/spill path is touched: main has no conversation disk tier (#1937 is still open). ## Checks - [x] Title starts with its area - [x] One problem per pull request - [x] Opt-in; the default path and startup line are byte-identical - [ ] Every backend the change touches was built: CUDA only so far; HIP and SYCL are not available here
本站相关内容
相关页面的快捷入口。