Pull requests / #3615

#3615 [Engine] conversation cache: --conversation-cache-min-tokens WRITE[,READ] gates parked reads

open · draft · @G0-0000 · 0 Kommentare · Auf GitHub

Server & APIAMD / HIPNVIDIA / CUDA

Beschreibung

## Problem

`--conversation-cache-min-tokens N` (WRITE, #1597, now in main) stops short side requests from parking themselves and evicting the long conversation they are interleaved with. The read side has no equivalent: every request calls `ConversationCache::best()`, and a short request can restore a short parked copy that it rewrites at once.

## Change

`--conversation-cache-min-tokens WRITE[,READ]`:

- WRITE keeps today's meaning: the shortest outgoing conversation worth a slot (`ConversationCache::wants`).
- READ is the shortest complete prompt, cached history included, worth searching the parked conversations for. Below it the engine calls no `best()` and restores nothing. The active session's prefix, its checkpoints and the batch slots are chosen before the gate and are unaffected.
- A single value is `WRITE,0`, the default is `0,0`, and `READ=0` searches every prompt as before. A startup with READ=0 prints the line it printed before.

This extends #1597 rather than replacing it; that flag is already in main and keeps its meaning.

## Validation

Passed:
- CUDA build on a Tesla V100 (sm_70): `cmake -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=70 ... && cmake --build build-pr -j8`
- `conv_cache_test`: PASS (4792 checks); the same source also passed under `-fsanitize=address,undefined`
- `strata --serve --help` shows the new `WRITE[,READ]` syntax

Not run:
- An end-to-end GPU run of this exact branch. The gate wiring is static analysis plus unit tests on the parser, the boundary and the full-prompt counting.
- HIP and SYCL builds: no such hardware here. The change is in shared code and looks backend-neutral; help welcome.

Failed: none.

## Limits

- READ counts the complete prompt, cached history included, so it only affects short requests; a long conversation's every turn is above any useful READ.
- No disk/spill path is touched: main has no conversation disk tier (#1937 is still open).

## Checks

- [x] Title starts with its area
- [x] One problem per pull request
- [x] Opt-in; the default path and startup line are byte-identical
- [ ] Every backend the change touches was built: CUDA only so far; HIP and SYCL are not available here

Mehr auf der Site

Links zu Install, Modellen, Releases.