#6010 [Docs] conversation cache: what more clients than slots costs, with a measurement (eight agents on 2x P100: 792 s -> 454 s with 16 slots)

closed · @Keoian · 0 Kommentare · Auf GitHub

Multi-GPUNVIDIA / CUDAModels & quantsDocumentation

Beschreibung

## Problem and result

The settings table says what `--conversation-cache-slots` and `--conversation-cache-mib` are, but not what a user sees when more clients take turns than there are slots. With the defaults (4 parked conversations, at most 8192 MiB) a fifth agent pushes the oldest conversation out, and that agent reads its whole prompt again when it returns. Nothing in the reply says so; it only looks slow.

This PR adds one sentence to the settings row and one row with a measurement to the History table. No code changes, no default changes.

Measured on 2x Tesla P100 16 GB (sm_60, `--layer-split 24`, both cards power-capped at 150 W in every sample), 2x Xeon E5-2696 v3, 125 GiB RAM, Qwen3.8-Flash-Next (a Q4_K_XL / IQ4_NL pack), `--max-context 262144 --kv k8v4 --spec 4`, MTP on, `--prompt-cache-spread`. Eight agents, each with its own first line of the system prompt and the same 8 tools, take turns on one server, one request at a time; fresh engine per column:

| request | defaults (6 checkpoints, 4 parked, 8192 MiB) | `--prompt-cache 12 --conversation-cache-slots 16 --conversation-cache-mib 16384` |
|---|---|---|
| round 1, each of 8 agents (9,812 to 9,818 tokens) | all read, 37.0 to 39.3 s | all read, 37.9 to 39.2 s |
| round 2, each agent's next turn (12,150 to 12,683 tokens) | 0 reused, all read, 49.9 to 52.4 s | 9,807 to 9,813 reused, 2,339 to 2,872 read, 12.8 to 14.9 s |
| agent 1 once more (15,621 tokens) | 0 reused, 62.3 s | 12,678 reused, 15.6 s |
| agent 6 once more (15,200 tokens) | 12,154 reused, 17.0 s | 12,154 reused, 16.9 s |
| all 18 requests | 196,234 tokens read, 792 s | 105,080 read, 454 s |
| evictions (engine log) | 19 | 0 |

The codes planted in the first messages of agents 1 and 6 came back right in both runs (the two last requests).

The same two engines also ran one coding agent alone (36 turns to 99,389 tokens, three jumps back, one cleared tool result, 40 requests): 140,599 tokens read in 645 s with the defaults, 130,549 in 648 s with the larger limits. The two rewrites far back resumed nearer the edit with 12 checkpoints (10,846 -> 8,210 and 28,003 -> 20,597 tokens read) and an ordinary turn took 0.2 to 0.8 s longer (parking: 17.2 s over 32 snapshots against 23.9 s over 31).

## Validation

**Passed:**
- The two sessions above, one run of each column. With the larger limits the cache held at most 14.5 GiB, the lowest `MemAvailable` was 11.2 GiB, no parking was skipped and swap use did not change.
- The new sentence's link target `#history--older-measurements` exists and is already used elsewhere in the file.

**Failed:** none observed.

**Not run:**
- A second run of the defaults (no run-to-run floor for the times; the token counts follow from the cache's rules).
- An exactness comparison. Adaptation was on and the two columns read different amounts of each prompt, so the 4-token replies differ (other tokens in seven of the rotation's replies, largest logprob difference 1.18); that compares a whole read against a restored conversation, not two settings.
- An upstream release build. The engine is 0.1.57 built from a fork: main at 9f7d977d plus the commits of #5436, #5438, #5495 and #5986 and one prompt-read stop change, with `STRATA_SPLIT_BORROW=1` set. No borrow happened in the rotation (the agents' system prompts differ from the first line).
- The three limits one at a time: the slots, the budget and the checkpoint count were changed together.

## Limits

One machine, one seed. Agents of 10K to 16K tokens: longer histories fill the budget sooner, and then the oldest leaves as before. The budget is host RAM.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01N976Q2e2TARptyrfV8H8mz

Mehr auf der Site

Links zu Install, Modellen, Releases.