#6009 [Docs] speed projection: a request that switches reads its prompt again, the other setting's parked conversation stays (measured on 2x P100)
open · @Keoian · 0 Kommentare · Auf GitHub
BenchmarksMulti-GPUNVIDIA / CUDAModels & quantsSecurityDocumentation
Beschreibung
## Problem and result
`docs/DETAILS.md` says of the experimental speed projection's per-request switch: "Switching drops the conversation cache once." I read that as: clients with the vector on and clients with it off cannot share a server without re-reading their prompts at every switch. That is not what the engine does. It looks for a parked conversation read with the request's own setting (`conversations.best(ids, req_imgs, want_cvec)`), so each client finds its own conversation again; only a conversation that itself changes setting reads its prompt again.
This PR rewords that sentence and adds one row with the measurement to the History table. No code changes.
Measured on 2x Tesla P100 16 GB (sm_60, `--layer-split 24`, both cards power-capped at 150 W), 2x Xeon E5-2696 v3, Qwen3.8-Flash-Next (a Q4_K_XL / IQ4_NL pack), `--max-context 262144 --kv k8v4 --spec 4`, MTP on, `--prompt-cache 12 --conversation-cache-slots 16 --conversation-cache-mib 16384`, the vector loaded with the four documented arguments and `"sampling": {"experimental_speed_projection": false}`. Agent X sends nothing (off), agent Y sends `"experimental_speed_projection": true`; one server, one request at a time:
| request | prompt tokens | reused | read |
|---|---|---|---|
| X1: agent X, off | 9,806 | 0 | 9,806 |
| X2: X's next turn | 12,602 | 9,801 | 2,801 |
| Y1: agent Y, on | 9,806 | 0 | 9,806 |
| Y2: Y's next turn | 12,781 | 9,801 | 2,980 |
| X3: X returns | 15,008 | 12,597 | 2,411 |
| Y3: Y returns | 15,552 | 12,776 | 2,776 |
| X4: X returns (explicit `false`) | 17,397 | 15,012 | 2,385 |
| X5: X's same prompt with the vector on | 17,397 | 0 | 17,397 |
| X6: X's same prompt, off again | 17,397 | 17,392 | 5 |
| Y4: Y returns | 18,354 | 15,556 | 2,798 |
## Validation
**Passed:**
- The session above (engine log: `restored 12597 tokens (checkpoint)`, `restored 12776 tokens (checkpoint)`, `restored 15012 tokens (live)`, `restored 17392 tokens (checkpoint)`, `restored 15556 tokens (live)`; no eviction).
- Loaded and off against an engine started without the vector, both with `--adapt-every 0`, five greedy 8-token requests with logprobs (three short chats, an agent's 9,806-token first request with tools, its next turn of 12,602 tokens): largest logprob difference 0.000000, same tokens. The same five with the vector on: other tokens on both agent requests, largest difference 0.46.
- The engine log on the layer split: `control vector mode = project, dir = per-layer, layers 4..44 (41 steered)`; VRAM free with everything loaded is the same with and without the vector (789 / 597 MiB).
- A 400-token greedy reply four times, off / on / off / on: 42.6, 43.2, 43.0, 42.4 tok/s.
- The new sentence's link target `#history--older-measurements` exists and is already used elsewhere in the file.
**Failed:** none observed.
**Not run:**
- An upstream release build. The engine is 0.1.57 built from a fork: main at 9f7d977d plus the commits of #5436, #5438, #5495 and #5986 and one prompt-read stop change, with `STRATA_SPLIT_BORROW=1` set. No borrow happened in this session (the two agents' system prompts differ from the first line). `conversations.best(ids, req_imgs, want_cvec)` is the same line on main (`generate.cpp`).
- One GPU, `--batch`, pictures, the conversation cache off (then there is nothing parked and the old sentence describes it).
- How much the vector changes answers or refusals.
## Limits
One machine, one seed, one scripted session. The row's speed numbers are four replies and do not show a difference either way.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01N976Q2e2TARptyrfV8H8mz
Mehr auf der Site
Links zu Install, Modellen, Releases.