#4232 [GUI] API monitor: show the tokens held in the engine's KV cache
open · @qwertie · 0 comentários · No GitHub
Server & APIAMD / HIPModels & quantsDocumentationWindowsLinux
Descrição
## Problem and result The API monitor shows each request's body, output and reasoning, but there is no way to view entire context windows, or to see the requests in the form the model sees them. This PR lets users see, for each slot in VRAM, the raw tokens the engine holds in its KV caches. Seeing them answers questions the request bodies can't. For example, - when I started a conversation and said "hi there", using this feature I could see why the context window was at 37K in T3 Code (the API monitor showed less than half of this context) - a Codex conversation's cache began with `<|im_start|>system Reasoning effort is set to xhigh...`, which is a bad practice because it forces a change of effort to re-read the whole conversation (unless `"effort_position": "end"` is set). - having configured 3.8-flash-next for two parallel conversations, I saw there are three KV caches in VRAM even though there are never more than two concurrent conversations (that's gigabytes wasted) - when an agent spawns a serial sub-agent, the solo KV cache is swapped out to RAM, leaving two of three context windows empty! This PR adds a **Context in VRAM** panel to `/api-monitor`, so it's on only with `"api_monitor": true` and nothing changes otherwise: ```text Context in VRAM ├── Refresh GET /api/context ├── session picker with --batch: the solo session, then each slot ├── Plain text the cached tokens as text ├── Colored tokens one background per token; the colour comes from the token id, │ so a token always has the same colour; tooltip: position and id; │ 8,000 tokens a page └── Save .txt [with token ids] the text, optionally followed by the ids as uint32 LE ``` The main page links to the monitor (Sampling panel, Monitor tab) only when it is on. `/health` adds `"api_monitor": true` only then, so its default answer is unchanged. ## Related discussion none ## Approach by Opus 5.5 The engine has no command that returns its cache, so the server mirrors it from what it sends and reads. Every `GEN` carries the whole prompt, so afterwards the cache holds the prompt plus every printed token except the last, which is never fed back. I checked the rules against `generate.cpp`, `q36_engine.cu` and `ds4_main.cpp`: - Tokens printed after a `STOP` count too. DeepSeek's engine has also fed its last printed token when it honours a `STOP` (`stop_feeds_last`). - An `ERR` keeps the previous list. A stop while the prompt is being read, a restored disk session, non-finite logits, or a lost engine make the list unknown, and the panel says why. - With `--batch`, a request that is alone runs on the solo path. `generate.cpp` reads every admission in its own session before copying it into a slot, so that session is listed next to the slots: three sessions with `--batch 2`. The Qwen3.6 path has no session of its own (`own_solo_session`). Colours: a fixed, seeded palette of 4,096 per theme, sampled evenly by OKLab volume over the colours that keep 7:1 contrast with the theme's text. A hash of the id picks the colour, so muted colours, which take up less of that space, come up less often. The page builds the palette once (about 5 ms) and gives each span both themes' colours. The coloured view is paged because painting 150K spans into one `<pre>` froze the tab. Compatibility: with the monitor off, request handling is unchanged apart from keeping this list of ids per request (one list copy of the prompt, next to the comma-joined prompt the server already builds). ## Validation **Passed:** - `python -m unittest discover -s serve -t . -p "test_*.py"` on c70189fa, Windows 11, Python 3.14.6: 1048 tests OK (21 skipped). This includes the new tests: the end-of-request rules (prompt and all but the last token, tokens drained after a STOP, DeepSeek's STOP, a stop while the prompt is read, ERR, non-finite logits), the endpoint and its key check, per-token text with a character split across tokens, and with a fake `--batch` engine a lone request in the solo session and two at once in the slots. - `node --check serve/web/monitor.js serve/web/app.js` on c70189fa. - Real engine, Windows 11, Radeon 8060S (Strix Halo, gfx1151), Qwen3.8-Flash-Next IQ3_S, 204,800 context, `"parallel": 2`, with Codex and Claude Code as clients. Run on this branch's server code: the later commits change only `monitor.html`, `monitor.js` and the docs. The panel showed the solo session after a lone request, including a sub-agent's conversation while its parent waited (both slots empty), and the full history after a client-side compaction. The colored view of a 150K-token cache: 517 ms to the first page, 58 ms per page change (a Chromium-based browser). **Failed:** none observed **Not run:** - The DeepSeek and Qwen3.6 paths on hardware: only their rules were read from the engine source (`stop_feeds_last`, `own_solo_session`); the tests use stub engines. - Linux and macOS browsers. ## Limits The list is the server's mirror, not a dump of the GPU's memory. Where it differs (also in docs/DETAILS.md): - With draft tokens (MTP, lookup), an answer that ends inside a run of accepted drafts leaves a few more tokens in the cache than the list shows. On Qwen3.8-Flash-Next with MTP, both replies I checked this way ended with `<|im_end|>\n<|endoftext|>` already in the cache. - After a restored disk session, or a stop while the prompt is read, the list is unknown until the next request. - A `--batch` slot in use shows its list when its request ends. - Conversations the engine parks in RAM (the conversation cache) are not shown. Showing these exactly needs engine commands that list the sessions and the parked conversations. A working version of that exists; it changes the engine, so it would go through an issue before a PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
No site
Links install, modelos, releases.