Issues / #471
#471 A prompt cancelled mid-read is logged and recorded as fully read (wrong tok/s, full size in DONE//metrics)
closed · @homeofe · 1 comentarios · En GitHub
BenchmarksServer & APINVIDIA / CUDAModels & quants
Descripción
A request cancelled during its prompt read is logged, and recorded in `DONE` / `/metrics`, as if the **whole** prompt had been read. The prompt-read rate it prints is therefore far too high. ## Seen (engine 0.1.33, CUDA, RTX 2080 Ti, IQ3_S, `--prefill auto` → 3,072-token chunks) A streaming request with a fresh ~98K-token prompt. The client hangs up ~10 s in, and the engine stops after 4 chunks (12,288 tokens, 17.5 s after the request was sent): ``` strata serve: prompt 98179 tokens = 0 reused + 98179 read in 17565 ms (5589.6 tok/s), 0 generated in 0 ms (0.0 tok/s), drafts accepted 0 of 0, 0 checkpoints (cancelled) strata serve: KV streaming: 100.00% of 0 block reads hit VRAM, 0.0 MiB read from RAM [strata] done: 0 tokens in 18 s (0.0 tok/s) (disconnect, cancel=True), ... ``` - **Tokens read:** only 12,288 were read (the `live.prompt_read` progress in `/metrics` showed it). This card reads ~690 tok/s at that depth, not 5,590. - **History:** `/metrics` history records the request with the full 98,179-token prompt. - **Runs:** the same in 6 of 6 runs (3 streaming, 3 non-streaming). ## Where `src/program/generate.cpp` (main, ~5567-5582). The `DONE` record and the `strata serve: prompt …` line use `n` (the request's prompt length), and `fresh = n - resume`, whether or not the request was cancelled. `cancelled` only appends ` (cancelled)`. ## Suggestion When `cancelled`, report the tokens actually read: the position the prompt pass reached, something like `… = 0 reused + 12288 of 98179 read in 17565 ms (699.6 tok/s) … (cancelled)`, and either pass that count in `DONE` or mark the record so the server doesn't count it as a full prompt. I'm not sending a PR because the right semantics for `DONE` (prompt length vs. tokens processed, and what the server's prefix cache assumes after a cancel) are your call. I'm happy to make the change once you say which. The data comes from measuring #431 on CUDA. The server there ran 0.1.33 plus the #430 change from #436, not 0.1.34's watcher, but this log path is the same on `main`.
En el sitio
Enlaces a install, modelos, releases.