Issues / #471

#471 A prompt cancelled mid-read is logged and recorded as fully read (wrong tok/s, full size in DONE//metrics)

closed · @homeofe · 1 Kommentare · Auf GitHub

BenchmarksServer & APINVIDIA / CUDAModels & quants

Beschreibung

A request cancelled during its prompt read is logged, and recorded in `DONE` / `/metrics`, as if the **whole** prompt had been read. The prompt-read rate it prints is therefore far too high.

## Seen (engine 0.1.33, CUDA, RTX 2080 Ti, IQ3_S, `--prefill auto` → 3,072-token chunks)

A streaming request with a fresh ~98K-token prompt. The client hangs up ~10 s in, and the engine stops after 4 chunks (12,288 tokens, 17.5 s after the request was sent):

```
strata serve: prompt 98179 tokens = 0 reused + 98179 read in 17565 ms (5589.6 tok/s), 0 generated in 0 ms (0.0 tok/s), drafts accepted 0 of 0, 0 checkpoints (cancelled)
strata serve: KV streaming: 100.00% of 0 block reads hit VRAM, 0.0 MiB read from RAM
[strata] done: 0 tokens in 18 s (0.0 tok/s) (disconnect, cancel=True), ...
```

- **Tokens read:** only 12,288 were read (the `live.prompt_read` progress in `/metrics` showed it). This card reads ~690 tok/s at that depth, not 5,590.
- **History:** `/metrics` history records the request with the full 98,179-token prompt.
- **Runs:** the same in 6 of 6 runs (3 streaming, 3 non-streaming).

## Where

`src/program/generate.cpp` (main, ~5567-5582). The `DONE` record and the `strata serve: prompt …` line use `n` (the request's prompt length), and `fresh = n - resume`, whether or not the request was cancelled. `cancelled` only appends ` (cancelled)`.

## Suggestion

When `cancelled`, report the tokens actually read: the position the prompt pass reached, something like `… = 0 reused + 12288 of 98179 read in 17565 ms (699.6 tok/s) … (cancelled)`, and either pass that count in `DONE` or mark the record so the server doesn't count it as a full prompt. I'm not sending a PR because the right semantics for `DONE` (prompt length vs. tokens processed, and what the server's prefix cache assumes after a cancel) are your call. I'm happy to make the change once you say which.

The data comes from measuring #431 on CUDA. The server there ran 0.1.33 plus the #430 change from #436, not 0.1.34's watcher, but this log path is the same on `main`.

Mehr auf der Site

Links zu Install, Modellen, Releases.