Issues / #2391

#2391 llama-benchy misreports prefill speed

open · @crusaderky · 0 comentários · No GitHub

BenchmarksServer & APIModels & quants

Descrição

Strata 0.1.43.1 sends bogus information to llama-benchy re. prefill speed.
The program works fine with llama.cpp.

```
python -m llama_benchy --no-cache --runs 3 --pp 8192 --tg 128 --tokenizer Qwen/Qwen3.8-Flash-Next --base-url http://127.0.0.1:8082/v1 --model qwen3.8-flash-next-iq3_xxs

| model                      |   test |                    t/s |     peak t/s |   ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:---------------------------|-------:|-----------------------:|-------------:|------------:|---------------:|----------------:|
| qwen3.8-flash-next-iq3_xxs | pp8192 | 1390190.06 ± 144331.69 |              | 7.25 ± 0.67 |    5.96 ± 0.67 | 3649.53 ± 21.55 |
| qwen3.8-flash-next-iq3_xxs |  tg128 |           85.54 ± 5.11 | 86.00 ± 5.35 |             |                |                 |
```
whereas on http://localhost:8082/#monitor I read ~2,200 tok/s prefill.
The decode speed is accurate.

No site

Links install, modelos, releases.