Issues / #2391
#2391 llama-benchy misreports prefill speed
open · @crusaderky · 0 commentaires · Sur GitHub
BenchmarksServer & APIModels & quants
Description
Strata 0.1.43.1 sends bogus information to llama-benchy re. prefill speed. The program works fine with llama.cpp. ``` python -m llama_benchy --no-cache --runs 3 --pp 8192 --tg 128 --tokenizer Qwen/Qwen3.8-Flash-Next --base-url http://127.0.0.1:8082/v1 --model qwen3.8-flash-next-iq3_xxs | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:---------------------------|-------:|-----------------------:|-------------:|------------:|---------------:|----------------:| | qwen3.8-flash-next-iq3_xxs | pp8192 | 1390190.06 ± 144331.69 | | 7.25 ± 0.67 | 5.96 ± 0.67 | 3649.53 ± 21.55 | | qwen3.8-flash-next-iq3_xxs | tg128 | 85.54 ± 5.11 | 86.00 ± 5.35 | | | | ``` whereas on http://localhost:8082/#monitor I read ~2,200 tok/s prefill. The decode speed is accurate.
Sur le site
Liens install, modèles, releases.