Issues / #2940

#2940 0.1.45: non-thinking requests inherit the card's presence_penalty 1.5 (temperature 0 too), ~25% slower decode

open · @Cass67 · 0 commentaires · Sur GitHub

BenchmarksWindows

Description

Since 0.1.45 (a8091f4d, #2685) a request with thinking off picks up the model card's instruct set under the config's sampling block, so presence_penalty 1.5 ends up on every non-thinking request that doesn't send one. That includes temperature 0 requests, which I'd expect to be plain greedy, and a config sampling block with temperature/top_p/top_k still inherits the card's presence.

On 4x 7900 XT (UD-Q4_K_XL, layer split 12,24,39, --pipeline-windows 2, MTP) it cost me about a quarter of decode: code prompt 89 -> 65-72 tok/s, prose 51 -> 44. STRATA_DECODE_TIMING shows the windows going from 32 to 47 ms, with draft and commit no longer hidden. Same request with an explicit presence_penalty 0 is back to 91-96. It looked like an engine regression at first, but the 0.1.44 engine binary was just as slow under the 0.1.45 server.py.

Maybe temperature 0 should skip the card set, and a config sampling block should replace it rather than merge on top? Setting presence_penalty 0 in the config works around it for now.

Sur le site

Liens install, modèles, releases.