Issues / #2659
#2659 Benchmark: Warmup seems to spoil first round results
open · @sliedes · 0 Kommentare · Auf GitHub
Beschreibung
I realize this is probably hard to fix without invalidating existing benchmarks... I'm seeing the first benchmark round in chat being consistently way faster: ``` chat: 3 short questions x 21 runs, up to 100 new tokens run 1 q1: decode 100.6 tok/s, first token 0.387 s, 100 tokens run 1 q2: decode 75.3 tok/s, first token 0.382 s, 100 tokens run 1 q3: decode 77.2 tok/s, first token 0.413 s, 100 tokens run 2 q1: decode 84.6 tok/s, first token 0.43 s, 100 tokens run 2 q2: decode 83.1 tok/s, first token 0.369 s, 100 tokens run 2 q3: decode 84.3 tok/s, first token 0.383 s, 100 tokens ``` ``` chat: 3 short questions x 21 runs, up to 100 new tokens run 1 q1: decode 100.4 tok/s, first token 0.381 s, 100 tokens run 1 q2: decode 77.8 tok/s, first token 0.366 s, 100 tokens run 1 q3: decode 80.0 tok/s, first token 0.399 s, 100 tokens run 2 q1: decode 89.9 tok/s, first token 0.413 s, 100 tokens ``` ``` chat: 3 short questions x 3 runs, up to 100 new tokens run 1 q1: decode 96.2 tok/s, first token 0.409 s, 100 tokens run 1 q2: decode 72.2 tok/s, first token 0.39 s, 100 tokens run 1 q3: decode 75.9 tok/s, first token 0.435 s, 100 tokens ``` Looking at the code, it warms up with `PROMPT[0]`, which I'd suspect is the reason: https://github.com/Niko1221/Strata/blob/296b59d2f73c8f04a805df29c2fa97655adace78/tools/strata_bench.py#L258
Mehr auf der Site
Links zu Install, Modellen, Releases.