Issues / #2659

#2659 Benchmark: Warmup seems to spoil first round results

open · @sliedes · 0 commentaires · Sur GitHub

Benchmarks

Description

I realize this is probably hard to fix without invalidating existing benchmarks...

I'm seeing the first benchmark round in chat being consistently way faster:

```
chat: 3 short questions x 21 runs, up to 100 new tokens
  run 1 q1: decode 100.6 tok/s, first token 0.387 s, 100 tokens
  run 1 q2: decode 75.3 tok/s, first token 0.382 s, 100 tokens
  run 1 q3: decode 77.2 tok/s, first token 0.413 s, 100 tokens
  run 2 q1: decode 84.6 tok/s, first token 0.43 s, 100 tokens
  run 2 q2: decode 83.1 tok/s, first token 0.369 s, 100 tokens
  run 2 q3: decode 84.3 tok/s, first token 0.383 s, 100 tokens
```

```
chat: 3 short questions x 21 runs, up to 100 new tokens
  run 1 q1: decode 100.4 tok/s, first token 0.381 s, 100 tokens
  run 1 q2: decode 77.8 tok/s, first token 0.366 s, 100 tokens
  run 1 q3: decode 80.0 tok/s, first token 0.399 s, 100 tokens
  run 2 q1: decode 89.9 tok/s, first token 0.413 s, 100 tokens
```

```
chat: 3 short questions x 3 runs, up to 100 new tokens
  run 1 q1: decode 96.2 tok/s, first token 0.409 s, 100 tokens
  run 1 q2: decode 72.2 tok/s, first token 0.39 s, 100 tokens
  run 1 q3: decode 75.9 tok/s, first token 0.435 s, 100 tokens
```

Looking at the code, it warms up with `PROMPT[0]`, which I'd suspect is the reason:

https://github.com/Niko1221/Strata/blob/296b59d2f73c8f04a805df29c2fa97655adace78/tools/strata_bench.py#L258

Sur le site

Liens install, modèles, releases.