Issues / #410

#410 Greedy output varies between server requests, but not between fresh engine runs (`STRATA_IQ_MT_MIN=1`, 0.1.32)

closed · @brenoperucchi · 4 commentaires · Sur GitHub

Setup & installServer & APINVIDIA / CUDAModels & quantsDocumentationWindows

Description

Follow-up to #152. With `STRATA_IQ_MT_MIN=1`, temperature 0 still gives different answers to identical requests on
this machine. The engine itself looks deterministic: the variation comes from state the server carries from one
request to the next, and the adaptive expert tier explains at least part of it.

**Machine:** RTX 5090 32 GB, Ryzen 9 5950X (Zen 3, AVX2 only), 96 GB DDR4-3200, Windows 11 26200.
**Engine:** 0.1.32 from the release zip, with the `v0.1.32` checkout's `serve/server.py`.
**Model and config:** Flash-Next IQ2_XS with `--expert-cache auto --spec 4 --mtp <rt> --max-context 32768 --kv int8
--pcie-frac 0.55 --spec-min-p 0.70` and `STRATA_IQ_MT_MIN=1`.
**Requests:** `/v1/chat/completions`, `temperature 0`, `seed 42`, `cache_prompt false`, `max_tokens 256`, one after
another on one server. We compare the SHA-256 of the returned content.

## Results

| Setup | Prompt | Distinct outputs |
|---|---|---|
| Engine CLI, a new process each run, `--tokens` (41 fixed ids) `--max-new 64` | short | 1 of 5; the `STRATA_DUMP_FIRST_LOGITS` file is identical too |
| Server, defaults | short (27 tokens) | 2 of 5 |
| Server, `--prompt-cache 0` | short | 3 of 6 |
| Server, `--prompt-cache 0 --adapt-swaps 0` | short | 1 of 6, with the same draft acceptance every time |
| same | medium (2,671 tokens) | 3 of 4 |
| Server, `--prompt-cache 0 --adapt-swaps 0 --no-prefill-borrow` | medium | 4 of 4 |
| same | long (5,952 tokens) | 1 of 4 |

With the defaults, draft acceptance on the short prompt moved between 36 and 37 of 48-49 across requests even when
the text came out the same.

On 0.1.31 we had already ruled out the GPU/CPU split by itself: with `--expert-cache 1 --pcie-frac 0` (decode cache
hit rate 0.1%) outputs still varied, 2 to 3 distinct in 5. The adaptive tier was on in that run.

## What we think is going on

The adaptive tier moves experts between RAM and VRAM based on what earlier requests routed. The GPU and the CPU round
the same expert differently (the startup message says so), so the same token can be computed on a different device
from one request to the next. Turning the tier off made the short prompt reproducible.

We cannot explain the medium prompt yet. It still varies with the tier off and with `--no-prefill-borrow`, while the
longer prompt does not. Something else that persists between requests seems to affect the prompt path at that length.

We are not reporting a quality problem. Greedy repeats are not byte-identical, which gets in the way of A/B tests.
The docs say `STRATA_IQ_MT_MIN=1` makes the answer independent of the drafting. It may help to add that the
adaptive tier (and maybe other server state) still changes it, and that `--adapt-swaps 0` removes that part. We can
run more tests here if you tell us what to try.

Sur le site

Liens install, modèles, releases.