#3327 [Engine] the same build is not run-to-run identical at temperature 0 on long answers (RTX 4090) - how should an end-to-end bit-identity check be done?

Descripción

Observation, not a bug claim: on this machine the generated text of the **same binary and the same
configuration** differs between starts, so an end-to-end comparison of two builds by their answers cannot
demonstrate (or refute) "bit-identical". Release notes and community reports use that phrase as a gate for
new switches, and we cannot measure it on a live server here. We would like to know what the intended
end-to-end check is, since the repository's parity harnesses are kernel-level.

## Problem

One build, one config, fixed prompts, `temperature: 0`, `chat_template_kwargs.enable_thinking: false`, the
learned expert profile pinned to an identical copy for every start: 2 of 6 prompts come back identical
when the server is started twice.

## Environment

Strata `e88ee39d` (0.2.0-beta, engine 0.1.45.3, `engine/BUILD.json` src hash `25c992dfc5c6cb06`), built
from source on the machine (CUDA arch `[89]`). Ubuntu 26.04.1, kernel 7.0.0-38, NVIDIA 615.71.09,
CUDA 13.4, RTX 4090 24 GB, Ryzen 9 7950X, 46 GiB RAM. Model: Flash-Next IQ3_XXS, 204,800-token context,
INT8 KV, packed experts, `--expert-cache auto --prefill auto --spec 4 --mtp --pool-workers 10`,
`--pcie-frac 0.00`.

## Reproduce

```bash
python identity-gate.py --arm def --arm def --rounds 2 --long-words 1500
```

What the script does (it starts its own server per arm, like `tools/ab_engine.py`, port 8090, the
production service stopped): builds the config from the working one, removes `--expert-profile-save` and
`--expert-profile-save-every`, and copies `--expert-profile` to a fresh identical copy for every start, so
no arm inherits routing state from a previous one. Then one warm-up request and the same six requests per
arm, in the same order: the three short prompts of `tools/ab_engine.py`, one arithmetic question, one
short code task, and `ab_engine.long_prompt(1500)`. SHA-256 of the returned text is compared. (Script is
mine, not from the repository; the prompts are the repository's own fixtures.)

## Actual (first two rounds of the control arm, one build)

| prompt | start 1 | start 2 | identical |
|---|---|---|---|
| short chat 0 (merge two sorted lists) | len 894 | len 894 | yes |
| arithmetic (one number) | len 3 | len 3 | yes |
| short code task (Fibonacci) | len 690 | len 738 | no, first difference at char 1 |
| short chat 1 (refrigerator) | len 1417 | len 1420 | no |
| short chat 2 (ten tips) | len 1121 | len 1137 | no |
| long (summarize a word list) | len 640 | len 768 | no |

The differences are not truncations: they start inside the first sentence or two and then the texts
diverge ("…consisting of coils both inside and outside the appliance…" vs "…consisting of an evaporator
coil inside the fridge and…"). With two switch arms measured against the same baseline the counts land in
the same range as the control (5 of 12 and 6 of 12 comparisons matching, control 4 of 12), so on this
machine nothing can be said about whether a switch changes the output.

Short, strongly constrained answers (one number, a short deterministic function) do reproduce; anything
longer than a couple of sentences does not.

## What is ruled out

Not the saved expert profile: with `--expert-profile-save`/`--expert-profile-save-every` removed and the
loaded profile replaced by a fresh byte-identical copy for every start, the divergence stays. (With the
working configuration, where the profile is rewritten during the run, 4 of 7 prompts differed - so the
profile makes it worse, it is not the cause.) Not the prompt text and not the sampling parameters: they
are fixed, greedy, thinking off.

## What was not tested

Which part of the pipeline is non-reproducible. Plausible candidates, named as hypotheses only: the
adaptive expert tier computing the same expert on the host and on the GPU in different runs (different
floating-point paths, near-ties flip), the speculative/MTP verification path, kernels that reduce with
atomics.

## Question

The repository demonstrates bit-identity with kernel- and feature-level harnesses
(`src/kernels/cpu/*_parity.cpp`, `src/kernels/bf16_gemv_parity.cpp`, `bench/micro/fusions_parity.cu`,
`tools/conversation_cache_parity.py`, `tools/split_mtp_batch_parity.py`) and evidence files such as
`bench/results/2026-10-12-p40-sm61-kernels/dequant-wide-parity.txt`. Is there an intended **end-to-end**
way to show it on a running server - a set of switches that makes serving reproducible (a fixed expert
tier, no adaptive overlap), a reference server configuration used for such claims, or a tool that compares
logits/tokens rather than text? With that, we can attach this kind of evidence to our own reports; without
it, on this hardware an answer-based gate measures the machine, not the change.

## Reproduction status

Reproduced locally: the control arm was run twice as separate server starts at the same commit, and the
same divergence appears for two other arms and across rounds of the same arm (six starts in total, three
configurations).