#3323 [Benchmark] RTX 4090 24 GB: engine 0.1.45 vs 0.1.45.3 changes nothing measurable at 16K-32K; short prompts not attributable

Descripción

## Problem and result

The engine of this machine was updated from 0.1.45 to 0.1.45.3 (0.2.0-beta). Two folders of measurements
answer one question: does the update change what this machine feels?

Result: **no, within +-1% at 16K and 32K tokens.** Prompt reading and the conversation cache are
unchanged; the short-prompt column and the decode column cannot be attributed to the engine at all on
this machine. We propose no engine change - this is a results-only submission, and the negative half of
it is the useful part: our own earlier reading of a long-context table (+0.5…+1.7%) does not survive the
repository's own suite.

## Related discussion

none required (a results-only submission, no behaviour change proposed). For continuity: our 0.1.45
measurements for this same machine were PR #3109, closed by the tracker reset on 2026-10-19; the numbers
here re-measure part of that baseline on the old binary in the same session, so the comparison does not
depend on the old thread.

## Approach

Two engines, one machine, everything else identical (same server config, same model files, same
arguments): 0.1.45 kept as `engine/strata.0145.bak` and 0.1.45.3, both **built from source on this
machine by `update.sh`** (CUDA arch `[89]`), never downloaded. The auto-selected prompt chunk is the
same in both (8192 tokens, 96-slot ring), so the runs are comparable token by token.

The measurements are ordered so that the version is separated from the state of the machine:

- `python tools/strata_bench.py --runs 3 --long` against a freshly started server, run **twice per
  engine with the order of the pair reversed** (pair 1: 0.1.45 then 0.1.45.3; pair 2: the reverse). This
  machine drifts like the one in #1332: the same setting gave 1294-1924 tok/s at 4K across four runs, so
  one pair cannot separate drift from a real difference. What is claimed below is what agrees in both
  pairs; what does not is named as not established.
- a control run of the same 0.1.45.3 binary again in the same session, which gives the noise band of the
  second instrument (prompt +-0.0…0.7% at 32K and above, decode +-0.6…6.6%);
- an interleaved binary A/B (2 pairs, both binaries alternating inside one session) with the second
  instrument, and a supplementary suite at 128K-200K, because the tool stops at 32K while this
  configuration's whole point is the 204,800-token context.

## Validation

**Passed:**

- `python tools/strata_bench.py --url http://127.0.0.1:8080 --api-key <key> --runs 3 --long`, four times
  (2 pairs x 2 engines), commit `e88ee39d` / engine 0.1.45.3 and the kept 0.1.45 build; Ubuntu 26.04.1,
  kernel 7.0.0-38, NVIDIA 615.71.09, CUDA 13.4, RTX 4090 24 GB, Ryzen 9 7950X, 46 GiB RAM. Observed,
  pair 1 / pair 2: 16K prompt -0.1% / -0.8%, 32K prompt +0.1% / -0.9%, conversation-cache hit 69.4% in
  all four runs, TTFT at 16K/32K within +-1%; 4K prompt -12.7% / -20.8% (the two pairs disagree about
  which engine is faster); `chat decode` +2.6% / +9.1% and `agent decode` -4.0% / +8.7% - the sign flips
  with the order, so no decode claim is made.
- the recall check (needle at 32K and 128K, depths 10/50/90) passed 6/6 on both engines; draft
  acceptance over the whole long-context suite 4615/6415 (71.9%) -> 4567/6241 (73.2%).
- the long-context suite (4K…200K, 3 runs each, warm-up, control run of the same binary, and the 0.1.45
  baseline of the short cases measured on the old binary in the same session): +0.5…+1.7% at 32K and
  above in one session, x1.010 in the interleaved A/B, and 0…-1.3% in the repository's suite. Taken
  together we report **no** prompt improvement.

**Failed:**

- the full unit-test run at this commit: 1881 tests (1012 `tools/`, 869 `serve/`), 57 not passing, against
  1703 tests / 16 not passing at 0.1.44.1. Every delta was traced to this machine or to the tests
  themselves, none to the engine or the server; the counts and the names are in
  `tests-0.2.0-beta-summary.txt` in the folder, and the two test-side problems are filed as a separate
  issue.
- an attempt to compare the two builds by their answers (fixed prompts, temperature 0, thinking off) did
  not work as a gate: the control arm - the same configuration started twice - differs in 4 of 6
  prompts. That is filed as a separate observation, because it affects how "bit-identical" can be
  demonstrated at all.

**Not run:**

- concurrency (`--clients`), cold start with an empty page cache, vision requests, the 0.2.0
  application-layer features;
- any other machine or backend: this is one RTX 4090 (sm_89) with a source build; HIP, SYCL and Metal
  were not built here;
- the opt-in switches of the release (`STRATA_GDN_CHUNKED`, the sm_86 switch set, `STRATA_IO_PREFETCH`)
  and the file tier of #149: measured on this card, reported in a separate PR, not in this one.

## Limits

The long-context supplement uses our own prompts and lengths and is not comparable with other community
reports; the numbers of `tools/strata_bench.py` are. The 4K column is dominated by session state on this
machine in both directions, and the decode column is inside the band of repeated runs - both are reported
without conclusions. No claim is made about other quants, other contexts or other cards.