Pull requests / #3109

#3109 bench: RTX 4090 reports for 0.1.45 (IQ3_XXS + IQ3_S, up to the 204800 limit): STRATA_DEQUANT_WIDE regression at 4K, isolated by A/B; full test suite

open · @Dmitry-B · 0 Kommentare · Auf GitHub

BenchmarksServer & APINVIDIA / CUDAModels & quantsDocumentationLinux

Beschreibung

Two community benchmark reports and a full test-suite report for engine 0.1.45 on one
RTX 4090 (sm_89) box. 0.1.44.1 was re-measured in the same sessions by swapping the
engine binaries; every claim below has a same-binary control in the report files.

## Summary

- **No change from 32K through the 204800 context limit**, on both IQ3_XXS and IQ3_S:
  every prompt-reading delta (+0.0% to +1.2% at 156K/185K/200K, ±3% below) sits inside
  the same-binary control band. TTFT at 200K unchanged (61.30 s vs 61.31 s on IQ3_XXS).
  Recall unchanged: 6/6 at 32K/128K (both styles), 3/3 at 192K, both versions.
- **One regression, isolated by A/B: `STRATA_DEQUANT_WIDE` on sm_89.** It is now on by
  default on sm_86; sm_89 is not in the release's test set. On this RTX 4090 it costs
  ~10-15% of 4K-prompt reading and makes the case unstable.
- **Full Python test suite (1703 tests): no regression of 0.1.45.** Two test-side
  defects with reproductions and suggested fixes (below).

## 1. `STRATA_DEQUANT_WIDE` on sm_89: 4K-prompt reading

IQ3_XXS, context 204800, code-style 4K prompt (below the chunk; the auto chunk is 8192
in both versions, so the columns are bit-comparable):

| build | 4K prompt tok/s |
|---|---|
| 0.1.44.1 | 2152 / 2163 / 2168 (tight) |
| 0.1.45, default | nine samples in three runs: 1380-2119, medians 1815-1987 (unstable) |
| 0.1.45, `STRATA_DEQUANT_WIDE=0` | 2107 / 2133 / 2145; control 2129-2145 (tight) |

TTFT at 4K: 1.85 s (0.1.44.1) vs 1.99-2.25 s (0.1.45 default), restored by the switch.
Other lengths are unaffected by the switch (32K: 3289-3321 off vs 3301 default; 128K:
3271-3277 vs 3280). The ru-style 4K case on the IQ3_S config shows the same direction
(-8.3%, below both the control and 0.1.44.1) but was not A/B'd there. Env delivery was
verified in `/proc/<engine>/environ`. Files:
`runs-0.1.45-no-dequant-wide*.json` in the IQ3_XXS code report.

## 2. What was measured

- `2026-10-18-community-rtx4090-iq3xxs-200k-{code,ru}` — IQ3_XXS, 204800. One session:
  warm-up -> code -> ru -> same-binary control -> 0.1.44.1 (engine swap) -> back.
  Long lengths (targets 163840/194560/209715 -> actual 156,291/185,531/200,015 tokens)
  ran in a second session of the same protocol; the report tables merge them into one
  table per style. Generation columns are inside the method's noise (±5-6% on identical
  binaries); no generation conclusion is drawn.
- `2026-10-18-community-rtx4090-iq3s-200k-{code,ru}` — IQ3_S on the same box (~50 GB
  of experts vs 48 GB RAM + 24 GB VRAM: the file tier does the work, decode hit rates
  66-88%). Same protocol, same verdict: no change at any length, recall unchanged.
  Absolute speeds are comparable only within this config.
- Not measured: the release's other opt-in switches, long lengths in the ru style
  (measured code-only this time), generation at long lengths.

## 3. Test suite: 1703 tests, 16 failures, none a regression

`2026-10-18-community-rtx4090-test-suite` (raw log included). Ten failures are the
known 0.1.44.1 baseline on this machine (the `test_setup_golden` fixture's
`strata-vision.exe` entries on Linux; two `test_setup_engine_hash` cases specific to a
locally-built engine). Four are `tools.test_strata_mcp.StartStop` seeing this box's
production server on port 8080 — the module passes 24/24 with the port free (verified).
Two are test-side defects:

1. `test_strata_bench.RedactionTests.test_paths_names_ips_and_keys` — the redactor
   rewrites the username before the literal repo ROOT, so `<strata>` never appears when
   the repo lives under `$HOME` with a matching username. Fails deterministically in
   `~/Strata`; the same module passes from a clean worktree of the same tag at `/tmp`.
   Suggested fix: substitute ROOT before the username rules, or use a synthetic ROOT.
2. `serve.test_keepawake.test_service_holds_for_the_whole_request_including_early_close`
   — the mock request finishes before `KeepAwake`'s daemon thread takes the lock, so the
   `ON` call never happens (5/5 standalone on a fast CPU; passed in a later full-suite
   run — the expectation is timing-dependent). Suggested fix: drive `KeepAwake` directly
   or make the mock generator hold briefly.

## Provenance

RTX 4090 (sm_89), Ryzen 9 7950X, 48 GB RAM, Ubuntu 26.04.1, driver 615.71.09,
CUDA 13.4, Python 3.14.4. Both engines built from source the same way
(`"source": "local"`, archs `[89]`), versions verified in `git describe`,
`engine/BUILD.json` and `/props`. The agent driving the machine works against a
different server, so no observer traffic polluted the runs. Full method, tables and
caveats in the READMEs.

Mehr auf der Site

Links zu Install, Modellen, Releases.