#3637 [Benchmark] RTX 4070 Ti SUPER 16 GB · Swift-Qwen3.8-Flash-Next IQ3_XXS (engine 0.1.47, app 0.2.2, custom track)

Beschreibung

## What did you measure, or what went wrong

All measurements are from **one machine**, on **engine 0.1.47**, app **0.2.2** (tag `v0.2.2`, commit `011c9e60`), upgraded from 0.1.44.1 on 2026-10-22. Nothing here is from 0.1.44.1 except the explicitly labelled baseline rows.

**Hardware / software** (auto-collected by `strata bench`): RTX 4070 Ti SUPER 16 GB (`17171480576 B`, dedicated, `285 W` limit), AMD Ryzen 7 7800X3D, 61 GB RAM, Fedora Linux 44, GPU driver `615.78.08`, CUDA `13.4`, built `sm_89` (engine from source; prebuilt CUDA-13 package hash-verified against the official `SHA256SUMS.txt`). NUMA: 1 node.

**Model**: `swift-1.5-iq3_xxs` = Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS, IQ3_XXS, 2 shards
`sha256 3bddaa667c750f63baca766df9433e1afd35a139f401c5ca5e7ff560ceff3d23` (shard 1, 39,785,790,560 B)
`sha256 b0b15f782af71eb471909d2f3313b2927c324962f86c084325c247cd411a0160` (shard 2, 36,180,282,560 B)
mmproj `mmproj-Swift-Qwen3.8-Flash-Next-BF16.gguf` (0.91 GB).

**Serving shape**: llama-swap → `serve/server.py --engine strata`, one slot per line, `--max-context 262144 --kv int8 --kv-resident 32768 --spec 6 --spec-min-p 0.5 --prefill auto:32768 --conversation-cache-mib 4096 --adapt-every 1 --adapt-swaps 80 --adapt-decay 0.97 --adapt-async 1 --vram-reserve-mib 700`, plus `--vision --vram-elastic` on the vision line. Expert cache self-reports `5514 slots, 8.93 GiB of VRAM`; KV streaming `32768 of 262144 cells … 3.09 GiB pinned RAM`.

### 1. Standard suite `standard-v1` (custom track, protocol `strata-qwen38-thinking-v1-draft.1`)

```
strata bench run --suite standard-v1 --url http://127.0.0.1:5808 --out <new-dir> --output json \
  --track custom --track-notes "0.1.47 source build; answer room (reasoning_budget_tokens:0); CPU vision encoder (vision.gpu=false, elastic=false); ctx 262144, KV int8, MTP; custom expert pack" \
  --quantization "IQ3_XXS (Swift 1.5 GSQ-RCO, 2 shards)" \
  --custom-pack /home/loong/strata/strata/data/expert-profile.bin \
  --build-options "CUDA 13.2, gcc-15, sm_89, source build" \
  --background "host app + llama-swap (zero session requests inside the window) + user browser"
```

`run_id 20261021T185312Z-20f0f351` · corpus `e75eff8b9feea04f745520b392c4c17d97aefc1ba185e222aa6740efa8f68407` · 22 requests (1 warmup + 9 performance + 12 sanity; **optional concurrency stage not selected**) · wall clock 180 s · `failed: 0`.

| phase | prompt tok/s (median) | decode tok/s (median) | TTFT (median) | total s (median) |
|---|---|---|---|---|
| short | 119.9 | 80.8 | 0.67 s | 6.45 s |
| prompt-4k | 1637.2 | 83.2 | 2.53 s | 8.78 s |
| prompt-16k | 3046.4 | 84.5 | 5.41 s | 11.51 s |

Quality gate **12/12** (instruction 4/4 · retrieval 4/4 · tool_call 4/4). Request counts: warmup 1 (1 truncated), performance 9 (1 completed / 8 truncated, i.e. wrote to the spec'd output ceiling), sanity 12/12 completed.

Artifacts (`results.json` `4dc7feade27fee86…`, `outputs.jsonl` `d58a398d4590c4dc…` 509 KB, `config.json` `cde6a66e6fcd8322…`, CLI-generated `README.md` `a4b3c23e77e9b9e3…`, `provenance/{chat_template.jinja,corpus-build.json,corpus-manifest.sha256}`). Not attached to this issue (509 KB outputs + a 34-request corpus of outputs is a PR-scale payload) — say the word and I will open a results-only PR.

Nonconforming fields, verbatim from `results.json.missing`: `/protocol/model_card_revision` (profile revision not pinned), `/run/optional_duration_s` (stage not selected), `/hardware/storage` (drive holding the model files not identified), `/model/revision` + `/model/source_revision` (not readable from the engine; needs `--model-revision`). Suggested doc/tool note: `--model-revision` being optional means a "standard" run silently ships with `revision: null`; a one-line warning in the README output would make that visible to a reader.

Single long-context datapoint from the same line, same engine (session traffic, not the bench): `prompt 157433 tokens = 0 reused + 157433 read in 57335 ms (2745.9 tok/s), 1950 generated in 25974 ms (75.1 tok/s)`.

### 2. Vision encoder A/B on a 16 GB card (CPU encoder = upstream default below 24 GB)

Premises: engine 0.1.47 · IQ3_XXS · ctx 262144 · KV int8 · spec6+MTP · `--vram-reserve 700` · greedy decode · probe = 400-character verbatim-repeat, `max_tokens 900`, `temperature 0`, thinking off · same config file, one variable (`vision.gpu`, `vision.elastic`).

| | arm G = GPU encoder + elastic (previous production setting) | arm C = CPU encoder (current production setting) |
|---|---|---|
| encoder line in log | `strata-vision: warmed up at 1024 image tokens in 0.4 s` | `strata-vision: on the CPU, 8 threads, no warm-up` |
| decode 212 tokens (1st / 2nd) | 105.0 / **177.1** tok/s | 108.9 / **180.7** tok/s (**+2.0% / +3.7%**) |
| first image (incl. cold engine load) | 34 s | 37 s |
| second image (hot) | 0 s | 1 s |
| expert slots / free VRAM | 5512 slots · 423 MiB | 5512 slots · 423 MiB (unchanged) |
| elastic behaviour | `VRAM 2048 MiB kept free: the expert cache 5512 -> 4619` | none (`elastic:false`, encoder takes no VRAM) |

Direction matches the upstream numbers in `docs/DETAILS.md:1779-1784` (CPU +4.8/+5.2% on P100/A4000); on this card the gain is smaller (**+2.0/+3.7%**) and the hot-image cost is +1 s. We kept the CPU encoder in production.

### 3. `--conversation-cache-mib` gains a large part of their effect on the second conversation (0.1.47)

Premises: engine 0.1.47 · two **64k** conversations alternating (equal length, different first lines) · arm A `strata-swift-iq3xxs-text` with `--conversation-cache-mib 4096` · arm E `strata-swift-abl-iq3xxs-text`, cache **not** enabled, weight tensor layout byte-identical to Swift (single variable = the cache switch) · thinking off · `max_tokens 32` · CPU-encoder line · **zero session requests inside the measurement window** (see methodology note).

| round | arm A (cache enabled) | arm E (no cache) |
|---|---|---|
| 1 X / 1 Y (prompt 66423) | 57.9 s `reused 0` / 23.5 s `reused 0` | 59.1 s `reused 0` / 23.4 s `reused 0` |
| 2 X / 2 Y (66461) | **0.5 s `reused 66416` · re-read 45** / **23.7 s `reused 0` · re-read 66461** | **0.4 s `reused 66429` · re-read 32** / 23.4 s `reused 0` · re-read 66461 |
| 3 X / 3 Y (66499) | **0.5 s `reused 66454` · re-read 45** / **23.6 s `reused 0` · re-read 66499** | 0.4 s `reused 66466` / 23.5 s `reused 0` |

Baseline on 0.1.44.x, same probe, same config: **both** conversations 0.6–0.9 s in rounds 2–3. So the alternating-conversation gain went from ~40x to **0x for the second conversation**; only the most recent conversation reuses (6 prompt checkpoints, 0.4–0.5 s), the other re-reads ~23 s every round.

Root cause, from the engine's own log lines in the same window (arm A; `/metrics` cross-checked every 10 s — `budget_mib 4096, slots 4`, `parked 0→1→2(2,078,850,648 B)→1(1,881,958,796 B, evictions 1, parks 3, restores 1)`):

```
conversation cache: parked 66428 tokens in 338.9 ms; parked=2 bytes=2078850648 evictions=0 snapshot_bytes=1841864556 reused_kv_bytes=0
conversation cache: skip parking (physical RAM admission; need 1756 MiB plus 2560 MiB floor, 2918 MiB available, source meminfo; evicted 1, 0 still parked)
conversation cache: restored 66416 tokens (checkpoint) in 65.4 ms; parked=0 bytes=1013968728
conversation cache: parked 66466 tokens in 165.0 ms; parked=1 bytes=1881958796 evictions=1 snapshot_bytes=1881958796 reused_kv_bytes=1013769600
conversation cache: skip parking (physical RAM admission; need 1757 MiB plus 2560 MiB floor, 3130 MiB available, source meminfo; evicted 0, 0 still parked)
```

So on this box the gate — **one 64k conversation snapshot needs 1756 MiB, plus a fixed 2560 MiB floor, against 2918–3130 MiB of `MemAvailable`** (a 48 GB model is resident) — cannot admit the second conversation, and parking one evicts the other (`evicted 1`, steady state `parked 1 · evictions 1`). The 0.5 s rounds are the `(checkpoint)` restore path (65 ms), not the parked snapshot. `--conversation-cache-mib 4096` / `--conversation-cache-slots 4` is therefore unreachable in practice here, and the doc claim in `docs/DETAILS.md:1175-1178` ("park up to four conversations … when their requests alternate") does not hold on a machine where the model itself takes most of RAM.

The gate is `src/program/generate.cpp:9422-9450`: `admit()` samples host memory, the loop evicts the oldest parked conversation and re-checks, and if it still fails it prints `skip parking` and returns. The floor is `conversation_cache_min_free_mib`, default `2560` (`:723`), settable via `--conversation-cache-min-free-mib` (`:2639`). Two suggestions: (1) the admission arithmetic compares against `MemAvailable` without accounting for the model's own resident bytes, so a box that is healthy *because* the model is resident is exactly the box where the cache silently never engages — counting the model, or documenting "this needs N GiB of free RAM per parked conversation", would save people the investigation; (2) surface `skip parking` in `/metrics` `conversation_cache` as a counter — today the card reports `parked 1` and nothing about why the second park was refused, so from the client side it just looks like the feature is off.

Boundaries: single alternating pair, two independent repeats with the same outcome; the MiB numbers are bound to this box (61 GB RAM, 48 GB model resident) — a smaller model changes the headroom. Not tested: `slots` > 4, `budget_mib` 8192, and whether the 0.1.44.x baseline (both conversations 0.6–0.9 s) was measured under the same admission gate.

### 4. IQ sampling, three arms, sampled subset (engine 0.1.47)

Premises: 16 of 64 questions by `stride=4` · thinking off · `max_tokens 1024` · CPU-encoder line · zero session requests inside the window · probe self-test passed first (right answers scored right, wrong answers scored wrong).

| arm | model | score | legacy subset | trap+calibration+reverse | median / question |
|---|---|---|---|---|---|
| A | `strata-swift-iq3xxs-text` | **16/16** (0 truncated) | 5/5 | 3/3 | 0.6 s |
| B | `strata-swift-iq3xxs-256k` (vision line) | **16/16** (0 truncated) | 5/5 | 3/3 | 0.6 s |
| E | `strata-swift-abl-iq3xxs-text` (ablated weights) | **15/16** (0 truncated) | 5/5 | 2/3 | 0.6 s |

E's single loss is a long-tail fact: "who wrote llama.cpp" → **"Georg Gergel"** (truth: Georgi Gerganov). Both Swift arms answered correctly. Small sample; the observation is about the shape (ablated weights invent a plausible name), not a rate.

### Methodology note (why the numbers are clean, and a trap for anyone reproducing them)

On a single-GPU box with an on-demand loader, **the measuring agent's own traffic contaminates the run**: every assistant turn is a request to the same endpoint, so a session sharing the endpoint force-swaps the probed engine (mutual-exclusion group), and the parked/prompt caches do not survive the reload. Our first attempts at items 3 and 4 produced exactly that signature (`reused 0` rows with wall clocks of 135–158 s, plus repeated identical rows because the probe read "last matching log line" instead of this request's line). Both were discarded and re-run inside a quiescent window with the assistant model unloaded, and the probe was changed to record the log byte offset before each request and read only the appended segment. Any cache-related measurement on a shared endpoint needs that isolation; it is worth a line in the benchmark docs.

### Reproduction status

**Measured locally** — one machine, one run per item (item 1 has `n=3` intra-run repeats, no second run). Not run: the optional concurrency stage, and a repeat of the whole suite.