#6509 [Engine] Engine 0.1.55+ decodes 35–55% slower than 0.1.52 on RTX 3080 Ti + RTX 4070 (Windows): decode expert-cache hit falls from ~92% to 55–70% although the cache gets ~40% more slots
open · @real4root · 0 comentários · No GitHub
BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows
Descrição
### Area
[Engine] (speed, kernels, models, the server and API)
### Desktop app, Terminal or source?
Terminal (`strata` CLI on Windows, installed by the updater; older engine builds kept by the updater under `components\engine\`)
### What happened
With the same model files, the same pack, the same expert profile and the same Control (0.1.63), engine 0.1.52 decodes at 62 tok/s with setup defaults and 83 tok/s tuned, while engine 0.1.55 decodes at 26–27 tok/s and the current engine 0.1.63 at 40–47 tok/s after all the tuning I could find. The decode expert-cache hit rate falls from 92–94% on 0.1.52 to 56% on 0.1.55 and 69% on 0.1.63, **even though 0.1.63 gets more cache slots** (2358 slots / 6.88 GiB against 1675 slots / 4.88 GiB on 0.1.52). Prompt reading is faster on 0.1.63 (16K prompt: 1028 against 689–931 tok/s), so only decode regressed.
This looks like the same problem as #6208 (worse resident set from `--expert-cache auto` with the same pack and profile), here on Windows with two cards and a different quant.
How I got here: the engine auto-updated from 0.1.52 to 0.1.55 in the same days that I moved the GPUs to a new Xeon Gold 6254 box. I only saw the benchmarks after the update, so I spent about three weeks blaming the hardware (PCIe x8, memory channels, CPU) and swapping CPUs and boards. The old engine binaries, run on the same machine, show that the CPU makes little difference and the engine version makes most of it.
I have not bisected between 0.1.52 and 0.1.55. I can do it if you point me at the intermediate builds.
#### Measurements
`strata bench run --suite standard-v1` (thinking on, as the suite runs it). Decode tok/s for the short / 4K / 16K cases, prompt tok/s for 4K / 16K. Cache hit = the server's `decode expert cache N% hit` lines (min–median–max over the run). 256K context, `--kv int8 --kv-resident 32768 --expert-cache auto --prefill auto --spec 4`, layer split auto (CUDA1 = 3080 Ti runs layers 22–47). 20 expert-pool workers + the host thread. 1 run per row (3 samples per case inside standard-v1).
**System:** Xeon E5-2696 v4 (22C/44T), 4×64 GB DDR4-2400, RTX 4070 12 GB (display, GPU0) + RTX 3080 Ti 12 GB (GPU1), both PCIe 3.0 x16, Windows 11 (10.0.26200), driver 617.42.
| Engine | Settings | Decode short / 4K / 16K | Prompt 4K / 16K | Decode cache hit | Expert cache |
|---|---|---|---|---|---|
| 0.1.52 | setup defaults, MTP | 62.3 / 59.5 / 61.4 | 450 / 689 | ~91% | 1675 slots, 4.88 GiB |
| 0.1.52 | + `STRATA_ARENA_PIN_GIB=0` | 70.8 / 69.3 / 66.6 | 485 / 931 | 87.4–92.2–94.8% | 1675 slots, 4.88 GiB |
| 0.1.52 | + `STRATA_ARENA_PIN_GIB=0 STRATA_ROUTE_TAIL_SKIP=7` | **83.3 / 75.0 / 70.6** | 491 / 931 | 91.7–94.5–97.1% | 1675 slots, 4.88 GiB |
| 0.1.52 | setup defaults, `--kv k8v4` | 62.6 / 63.7 / 66.7 | 436 / 677 | 86.3–92.2–95.5% | |
| 0.1.55 | setup defaults, MTP | **26.0 / 26.7 / 27.5** | 428 / 681 | 35.1–56.0–66.9% | |
| 0.1.63 | best I found (below), DFlash2 | **40.0 / 46.0 / 46.9** | 545 / 1028 | 50.0–69.2–78.7% * | 2358 slots, 6.88 GiB |
\* hit rate from a 0.1.63 run with `STRATA_ARENA_PIN_GIB=0 STRATA_ROUTE_TAIL_SKIP=7`, same as the tuned row except the GPU-role and PCIe flags.
The 3080 Ti part of the cache: 0.1.52 `expert cache 1172 slots (3.41 GiB), 1172 of its 13312 profiled pairs`; 0.1.63 `expert cache 1715 slots (5.00 GiB), 1715 of its 13312 profiled pairs`. Both slot counts are from runs with the same `--pcie-frac 0.05 --vram-reserve-mib 800` that the Control passed. So 0.1.63 holds ~46% more experts on the main card and still misses far more often.
#### My own greedy client bench (temperature 0)
Streaming `/v1/chat/completions`, 4 tasks: Russian chat (400 tokens), code (600), English essay (500), an ~6K-token prompt with a 120-token answer. Decode = (completion tokens − 1) / (last token time − first token time), averaged over the 4 tasks, median of 2–3 rounds.
| Engine | Settings | Decode tok/s | Prompt tok/s |
|---|---|---|---|
| 0.1.52 | setup defaults, `--kv k8v4` | 56.5 | 504 |
| 0.1.52 | setup defaults | 56.4 | 412 |
| 0.1.52 | + `STRATA_ARENA_PIN_GIB=0` | 74.9 | 709 |
| 0.1.52 | + `STRATA_ARENA_PIN_GIB=0 STRATA_ROUTE_TAIL_SKIP=7` | **80.2** | 706 |
| 0.1.55 | setup defaults | **18.4** | 322 |
| 0.1.63 | setup defaults (startup preset) | 34.6 | 665 |
| 0.1.63 | fully tuned (below) | **51.5** | 810 |
#### What I tried on 0.1.63 (own bench, decode tok/s)
| Tried | Decode | Prompt | Note |
|---|---|---|---|
| startup preset, for reference | 34.6 | 665 | |
| `--pool-cores` 16 / 21 workers | 34.8 / 35.1 | 667 | no change |
| `STRATA_ROUTE_TAIL_SKIP=7` | 43.0 | 672 | +24% |
| `--gpu-roles 1=main,0=helper` | 36.9 | 714 | +7% alone |
| tail skip + roles | 45.4 | 670 | |
| + `--layer-split 21` | 43.9 | 670 | worse |
| + `--pipeline-windows 2` | 44.7 | 652 | worse |
| + `--spec 8` | 44.6 | 669 | worse |
| + `--kv k8v4` | 46.3 | 700 | ~same |
| + `STRATA_ARENA_PIN_GIB=0` | **50.7** | **809** | +12%; Windows multi-GPU pins only 8 GiB by default |
| + PIN0 + `--kv k8v4` | 49.4 | 789 | |
| + PIN0 + `--pcie-frac 0.3` | 44.7 | 811 | −12% |
| + PIN0 + `--pcie-frac 0.05` | 51.0 | 821 | +1–2% |
| + PIN0 + `--pcie-frac 0` | 51.3 | 815 | |
| + PIN0 + `STRATA_PREFILL_HELP=1` | 51.8 | 808 | noise |
| + PIN0 + `--host-core last` | 49.9 | 816 | |
| + PIN0 + `--vram-reserve-mib 1100` | 47.8 | 830 | |
| + PIN0 + pcie 0.05 + `--vram-reserve-mib 0` | 52.4 | **550** | prompt drops: 0 MiB VRAM left free |
| + PIN0 + pcie 0.05 + `--vram-reserve-mib 800`, MTP | 50.5 | 806 | |
| + PIN0 + pcie 0.05 + `--vram-reserve-mib 800`, **DFlash2** | **51.5** | 810 | the final 0.1.63 setting |
The best 0.1.63 setting is `--gpu-roles 1=main,0=helper --pcie-frac 0.05 --vram-reserve-mib 800 --drafting dflash2`, `STRATA_ROUTE_TAIL_SKIP=7 STRATA_ARENA_PIN_GIB=0`. It is still 36% behind 0.1.52 with just PIN0 + tail skip.
#### Same engine, different CPUs (same GPUs, standard-v1)
These runs are from the weeks before, on other CPUs and boards, with the same two cards, model and pack. They show that the CPU is not the cause, and that the drop came with the engine update.
| Date | CPU / board | Engine | Decode short / 4K / 16K | Prompt 4K / 16K |
|---|---|---|---|---|
| Oct 29 | E5-2666 v3, X99 | 0.1.52, `--kv k8v4` | 70.0 / 61.4 / 68.4 | 503 / 717 |
| Nov 21 | E5-2696 v4, X99 | 0.1.52, `--kv k8v4` | 62.6 / 63.7 / 66.7 | 436 / 677 |
| Nov 2–3 | Xeon Gold 6254, 6-channel (one card possibly at x8) | 0.1.52 | 73–76 / 67–71 / 68–73 | ~383 / 543 |
| Nov 3 | Xeon Gold 6254, same box | **0.1.55** (auto-updated) | **39.6 / 37.7 / 39.7** | |
| Nov 3 | E5-2667 v4, X99 | 0.1.55 | 32.6 / 30.4 / 31.5 | |
| Nov 21 | E5-2696 v4, X99 | 0.1.55, setup defaults | 26.0 / 26.7 / 27.5 | 428 / 681 |
On the same engine the 2666 v3 and the 2696 v4 are within −10…+4% in decode. The Gold 6254 box lost half its decode overnight when the engine went from 0.1.52 to 0.1.55, with nothing else changed.
#### Quality check
To make sure the faster engine (and tail skip on it) doesn't trade quality for speed, I ran exact-answer tests at temperature 0:
| Test | 0.1.52 PIN0 | 0.1.52 PIN0 + tail skip 7 | 0.1.63 PIN0 + tail skip 7 |
|---|---|---|---|
| 20 short questions (math, logic, facts, code output; RU + EN) | 20/20 | 20/20 | 20/20 |
| 37 harder tasks: 16 multi-step math, 10 code tasks checked by running unit tests, 5 fact-retrieval in 16K/32K-token documents, 6 format-following | 37/37 | 37/37 | 37/37 |
| standard-v1 sanity | 12/12 | 12/12 | 11/12 (T2 missed a tool call) |
I found no quality difference, only the speed difference.
#### Related
- #6208: 0.2.15 decodes 28–36% slower than 0.2.7 on an RTX 3060, `--expert-cache auto` starts with a much worse resident set. Same symptom: fewer cache hits with the same pack and profile.
- #5115: 0.1.56 25% slower than 0.1.54.
Questions:
1. Is the 0.1.52 → 0.1.55 change in how `--expert-cache auto` builds the resident set (or how the prompt path borrows expert slots) the known cause? Is there a setting in 0.1.63 that brings back the 0.1.52 behaviour? (`STRATA_NV_CHUNK=0`, suggested in #6208, is next on my list.)
2. Is running engine 0.1.52 under Control 0.1.63 a supported combination for now, or are there fixes after 0.1.52 that make it unsafe (crashes, wrong output)? I am running it this way now.
_Prepared with AI assistance. All numbers are from my own runs, and I can share the full engine logs._
### Smallest steps or command that reproduce it
Model: unsloth `Qwen3.8-Flash-Next` UD-Q4_K_XL, all 4 shards, pack `unsloth-ud-q4_k_xl`. Engine arguments (paths shortened), identical for every engine; only the `exe` in the run config changes:
```
--pack <packs>\unsloth-ud-q4_k_xl --native <models>\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
--expert-profile <terminal>\data\expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5
--mtp <mtp>\rt --max-context 262144 --kv int8 --kv-resident 32768
```
Run config: `"gpu_roles": {"0": "helper", "1": "main"}`, `"drafting": "mtp"`, `env` as in the tables.
Steps:
1. Point the run config's `exe` at `components\engine\0.1.52-windows-x86_64-cuda-cuda13\strata.exe`.
2. `strata serve --config <cfg>.json --host 0.0.0.0 --port 8080`, wait for `loaded`.
3. `strata bench run --url http://127.0.0.1:8080 --suite standard-v1 --skip-model-hash`.
4. Read the `decode expert cache N% hit` lines in the engine log.
5. Repeat with `exe` pointed at the 0.1.55 and 0.1.63 engines.
### Reproduction status
Reproduced locally
### Strata version
Strata 0.2.17 (Control 0.1.63, engine 0.1.63); compared against engines 0.1.52 and 0.1.55 kept by the updater under `components\engine\`, all run under Control 0.1.63
### Graphics card, memory and system
RTX 3080 Ti 12 GB (main) + RTX 4070 12 GB (helper, also drives the display), driver 617.42, both PCIe 3.0 x16 · Xeon E5-2696 v4 22C · 4×64 GB DDR4-2400 · Windows 11 (10.0.26200), page file 8 GB
### Model, size and settings
unsloth Qwen3.8-Flash-Next UD-Q4_K_XL · 256K context · `--kv int8` · MTP drafting on (DFlash2 for the tuned 0.1.63 row) · pictures off
### Logs
The `decode expert cache hit rate` and `expert cache N slots` lines quoted above come from the engine logs; I can attach the full logs for every row.
_Prepared with AI assistance. All numbers are from my own runs, and I can share the full engine logs._
No site
Links install, modelos, releases.