#6208 [Engine] 0.2.15 decodes 28-36% slower than 0.2.7 on RTX 3060 12 GB: --expert-cache auto starts with a much worse resident set (same pack and profile)

open · @0xluk · 6 Kommentare · Auf GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Beschreibung

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

With the same model files, the same pack and the same expert profile, 0.2.15 decodes 28–36% slower than a 0.2.7-based build on an RTX 3060 12 GB. The decode expert-cache hit rate falls from ~96% to ~75–81%. The clearest signal is with adaptation off (`--adapt-swaps 0`): the resident set that `--expert-cache auto` builds from the profile hits 25–42% of routed experts on 0.2.15, against 72–81% on 0.2.7. Swapping then has to repair a much worse starting set, and it only gets part of the way.

I have not bisected it yet, so I don't know which commit changed this. I can bisect if that helps.

#### Measurements

1 agent, IQ3_S, one GPU, 131K context, `--kv int8`, MTP drafting on. "decode" = two fixed 400-token greedy chat answers. "new-prompt decode" = two 300-token answers on topics that are new in every run. Speed = (completion tokens − 1) / (last token time − first token time). Hit rate and PCIe share come from the server's `decode expert cache hit rate` lines. 1 run per row.

**System A** (Xeon E5-1650 v2 Ivy Bridge, AVX only; 64 GB DDR3-1600; RTX 3060 12 GB at PCIe 3.0 x16, capped at 115 W; 5 pool workers)

| Engine | Setting | Decode tok/s | New-prompt decode | Cache hit | Routed over PCIe |
|---|---|---|---|---|---|
| 0.2.7-based | default | **55.6** | 52.6 | 95.7–96.2% | 0.8–1.0% |
| 0.2.7-based | `--adapt-swaps 0` | 44.8 | 36.8 | **75.0–81.3%** | 5.0–6.6% |
| 0.2.15 `c7757280` (unmodified) | default | **34.0** | 31.2 | 78.1–81.0% | 15.4–17.1% |
| 0.2.15 `c7757280` (unmodified) | `--adapt-swaps 0` | 24.9 | 20.9 | **27.1–41.5%** | 37.5–42.7% |

**System B** (Xeon E5-1650 v3 Haswell, 6 cores for the engine; DDR4; RTX 3060 12 GB at PCIe 3.0 x16; both engines built AVX-only for a like-for-like comparison; 5 pool workers)

| Engine | Setting | Decode tok/s | New-prompt decode | Cache hit | Routed over PCIe |
|---|---|---|---|---|---|
| 0.2.7-based | default | **54.9** | 49.7 | 95.0–96.0% | 0.5–0.8% |
| 0.2.7-based | `--adapt-swaps 0` | 45.0 | 37.2 | 72.1–79.4% | 4.0–5.3% |
| 0.2.15 + local commits* | default | **39.7** | 37.8 | 73.6–74.6% | 10.7–10.9% |
| 0.2.15 + local commits* | `--adapt-swaps 0` | 30.9 | 26.4 | 25.1–32.1% | 26.7–28.5% |
| 0.2.15 + local commits* | `STRATA_ADAPT_FETCH=1` | 44.2 | 40.7 | 79.6–82.9% | 8.0–10.0% |

\* 0.2.15 with five local prefill/serve commits ported on top. On System A this build decodes the same as unmodified 0.2.15 (34.6 vs 34.0 tok/s, static-cache hit 30–40% vs 27–42%).

0.2.15 also gets slightly fewer cache slots: `expert cache 2513 slots, 4.82 GiB` against `2638 slots, 5.06 GiB` on 0.2.7. That 5% doesn't explain a static hit rate that falls from ~78% to ~33%.

#### What I ruled out (System A, 0.2.15)

| Tried | Decode tok/s | Cache hit | Note |
|---|---|---|---|
| default, for reference | 34.6 | 79–81% | |
| `--pcie-frac 0.35` (auto had picked 0.60) | 34.9 | 74–75% | PCIe share drops to ~7%, decode unchanged |
| `--lookup-max-t 0` | 34.7 | ~81% | |
| `--pcie-frac 0.35 --lookup-max-t 0` | 33.8 | 73–74% | |
| `--adapt-swaps 24` | 34.1 | 69–72% | |
| `--adapt-every 8` | 33.8 | 73–80% | |
| `--adapt-async 1` | 35.5 | 79–82% | |
| `STRATA_ADAPT_FETCH=1` | 41.5 | 86–87% | the only setting that helps (+20%) |
| `STRATA_ADAPT_FETCH=2` | 40.7 | 85–88% | |
| A new profile: `tools/collect_profile_traces.py --mode routing` on 20 prompts of `bench/profile_corpus.json`, then `tools/make_profile_mix.py` | 34.1 | 78% (static 27–34%) | no change from the shipped profile |
| Re-pack with 0.2.15's `tools/iq_pack.py` | 35.2 | 81–82% (static 31–33%) | `index.txt` and `native_experts.txt` are byte-identical to the old pack |
| `STRATA_PREFILL_LEND_PCT=0` | — | — | not usable: most requests returned no tokens |

Also checked:
- **Server:** each engine ran with the `serve/server.py` of its own tree.
- **Profile:** both engines read the same `data/expert-profile.bin`, with an identical SHA-256.
- **Our refill patch:** the 0.2.7-based build was rerun with its optional async-refill patch off. Default: 55.6 / 96%. Static: 44.8 / 75–81%.

#### Related

- #4870: adaptive-tier defaults cost decode on a 2× RTX 3090 split.
- #5000: route prior on 8 GB cards.

Both involve the cache tier, but neither describes the initial resident set getting worse between versions.

_Prepared with AI assistance. All numbers are from my own runs, and I can share the full engine logs._

### Smallest steps or command that reproduce it

Model: ISTA-DASLab `Qwen3.8-Flash-Next-GSQ-RCO-GGUF` IQ3_S, both shards, pack `iq3_s`. Engine arguments (paths shortened):

```
--pack <packs>/iq3_s --native <models>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf
--ple-gguf <models>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
--expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5
--mtp <mtp>/rt --max-context 131072 --kv int8 --ple-io direct --shared-expert-arena <tmpfs>/strata-iq3s.arena
--kv-resident 20480 --pool-cores 1,2,3,4,5 --mtp-draft-vocab data/draft_vocab_en.bin --conversation-cache-mib 1024
```

Environment for every run: `STRATA_PREFILL_CPU_SHARE=auto STRATA_PREFILL_CPU_SHARE_MAX=4608 STRATA_PF_AHEAD=auto STRATA_KV_PREFETCH=1 STRATA_CKPT_POOL=1 STRATA_CKPT_POOL_READY=4`. Some local variables were also set; 0.2.15 doesn't read them.

Steps:
1. Start the server: `serve/server.py --engine strata --config <cfg> --port 8090`.
2. Send one 400-token warm-up request.
3. Send two greedy chat requests, e.g. "Explain in detail how a modern CPU executes instructions…", `max_tokens` 400.
4. Read the `decode expert cache hit rate` lines.
5. Repeat with `--adapt-swaps 0` added.

### Reproduction status

Reproduced locally

### Strata version

Strata 0.2.15 (`c7757280`), engine built from source with `STRATA_ISA_FLOOR=avx` (CUDA arch 86); compared against 0.2.7 (`5e8a4cc`) plus local arena-reuse and prefill commits (the decode expert-cache code is 0.2.7's)

### Graphics card, memory and system

RTX 3060 12 GB (driver 595.104.02, CUDA 13 runtime) · System A: Xeon E5-1650 v2, 64 GB DDR3 · System B: Xeon E5-1650 v3, DDR4 · Linux 7.0 (Debian-based), engine in a container. System A's cards use a third-party FP16 rate-unlock kernel module; System B shows the same pattern.

### Model, size and settings

ISTA-DASLab Qwen3.8-Flash-Next GSQ-RCO IQ3_S · 131K context · `--kv int8` · MTP drafting on · pictures off

### Logs

The `decode expert cache hit rate` lines quoted in the tables come from the engine logs; I can attach the full logs.

_Prepared with AI assistance. All numbers are from my own runs, and I can share the full engine logs._

Mehr auf der Site

Links zu Install, Modellen, Releases.