#6478 [Engine] Expert-cache hit rate and decode throughput regress from engine 0.1.52 to 0.1.62/0.1.63 on two 11 GB Turing cards (Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS)
open · @SugiKi1996 · 2 コメント · GitHub で見る
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsSecurity
本文
### Area Engine ### Desktop app, Terminal or source? Source — own CMake build of the engine per version (no desktop app, no Terminal bundle). ### What happened Expert-cache hit rate and decode throughput regress from engine **0.1.52 to 0.1.62 / 0.1.63** on this machine, with the same config, the same pack and the same prompts; only the engine binary changes. **Engine 0.1.53 is clean**, so the regression window is **0.1.54 – 0.1.61**. Another user reported a similar slowdown on a 3090 in the same period, so it may not be Turing-only. **A/B, ABBA order (A1 → B1 → B2 → A2), 4 prompts × 2 rounds per arm, warm-up discarded:** | arm | samples | decode median | decode mean | expert-cache hit (mean) | |---|---:|---:|---:|---:| | **0.1.52** (current production) | 16 | **62.9 tok/s** | 63.2 | **96.8 %** | | 0.1.62 | 4 | 40.3 tok/s | 40.2 | 70.2 % | | 0.1.63 (app 0.2.17) | 16 | 35.3 tok/s | 34.9 | 64.6 % | Both A rounds land at 61–63 tok/s and both B rounds at 35 tok/s, so this is not run-to-run noise. The first request of 0.1.63 is the worst: 27.9 tok/s at 49.7 % hit (0.1.52 first request: 67.1 tok/s at 95.9 %). **Version bisect (one build and one run per version, same config and prompts):** | engine | app tag | decode | expert-cache hit | verdict | |---|---|---|---:|---| | 0.1.52 | our tree | 65.8 median | 96.5 % | good | | **0.1.53** | v0.2.7.1 | **59.5 – 65.9** | **95.8 – 97.4 %** | **good** | | 0.1.55 | v0.2.9 | — | — | not built here: configure fetches llama.cpp through `FetchContent` when `STRATA_GGML_DIR` is empty, and the download did not finish | | 0.1.62 | v0.2.16 | 40.3 | 70.2 % | regressed | | 0.1.63 | v0.2.17 | 33.5 | 64.5 % | regressed further | **Single-variable follow-up on 0.1.63 (each row changes exactly one thing):** | case | decode median | hit median | |---|---:|---:| | 0.1.63 baseline | 33.5 | 64.5 % | | `STRATA_POOL_DEP=0` | 32.8 | 62.1 % | | `STRATA_EXPDEC=0` | 33.0 | 63.4 % | | `STRATA_PIN_BY_NEED=0` | 32.8 | 62.1 % | | `STRATA_ROUTE_TAIL_SKIP=7` (old behaviour) | 41.5 | 72.0 % | | projection removed entirely (`--control-vector*`, `--cvec-*` dropped) | 30.7 | 63.4 % | Route-tail-skip accounts for only ~8 tok/s of the ~22 tok/s gap; the three expert-path switches do nothing here; removing our speed projection does not help either, so this is not an interaction with the projection. **Suspected area:** expert-cache residency/replacement. With the same slot budget the hit rate stops climbing at ~65 % instead of ~97 %, and every miss costs a PCIe read on a 6.2 GB/s link, which is what shows up as the decode loss. Nothing about this combination appears in the release notes for 0.2.9 – 0.2.17, so we cannot tell from them whether it is expected. ### Reproduction status Reproduced on this machine, on demand: same config file, only `exe` swapped, ABBA order, 2 rounds × 4 prompts per arm, measured 2026-11-20. Each engine is served through the `terminal/serve/server.py` that ships with its own source tree. Decode tok/s and `expert cache N% hit` are read from the engine's own log lines, not from client timing. Load time ~105–110 s per arm. ### Strata version Engines tested: **0.1.52** (good, current production), **0.1.53** (good), **0.1.62** (regressed), **0.1.63** (regressed further). App tags: v0.2.7.1 ships 0.1.53, v0.2.16 ships 0.1.62, v0.2.17 ships 0.1.63. ### Graphics card, memory and system 2 × RTX 2080 Ti 11 GB (sm_75), PCIe 3.0 x8 each, measured link 6.2 GB/s, no NVLink. 16 threads (8 physical cores), no ECC, 61 GiB RAM (~50 GiB free while serving). Ubuntu 26.04, NVIDIA driver 595.99.02, CUDA 13.2 runtime, toolkit 13.1. Model: Qwen3.8-Flash-Next, GSQ-RCO **IQ3_XXS** (2 shards), pack `qwen38-flash-original`. Build flags: `-DCMAKE_CUDA_ARCHITECTURES=75 -DSTRATA_NATIVE_EXPERTS=ON -DSTRATA_GEOMETRY=flash_next -DSTRATA_OWN_CPU_QUANT=ON -DSTRATA_PORTABLE=ON`, Ninja Release, nvcc 13.1. Launch line: layer split 24 (48 layers), `--expert-cache 2700 --expert-cache-per-layer`, `--prefill auto`, `--spec 4 --spec-min-p 0.5`, MTP draft head, `--max-context 131072 --kv q4_0`, `--vram-reserve-mib 600 --vram-reserve-later-mib 1082`, `STRATA_STAGE_TRIM=1`, control-vector speed projection (`--control-vector-scaled …:1.0 --control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer`). ### Ask 1. Is a hit-rate drop of this size on a GSQ-RCO pack expected in 0.1.54+, or is it a regression? 2. Which release between 0.1.53 and 0.1.62 changed expert-cache residency, so we can bisect in the right place? 3. Any knob we should try that we have not (we hold production at 0.1.52 meanwhile). We can run any additional A/B you want on this machine and report back with the same method.
関連リンク
インストール・モデル・リリースへのサイト内リンク。