#6208 [Engine] 0.2.15 decodes 28-36% slower than 0.2.7 on RTX 3060 12 GB: --expert-cache auto starts with a much worse resident set (same pack and profile)
open · @0xluk · 5 comentários · No GitHub
BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Descrição
### Area [Engine] (speed, kernels, models, the server and API) ### Desktop app, Terminal or source? Source (START-HERE.bat, setup.sh or a build) ### What happened With the same model files, the same pack and the same expert profile, 0.2.15 decodes 28–36% slower than a 0.2.7-based build on an RTX 3060 12 GB. The decode expert-cache hit rate falls from ~96% to ~75–81%. The clearest signal is with adaptation off (`--adapt-swaps 0`): the resident set that `--expert-cache auto` builds from the profile hits 25–42% of routed experts on 0.2.15, against 72–81% on 0.2.7. Swapping then has to repair a much worse starting set, and it only gets part of the way. I have not bisected it yet, so I don't know which commit changed this. I can bisect if that helps. #### Measurements 1 agent, IQ3_S, one GPU, 131K context, `--kv int8`, MTP drafting on. "decode" = two fixed 400-token greedy chat answers. "new-prompt decode" = two 300-token answers on topics that are new in every run. Speed = (completion tokens − 1) / (last token time − first token time). Hit rate and PCIe share come from the server's `decode expert cache hit rate` lines. 1 run per row. **System A** (Xeon E5-1650 v2 Ivy Bridge, AVX only; 64 GB DDR3-1600; RTX 3060 12 GB at PCIe 3.0 x16, capped at 115 W; 5 pool workers) | Engine | Setting | Decode tok/s | New-prompt decode | Cache hit | Routed over PCIe | |---|---|---|---|---|---| | 0.2.7-based | default | **55.6** | 52.6 | 95.7–96.2% | 0.8–1.0% | | 0.2.7-based | `--adapt-swaps 0` | 44.8 | 36.8 | **75.0–81.3%** | 5.0–6.6% | | 0.2.15 `c7757280` (unmodified) | default | **34.0** | 31.2 | 78.1–81.0% | 15.4–17.1% | | 0.2.15 `c7757280` (unmodified) | `--adapt-swaps 0` | 24.9 | 20.9 | **27.1–41.5%** | 37.5–42.7% | **System B** (Xeon E5-1650 v3 Haswell, 6 cores for the engine; DDR4; RTX 3060 12 GB at PCIe 3.0 x16; both engines built AVX-only for a like-for-like comparison; 5 pool workers) | Engine | Setting | Decode tok/s | New-prompt decode | Cache hit | Routed over PCIe | |---|---|---|---|---|---| | 0.2.7-based | default | **54.9** | 49.7 | 95.0–96.0% | 0.5–0.8% | | 0.2.7-based | `--adapt-swaps 0` | 45.0 | 37.2 | 72.1–79.4% | 4.0–5.3% | | 0.2.15 + local commits* | default | **39.7** | 37.8 | 73.6–74.6% | 10.7–10.9% | | 0.2.15 + local commits* | `--adapt-swaps 0` | 30.9 | 26.4 | 25.1–32.1% | 26.7–28.5% | | 0.2.15 + local commits* | `STRATA_ADAPT_FETCH=1` | 44.2 | 40.7 | 79.6–82.9% | 8.0–10.0% | \* 0.2.15 with five local prefill/serve commits ported on top. On System A this build decodes the same as unmodified 0.2.15 (34.6 vs 34.0 tok/s, static-cache hit 30–40% vs 27–42%). 0.2.15 also gets slightly fewer cache slots: `expert cache 2513 slots, 4.82 GiB` against `2638 slots, 5.06 GiB` on 0.2.7. That 5% doesn't explain a static hit rate that falls from ~78% to ~33%. #### What I ruled out (System A, 0.2.15) | Tried | Decode tok/s | Cache hit | Note | |---|---|---|---| | default, for reference | 34.6 | 79–81% | | | `--pcie-frac 0.35` (auto had picked 0.60) | 34.9 | 74–75% | PCIe share drops to ~7%, decode unchanged | | `--lookup-max-t 0` | 34.7 | ~81% | | | `--pcie-frac 0.35 --lookup-max-t 0` | 33.8 | 73–74% | | | `--adapt-swaps 24` | 34.1 | 69–72% | | | `--adapt-every 8` | 33.8 | 73–80% | | | `--adapt-async 1` | 35.5 | 79–82% | | | `STRATA_ADAPT_FETCH=1` | 41.5 | 86–87% | the only setting that helps (+20%) | | `STRATA_ADAPT_FETCH=2` | 40.7 | 85–88% | | | A new profile: `tools/collect_profile_traces.py --mode routing` on 20 prompts of `bench/profile_corpus.json`, then `tools/make_profile_mix.py` | 34.1 | 78% (static 27–34%) | no change from the shipped profile | | Re-pack with 0.2.15's `tools/iq_pack.py` | 35.2 | 81–82% (static 31–33%) | `index.txt` and `native_experts.txt` are byte-identical to the old pack | | `STRATA_PREFILL_LEND_PCT=0` | — | — | not usable: most requests returned no tokens | Also checked: - **Server:** each engine ran with the `serve/server.py` of its own tree. - **Profile:** both engines read the same `data/expert-profile.bin`, with an identical SHA-256. - **Our refill patch:** the 0.2.7-based build was rerun with its optional async-refill patch off. Default: 55.6 / 96%. Static: 44.8 / 75–81%. #### Related - #4870: adaptive-tier defaults cost decode on a 2× RTX 3090 split. - #5000: route prior on 8 GB cards. Both involve the cache tier, but neither describes the initial resident set getting worse between versions. _Prepared with AI assistance. All numbers are from my own runs, and I can share the full engine logs._ ### Smallest steps or command that reproduce it Model: ISTA-DASLab `Qwen3.8-Flash-Next-GSQ-RCO-GGUF` IQ3_S, both shards, pack `iq3_s`. Engine arguments (paths shortened): ``` --pack <packs>/iq3_s --native <models>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf <models>/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <mtp>/rt --max-context 131072 --kv int8 --ple-io direct --shared-expert-arena <tmpfs>/strata-iq3s.arena --kv-resident 20480 --pool-cores 1,2,3,4,5 --mtp-draft-vocab data/draft_vocab_en.bin --conversation-cache-mib 1024 ``` Environment for every run: `STRATA_PREFILL_CPU_SHARE=auto STRATA_PREFILL_CPU_SHARE_MAX=4608 STRATA_PF_AHEAD=auto STRATA_KV_PREFETCH=1 STRATA_CKPT_POOL=1 STRATA_CKPT_POOL_READY=4`. Some local variables were also set; 0.2.15 doesn't read them. Steps: 1. Start the server: `serve/server.py --engine strata --config <cfg> --port 8090`. 2. Send one 400-token warm-up request. 3. Send two greedy chat requests, e.g. "Explain in detail how a modern CPU executes instructions…", `max_tokens` 400. 4. Read the `decode expert cache hit rate` lines. 5. Repeat with `--adapt-swaps 0` added. ### Reproduction status Reproduced locally ### Strata version Strata 0.2.15 (`c7757280`), engine built from source with `STRATA_ISA_FLOOR=avx` (CUDA arch 86); compared against 0.2.7 (`5e8a4cc`) plus local arena-reuse and prefill commits (the decode expert-cache code is 0.2.7's) ### Graphics card, memory and system RTX 3060 12 GB (driver 595.104.02, CUDA 13 runtime) · System A: Xeon E5-1650 v2, 64 GB DDR3 · System B: Xeon E5-1650 v3, DDR4 · Linux 7.0 (Debian-based), engine in a container. System A's cards use a third-party FP16 rate-unlock kernel module; System B shows the same pattern. ### Model, size and settings ISTA-DASLab Qwen3.8-Flash-Next GSQ-RCO IQ3_S · 131K context · `--kv int8` · MTP drafting on · pictures off ### Logs The `decode expert cache hit rate` lines quoted in the tables come from the engine logs; I can attach the full logs. _Prepared with AI assistance. All numbers are from my own runs, and I can share the full engine logs._
No site
Links install, modelos, releases.