#6632 [Engine] sm_86 Windows 16 GB: engine 0.1.65 decodes 30-40% slower than 0.1.56/0.1.57 (RTX 3080 Ti Laptop)

open · @ByteBrainMaster · 2 评论 · 去 GitHub 看

BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

说明

**Problem:** On Windows with an `sm_86` 16 GB laptop GPU, `strata-engine` **0.1.65** decodes 30-40% slower than 0.1.56/0.1.57 on the same models, prompts and run config. v0.2.19 states the opposite direction for `sm_86` (RTX 3090 & 2x RTX A4000: decode +13%..+20%).

**Environment:**
- Laptop ASUS ROG Zephyrus M16 (GU603ZX); Windows 11 10.0.26300
- GPU NVIDIA RTX 3080 Ti Laptop, 16 GB, **sm_86**, driver 621.24, PCIe 4.0 **x8**
- CPU i9-12900H (6 P + 8 E), 47.7 GB RAM
- Strata app 0.2.19, control 0.1.65
- Engines compared: `0.1.52`, `0.1.56`, `0.1.57`, `0.1.65`, all `windows-x86_64-cuda-cuda13`
- Models: `coder-iq1_m` (IQ1_M pack), `swift-iq2_xs`, `iq3_xxs`, `gyro-gyro-s`
- Config of the coder run: `--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp ... --max-context 131072 --kv int8 --mmap-experts --pool-workers 9 --vram-reserve-mib 1280`

**Reproduce:**
1. Install each engine: `strata components use 0.1.57 --family qwen` (and `0.1.65` for the comparison).
2. Start the same model: `strata serve --profile %LOCALAPPDATA%\Strata\install\strata-coder-iq1_m.json --port 8080`.
3. Send one request of ~900 output tokens, twice, and read `timings.predicted_per_second`; or read the engine log line `... generated in N ms (X tok/s)`.

**Expected:** 0.1.65 decodes at least as fast as 0.1.57 on this card (release notes v0.2.19 promise +13%..+20% on `sm_86`).

**Actual:** 0.1.65 is ~35% slower, on every model tried.

Decode tokens/s, median of generations ≥ 100 tokens (samples from the engine's own request log on this machine; `n` in brackets):

| Model | 0.1.52 | 0.1.56 | 0.1.57 | 0.1.65 |
|---|---|---|---|---|
| `coder-iq1_m` | 29.4 (n=58) | — | 30.3 (n=4) | **19.3 (n=6)** |
| `swift-iq2_xs` | 32.4 (n=9) | 40.9 (n=4) | 36.1 (n=688) | **24.6 (n=3)** |
| `iq3_xxs` | — | 28.6 (n=13) | 23.7 (n=2) | **16.7 (n=3)** |
| `gyro-gyro-s` | 17.8 (n=2) | — | — | 17.8 (n=3) |

Direct back-to-back A/B, identical prompt, 900 output tokens, `coder-iq1_m`, 128K context, `--kv int8`:

| Engine | decode |
|---|---|
| 0.1.57 | 37.4 - 42.1 tok/s |
| 0.1.65 | 19.0 - 19.6 tok/s |

**Evidence:** with the *same* config, 0.1.65 leaves far less VRAM free and trips the engine's own warning:

```
strata verify: capturing the 3-token window (509 MiB of VRAM free)
strata serve: WARNING: only 493 MiB of VRAM free at the start of this request (below 1024 MiB):
when the card is this full the driver can page GPU memory to system RAM, decode falls to a few
tok/s and a window may never finish (#2065). Add --vram-reserve-mib 1280 ...
```

`--vram-reserve-mib` was already raised from 512 to **1280** for these runs; free VRAM still stayed at ~500-650 MiB in the verify windows on 0.1.65, while the same 1280 on 0.1.57 leaves ~2 GB free. On 0.1.56/0.1.57 the log reports `decode expert cache hit rate: 63.9% - 81.5%` for this config; on 0.1.65 it reports 53.9% and 63.9% at different points of the same kind of run.

Related notes from the docs: v0.2.19 says the new `sm_86` path needs *"zero environment variables"*, so there is no switch to A/B it on one machine; v0.2.4 says *"The RTX 30 kernels (sm_86) were also checked on an RTX A4000. Windows on RTX 30 is built but was not run here."*

**Reproduction status:** reproduced locally, on this one machine (four engines, four models, plus a direct A/B). I have not tested other `sm_86` cards or Linux.

**Question:** is the 0.1.65 `sm_86` decode path tuned for setups where the experts are VRAM-resident (the RTX 3090 / A4000 test boxes), and does it regress when only ~50-65% of the experts fit in VRAM (`--mmap-experts` on a 16 GB laptop)? If so, an opt-out (`STRATA_*`) or an automatic fallback when the expert-cache hit rate is low would let small-card users keep 0.1.57's speed. I can run any A/B and send full engine logs.

本站相关内容

相关页面的快捷入口。