#6511 [Benchmark] 2x Tesla P40 (sm_61) split: 0.2.17 gives +9% decode, but 220K prefill is still ~5% under the 0.2.13.1 MMQ path (same-day A/B)
open · @lyesrock · 1 comments · View on GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationLinux
Description
## Hardware and software - 2x Tesla P40 24 GiB (sm_61), PCIe gen3 x8, no P2P; 2x Xeon E5-2620 v4 (16c/32t), 92 GiB RAM; Linux 6.14, driver 580.196.07, CUDA 12.9. - Builds per docs (AMD_HIP/Pascal notes): `-DSTRATA_ENABLE_CUDA=ON -DSTRATA_EXPERIMENTAL_SM60=ON -DCMAKE_CUDA_ARCHITECTURES=61`, `STRATA_MMQ` off (the 0.2.14+ prompt kernels). Engine binaries kept side by side, same server config, fresh boots. - Model: Qwen3.8-Flash-Next IQ3_S (ISTA-DASLab), context 262144, `layer_split auto`, `parallel 2`, `--prefill auto --spec 3 --spec-min-p 0.7 --kv int8 --kv-resident 32768 --pcie-frac 0.0`, plus our opt-in `STRATA_SPLIT_LONG_LOOKUP=1` (#4928, dormant at batch 2). ## Method `terminal/tools/strata_bench.py` against the running server (suite 1, medians), 2026-11-21, 0.2.16 (0.1.62) fresh boot first, then 0.2.17 (0.1.63) fresh boot; then the A binary (0.2.13.1 + our split-lookup PR, engine-cuda12/strata-numa-wide) booted under the same 0.2.17 serve the same day and hit three times with a ~222K-token random-tagged prompt (rbench prefill1, client-side rate); then back to 0.2.17. A sanitized suite README (pre-sanitized by the tool) is available and I can open a PR adding the folder to docs/bench-results if wanted. ## Results | metric | 0.2.16 (0.1.62) | 0.2.17 (0.1.63) | |---|---|---| | chat decode tok/s | 39.70 (35.9-51.3) | 43.30 (39.5-53.9) **+9%** | | chat first token s | 0.70 | 0.68 | | 4K prompt tok/s | 358.0 | 359.7 | | 16K prompt tok/s | 506.6 | 500.6 | | agent decode tok/s | 43.30 | 47.30 **+9%** | | agent turn first token s | 5.05 | 5.07 | | agent cache hit % | 71.08 | 71.06 | Same-day 220K prefill (rbench, ~221-226K tokens, three runs each): | build | prompt tok/s | |---|---| | 0.2.13.1-era binary (MMQ prompt path, kept as `strata-numa-wide`) | 685.3 / 678.1 / 684.0 = **682.5** | | 0.2.17 (new prompt kernels + default CPU prefill share, f-bw-f/f-bw-h) | 647.6 / 648.8 / 649.2 = **648.5 (-4.9%)** | | 0.2.14 (measured 2026-11-16 at 240-250K: 635.9; today at ~222K: ~648) | — | So on this 2xP40 split the decode side improved twice this cycle (0.2.16: +7% over 0.2.14-era, 0.2.17: +9% over 0.2.16), but prefill never came back from the MMQ-to-kernels switch: f-bw-f's default CPU prefill share buys ~+2% over 0.2.14 and is still ~5% under the MMQ path. TTFT for a 222K prompt: ~325 s (0.2.13.1) vs ~345 s (0.2.17). ## What we did NOT test - Correctness (speed only); thinking on; the standard `strata bench run` protocol (suite 1 of strata_bench.py used instead). - 0.2.17 prefill switches individually (STRATA_PREFILL_CPU_SHARE_AUTO, the 3,072/8 old limits) — we run defaults. - One thing clients should know: 0.2.17 now rejects `prompt + answer > window` with a clean 400 instead of truncating — our rbench prefill (800k chars = 262.6K tokens) started 400ing where 0.2.16 truncated silently.
Related on this site
Quick links to related pages.