#6381 [Benchmark] 6x Tesla V100 16GB (sm_70): 0.1.63 with exact routing - prefill parity kept, decode +7% over the rank-7-skip 0.1.62, and the single-stream prefill gain confirmed on this rig
open · @Barmallini · 0 comentarios · En GitHub
BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsLinux
Descripción
## Problem and result Three measurements on the same 6x V100 box, one question per line: 1. **Does the 0.2.16/0.2.17 prefill tuning reach a concurrent shape?** Under our production load (4 concurrent 26K prompts on a 6-GPU layer split) 0.1.63 reads prompts at parity with 0.1.58 (849.7 vs 856.3 tok/s mean, -0.8%, inside run-to-run spread). The release-notes gain (+16% on a single 26K prompt) did not show under 4-way concurrency on 0.1.62, and 0.1.63 does not change that. The gain itself is real on this rig: a cold 119,152-token single request read at **1,074 tok/s** on 0.1.62 in production logs (Nov 19), where this box's older engines served single 11K reads at ~545 tok/s. 2. **What does exact routing cost?** Nothing here - it pays. 0.2.17 turns `STRATA_ROUTE_TAIL_SKIP` off by default (0.2.16's "Automatic" was rank 7 on CUDA, so our 0.1.62 prod ran the skip without us choosing it). The feared decode penalty did not materialize on this box: decode went **86.6 -> 92.6 tok/s (+7%)** between 0.1.62 (skip on) and 0.1.63 (exact routing), with identical draft acceptance (159 of 249) on the identical probe prompt. We stay on exact routing. 3. **Worth the update?** Yes: same prefill as 0.1.58, faster decode than 0.1.62, exact routing, plus the 0.2.17 tool-call parsing fixes and the split/PLE fixes (#6149, #6084, #5326, #4682) that sit directly in our configuration's paths. ## Machine and config - Proxmox VM, 6x Tesla V100 16GB (sm_70), 2x Xeon E5-2695 v4, 88 GiB RAM, Linux - 0.1.62 and 0.1.63: the prebuilt linux-x86_64-cuda-cuda12 engines (BUILD.json archs "60;61;70;75;80;86;120;89", CUDA 12.9); sm_70 confirmed in the fatbin (54 ELFs each). 0.1.58: v0.2.12 built from source, CUDA 12.9, `-DCMAKE_CUDA_ARCHITECTURES=70 -DSTRATA_EXPERIMENTAL_SM60=ON` - Qwen3.8-Flash-Next GSQ-RCO IQ3_S: native pack + the model's own second GGUF shard as `--ple-gguf`, all 24,576 experts resident, explicit layer split 9,18,27,35,43 + `--trim-stage-weights`, `--batch 4`, `--kv int8 --kv-resident 262144`, control-vector projection (layers 4-44), 262,144 context, conversation cache 20,480 MiB / 6 slots, `--spec 4 --spec-min-p 0.30`, `--pool-workers 29` - `STRATA_ROUTE_TAIL_SKIP` unset in every arm's env (0.1.62 read its own default = rank 7 on CUDA; 0.1.63 reads unset as off) - 0.1.63 start line: `session is up (engine 0.1.63)`; prompt path: `every chunk runs 48 of 48 layers on the int8 kernels`, `100% of the experts resident, the prompt path's streamed ring 96 slots` - New in 0.1.63 on this box: the residency policy chose `--ple-io ram` on its own (the 26.8 GiB PLE table sits in reclaimable file-backed cache, not RSS) and the engine suggests `LimitMEMLOCK=infinity` for up to 6% faster decode (#4671) - not applied yet, so the decode numbers above are unlocked ## Method 4 concurrent chat completions of 25,962/25,963 tokens each (fresh nonce prefix per run, engine log confirms `0 reused` = cold read), metric is the engine serve line `prompt N tokens = 0 reused + N read in X ms (Y tok/s)`. 2 runs per engine arm on a fresh start; the one warmup slot per arm is excluded and shown. The 0.1.63 bench ran on an isolated port with production clients refused - zero external traffic during measurement. The 0.1.62 arm was measured the same way on Nov 19; the 0.1.58 reference is the Nov 16 same-config rerun from #6053. Decode probe: 496-token prompt, 256 generated, temperature 0, same prompt text in both arms. ## Measurements Prompt read, tok/s (each cell one slot, engine serve line): | | 0.1.58 (Nov 16) | 0.1.62 (Nov 19) | 0.1.63 (Nov 20) | |---|---|---|---| | run 1 | 857.5 / 861.5 / 862.3 / (814.9 warmup) | 826.5 (warmup) / 854.0 / 850.1 / 835.3 | 826.8 (warmup) / 852.8 / 851.0 / 849.6 | | run 2 | 847.4 / 853.5 / 855.0 / 856.9 | 845.1 / 853.8 / 847.3 / 837.5 | 847.6 / 853.5 / 847.6 / 855.9 | | mean / median (excl. warmup) | 856.3 / 856.9 | 846.1 / 847.3 | 849.7 / 851.0 | | wall per request | 124.0 - 124.9 s | 124.8 - 126.0 s | 124.0 - 125.6 s | Decode probe (256 tokens, temperature 0): | | 0.1.62 (rank-7 skip on) | 0.1.63 (exact routing) | |---|---|---| | tok/s | 86.6 | 92.6 | | drafts accepted | 159 of 249 | 159 of 249 | Single-stream observation (not a controlled bench): on 0.1.62, one cold production request of 119,152 tokens read in 110.9 s = **1,074 tok/s** (`0 reused`), decode 2,253 tokens at 81.0 tok/s with 1,452 of 2,524 drafts accepted at full 137K context. ## Validation - Passed: every request of every run returned a token; `--ple-gguf` with the model's own second shard boots on 0.1.63 (the #5942 line of fixes - #5326 file identity, #4682 PLE-reader watchdog - is in); the vision tower and the MTP draft layer load; the bench port carried zero external requests (8081 refused connections during measurement). - Failed: nothing. Two startup notes, both ours to fix, not engine bugs: the v0.2.16+ checkout removes the tracked top-level `data/` (layout move 4058aa782), so `data/expert-profile.bin` must become `terminal/data/expert-profile.bin` in configs; and sequential per-card loading in the split search is by design (first one card, then the rest). - Not run: quality/KL of exact routing against rank-7 on our prompts (upstream measured the KL; our probe's draft acceptance was identical, which is suggestive, not a quality measurement); a controlled single-stream 26K arm on 0.1.63 (we only have the 0.1.62 production observation); `--peer-device` on our 6-card split (upstream measured +9-10% resident / +21-26% half-resident on 2x V100 with the #6023 share default fixed to 0; our topology is 6 cards with two NVLink pairs and one PCIe crossing - untested); `LimitMEMLOCK=infinity`; a soak long enough to re-check the engine RSS growth we watched on 0.1.62 (76.7 GiB after 19.5 h there). ## Limits - One box, one quant (IQ3_S), one shape (concurrent 4x26K) for the controlled bench; the single-stream line is a production observation with uncontrolled cache state, offered as direction, not a measurement of the release-note number. - The 0.1.58 column is a day-before (Nov 16) same-config rerun, not same-day; its decode figure is an era number (~86 tok/s, 0.1.44-0.1.58 line), so the decode comparison that matters is 0.1.62 vs 0.1.63, measured same-box same-week. - -0.8% prefill against 0.1.58 is within this box's spread between runs of the same engine; we claim parity, not a regression or a gain. ## For other source installs on the old layout Since 0.2.16 the prebuilt cuda12 engines ship sm_70 (54 ELFs, archs 60-120) - a Volta box no longer needs a source build. After checking out v0.2.16+: `serve/` is `terminal/serve/`, `data/` is `terminal/data/` (tracked paths moved in 4058aa782), and configs must point at the new paths. The `--pcie-frac` knob is gone; 0.1.63 probes PCIe per card at start (this box: 3.2-5.8 GB/s reads, pcie_frac 0.09-0.16 across starts) - removing the flag was a clean trade for us. ## Related - #6053 (closed): the 0.2.14 -11% sm_70 report and the community pile; #6066: sm_75 arms. - #6023: the V100 peer prompt path (NaN at share 35%, default 0 now) - the opt-in we plan to try next on our NVLink pairs. - #4671: the PLE memlock decode headroom.
En el sitio
Enlaces a install, modelos, releases.