贡献 / #1879
#1879 bench: community report, Tesla V100-SXM2 16 GB (Linux, sm_70), IQ3_S:…
open · @taxah92 · 0 评论 · 去 GitHub 看
BenchmarksNVIDIA / CUDAModels & quantsDocumentationLinux
说明
A community report for `bench/results/`: a same-day A/B of **0.1.41** (`fb58e0d`) against **0.1.42** (`61b3fb5`) on one Tesla V100-SXM2 16 GB (sm_70, Linux in a Proxmox guest, PCIe Gen3 x16, the engine's start-up probe measures 13.1 GB/s host→device). - **Hardware:** Tesla V100-SXM2-16GB, EPYC 7532 (32 vCPU), 94 GiB RAM, Ubuntu 26.04, CUDA 12.4.131; source builds for sm_70 (`-DSTRATA_EXPERIMENTAL_SM60=ON`), no local patches in either tree. - **Model and config:** Flash-Next GSQ-RCO IQ3_S, 1M YaRN, k8v4 KV, `--spec 3` with MTP, `--pcie-frac` unset in both arms; `STRATA_ROUTE_TAIL_SKIP=7` pinned in the 0.1.42 arm only (its new default). - **Results:** decode **+34…+62%** (medians of three runs, ranges do not overlap), prompt throughput unchanged, `tools/needle_bench.py` 32k+128k **6/6 on both versions**, and the same answers on an own 128K Russian needle at depths 10/50/90. - **What the log shows:** the measured PCIe share moved `pcie_frac` from the start-up rule's 0.36 to **0.10–0.15, at or on the floor of the 0.10–0.60 search range** (it re-measures up to three times per start and follows the CPU-pool timing: 45–52 GB/s against the 13.1 GB/s link); the tail-skip counter skipped 151,008 of 158,991 missed-expert entries. Expert-cache slot counts are identical on both versions, so the gain is not a different number of resident experts. One aside: `/metrics` keeps reporting the start-up `pcie_frac` (0.36) and does not follow the adaptive value. - **Question for the maintainers:** the share search ends at its floor on this host — is 0.10 the right floor for machines whose CPU pool is 3.5–4× their PCIe link? - **Limits:** one machine; the two new defaults were not separated (no `STRATA_ROUTE_TAIL_SKIP=0` or `STRATA_PCIE_FRAC_DEFAULT=old` arm); the long-context arms are single runs; TTFT was not measured (non-streaming requests); no KL/perplexity. Raw JSONs of both phases and the measurement stand (`ab.py`) are in the folder. This PR adds the folder, one row in `bench/results/COMMUNITY.md` and one entry in `docs/COMMUNITY_BENCHMARKS.md` — **no engine changes**.
本站相关内容
相关页面的快捷入口。