反馈 / #1884
#1884 0.1.42 on a V100 (99.9% GPU hit rate, rest pinned in RAM): route tail skip is ~1–2% slower decode, from lower draft acceptance
closed · @christopherrobertbrooks-tech · 4 评论 · 去 GitHub 看
说明
Tested v0.1.42 against v0.1.41 on one Tesla V100-PCIe-32GB (i7-13700KF, 16 GB RAM, low-RAM mode), Coder IQ1_M with `--vision` on, so the cache holds 11,644 of 12,288 experts and the rest sit in page-locked RAM ("resident RAM mode", no file reads). Same config for both, `STRATA_SM70_TABLE=1`, `STRATA_PREFILL_CPU_SHARE=0`, built with `-DSTRATA_EXPERIMENTAL_SM60=ON`. 3 interleaved rounds, 3 runs per size, medians:
| prompt | 0.1.41 decode | 0.1.42 decode | |
|---|---|---|---|
| 256 | 78.4 | 76.5 | −2.4% |
| 1,000 | 77.6 | 77.0 | −0.8% |
| 4,096 | 78.4 | 76.9 | −1.9% |
| 16,384 | 75.4 | 74.2 | −1.6% |
Prompt time was unchanged at every size. The tail skip did fire (about 1,000 missed experts skipped per run), but the GPU hit rate was already 0.999, and draft acceptance fell from 70.1% to 68.2%, which seems to cost more than the skip saves when the misses come from pinned RAM rather than PCIe. This matches the notes ("a machine that holds nearly every expert gains nothing"). Maybe the default could also stay off in resident RAM mode, the way it already does when every expert is in VRAM? We're staying on 0.1.41 for now; we didn't measure 0.1.42 with `STRATA_ROUTE_TAIL_SKIP=0`.
本站相关内容
相关页面的快捷入口。