Pull requests / #2011
#2011 STRATA_ADAPT_OVERLAP: opt-in on every backend (RTX 5090 Laptop, Linux CUDA: 84.6 vs 106.6 tok/s decode median with it); log the state in both directions
open · @Krypto-Whitehat · 0 Kommentare · Auf GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Beschreibung
Issue: Resolves #2010 ## Summary `STRATA_ADAPT_OVERLAP` is opt-in on every backend now, and the startup log names its state (with the reason when it cannot run). On CUDA on Linux it was on by default since 0.1.43, measured on an RTX A4000 (+5.5..8.3% decode) and a Tesla P100 (+4.4%). On three other boxes it costs 19..26% decode, and each of those three measured it the same way: one 0.1.43 build, only the flag changed between runs. - **This report** - RTX 5090 Laptop, WSL2 Ubuntu 24.04, Core Ultra 9 275HX (8 P + 16 E), native IQ3_XXS pack, `--expert-cache auto --expert-cache-per-layer` (10,320 slots, 16.78 GiB cache), PCIe Gen5 x8 at 28.7 GB/s probe: 106.6 tok/s median with `STRATA_ADAPT_OVERLAP=0` against 84.6 at the default, 5 generations of 256 tokens, slowest of the fast run (89.1) above the fastest of the slow one (87.9). Same-day v0.1.42 control on the same config: 98.1 tok/s. - **`jstrot` in #2010** - RTX 5060 Ti 16 GB, i7-14700K, 64 GB, pool limited to 16 threads: ~52 tok/s at the default, and `=0` brings it back to the 0.1.42 level of ~64. - **`demon851113` in #2010** - RTX 5070 Ti 16 GB desktop, Ryzen 7 7700 (8c/16t, AVX-512), 93 GB DDR5, PCIe 5.0 x16, IQ3_S native pack, `--expert-cache auto` (no per-layer), `--batch 2`: the same build `7695859a` for every run, one variable per run, median of 3 generations of 300 tokens: **68.4 at the default, 86.4 with `=0` (+26%)**. Control in the same session: 0.1.40.1 = 81.1 / 82.0 at `--batch 2` and 90.2 at `--batch 1`, against 79.1 for the 0.1.43 default at `--batch 1`, so with the overlap off 0.1.43 sits about 5% above 0.1.40.1. They also swept six neighbouring switches (`STRATA_ROUTE_TAIL_SKIP`, `STRATA_MMVQ_IL`, `STRATA_PCIE_FRAC_DEFAULT`, `STRATA_PREFILL_CPU_SHARE`, `STRATA_NAN_GUARD`) and none of them moves the number. The one measurement that favours the default is `gravitomagnetic` in #2010: +1.2% instruct / +2.4% greedy on an RTX 5090 desktop (Ryzen 7950X, 32 GB, 122 GB RAM, `--expert-cache 6500` auto-filling to 8,338 slots, about 33% of experts resident, no PCIe pressure in steady state). That is an interleaved A B A B A B run of the 0.1.42 and 0.1.43 tags, both at their own defaults, so the delta carries the overlap change together with everything else 0.1.43 shipped. It is a sound measurement of "is 0.1.43 slower on this box"; it does not isolate the switch on its own. What separates the two groups is not laptop versus desktop and not hybrid versus AVX-512 CPU - `demon851113` is a desktop with full AVX-512 and still loses 26%. It is how much of the expert set stays resident. The three losses are 16 GB and 24 GB cards with a small resident share; the win is a 32 GB card with about a third of the experts resident. `=1` turns it on, `=0` turns it off. The copies, the gate and the tokens are untouched, so the answers do not depend on the switch. ## What changed - `src/program/generate.cpp`, `adapt_overlap()`: the three platform branches (HIP opt-in, Windows CUDA opt-in, Linux CUDA on) become one `=1` turns it on. The per-box numbers move into the comment next to it, so the reason for a default sits where it is read. - `src/program/generate.cpp` serve startup block: the `why` predicates are read in both states (they are side-effect free) and the off state is logged once, as `strata serve: adaptive overlap: off (STRATA_ADAPT_OVERLAP=1 turns it on)` or `... off: not on a layer split` and so on. Until now only the on state was logged, so with `STRATA_ADAPT_OVERLAP=0` in a config nothing in the log said that the overlap was off. - `docs/DETAILS.md`: the setting entry states the new default and keeps every measured number with what it was measured on (A4000 +5.5..8.3%, 6 interleaved pairs; P100 +4.4%; RTX 5070 on Windows -2.6% on Q2_0; RTX 5090 Laptop -21%). Left alone on purpose: the copy path itself (`AdaptGate`, `soft_commit_exchanges`), every predicate that turns the overlap off, and the tokens. This is a default and a log line, nothing else. ## Extra Notes - Checked by build: `cmake -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89` with nvcc 13.4.92 and MSVC 14.44. The `strata` target compiles and links with 0 errors before the patch and after it (Windows CUDA, sm_89, so the Linux branch of the default is not what this compiles). - Not tested: HIP and SYCL (no toolchain on this machine), and the decode numbers are from the rigs in #2010, not from a run of this patch. - **The cost of this patch on the boxes that gained from the default:** an A4000 and a P100 now need `STRATA_ADAPT_OVERLAP=1` for their +5.5..8.3% / +4.4%. If you would rather not take that away by default, gating on the resident share instead of on the OS is where the measurements above separate: 16 GB and 24 GB cards with a small resident share lose 19..26%, a 32 GB card with about a third of the experts resident gains. I do not have a cheap and reliable runtime signal for "resident share" that I would trust as a default, so I am asking rather than guessing, and I would rather ship the opt-in flip now and add the gate once it is measured than hold this back on it. - One new line prints on every serve startup, also where the overlap could not run anyway; there it names why.
Mehr auf der Site
Links zu Install, Modellen, Releases.