Pull requests / #2751
#2751 cpu pool: STRATA_POOL_FUSED=1 - a native layer's gate/up and down without the barrier between them; pool_layer_bench
open · @aslater3 · 0 comentários · No GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quants
Descrição
## CPU pool: a native layer's gate/up and down without the barrier between them (STRATA_POOL_FUSED=1, opt-in) `run_split_multi_native` runs a layer's CPU experts in two phases: every expert's gate/up rows across all threads, then the host quantizes the intermediates, then every expert's down rows. With `STRATA_POOL_FUSED=1` the pool runs a new task mode instead (mode 7): - each expert's gate/up rows are cut into 8 parts and its down rows into 4; - the thread that finishes an expert's last gate/up part quantizes that expert's intermediate and sets a flag; - that expert's down parts wait only for its own flag, not for the whole layer; - the down parts are queued one expert behind the gate/up parts (`STRATA_POOL_FUSED_LAG`, default 1), so the flag is usually already set when a thread claims one. Every row goes through the same `native_gu_rows` / `native_down_rows` call as before, only cut differently, and a row's result does not depend on the cut. The output is the same bit for bit (the hash below). Q2_0 layers keep the two phases. The existing claim protocol is unchanged. Off by default: it is measured on one CPU so far. ### Why On a Ryzen 7 3800X (Zen 2, AVX2, 2 x DDR4-3200) the two halves of an expert are limited by different things. `kq_cpu_bench` (fixed in #2750), 7 threads on cores 1-7, 1 token: | | GB/s | | --- | ---: | | IQ2_S gate/up only | 27.3 | | IQ3_XXS gate/up only | 24.7 | | IQ4_NL down rows, inside the pool | ~39 | | Gate/up and down interleaved per thread (IQ2_S / IQ3_XXS / IQ3_S pair) | 34.5 / 31.7 / 32.7 | Gate/up is compute-bound on the i-quant decode (one core: 4-4.6 GB/s, IPC 3.3-3.7 under `perf`), and the down rows run at RAM speed. With the barrier, each phase leaves the other's resource idle. Using the SMT siblings as well gave 0-8% (14 threads), so more threads are not the fix. ### Measured `pool_layer_bench` (new, in this PR) drives the real `ExpertPool` the way decode does: one layer's CPU experts per call, 6 or 7 distinct experts, a quarter of them with 2 tokens, drawn from 4 GB of blobs. Pinned like the engine (7 workers on cores 1-7, the host on core 0). It prints a hash of every output over 300 fixed layers. Ryzen 7 3800X, nothing else running: | Gate/up / down | Two phases | Fused | Output hash, both | | --- | ---: | ---: | --- | | IQ2_S / IQ4_NL | 0.420 ms a layer | 0.387 (-8%) | `1c64c600a48b3217` | | IQ3_XXS / IQ4_NL | 0.496-0.504 | 0.456 (-9%) | `fc3f11e3f60dea42` | | IQ3_S / IQ4_NL | 0.525-0.526 | 0.469 (-11%) | `2f5f9c94d2c599c5` | Lag 2 measured the same or 1-2% slower; 8/4 parts were as good as 12/6 and better than 4/2 or 16/8. In the engine (SYCL build on an Arc A770, Coder IQ1_M, 8K code prompt, 3 same-day pairs, reference tokens in all six runs) the CPU pool call went from 25.45 / 21.83 / 21.62 to 23.01 / 20.42 / 20.50 ms a round and decode from 37.57 / 39.97 / 40.23 to 38.16 / 41.07 / 41.04 tok/s (+2.1%). That A/B also turned on a separate change to the single-token IQ2_S gate/up rows (+7% on that kernel alone), which is not in this PR, so the engine numbers are not this change alone. ### Also - `generate.cpp` (CUDA/HIP and SYCL): one stats line saying how many layer calls ran fused, because their down rows are then counted in the "gate/up" column. - `pool_layer_bench` in both CMake files. Not measured: AVX-512 CPUs, Intel hybrid CPUs, CUDA/HIP engine runs. The CUDA/HIP `generate.cpp` line is the same three lines as the SYCL one; I could not compile the CUDA build here. `STRATA_POOL_FUSED_PARTS=<gu>,<down>` overrides the cut for testing. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
No site
Links install, modelos, releases.