Pull requests / #2751

#2751 cpu pool: STRATA_POOL_FUSED=1 - a native layer's gate/up and down without the barrier between them; pool_layer_bench

open · @aslater3 · 0 Kommentare · Auf GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quants

Beschreibung

## CPU pool: a native layer's gate/up and down without the barrier between them (STRATA_POOL_FUSED=1, opt-in)

`run_split_multi_native` runs a layer's CPU experts in two phases: every expert's gate/up rows across all threads,
then the host quantizes the intermediates, then every expert's down rows. With `STRATA_POOL_FUSED=1` the pool runs a
new task mode instead (mode 7):

- each expert's gate/up rows are cut into 8 parts and its down rows into 4;
- the thread that finishes an expert's last gate/up part quantizes that expert's intermediate and sets a flag;
- that expert's down parts wait only for its own flag, not for the whole layer;
- the down parts are queued one expert behind the gate/up parts (`STRATA_POOL_FUSED_LAG`, default 1), so the flag is
  usually already set when a thread claims one.

Every row goes through the same `native_gu_rows` / `native_down_rows` call as before, only cut differently, and a row's
result does not depend on the cut. The output is the same bit for bit (the hash below). Q2_0 layers keep the two phases.
The existing claim protocol is unchanged. Off by default: it is measured on one CPU so far.

### Why

On a Ryzen 7 3800X (Zen 2, AVX2, 2 x DDR4-3200) the two halves of an expert are limited by different things.
`kq_cpu_bench` (fixed in #2750), 7 threads on cores 1-7, 1 token:

| | GB/s |
| --- | ---: |
| IQ2_S gate/up only | 27.3 |
| IQ3_XXS gate/up only | 24.7 |
| IQ4_NL down rows, inside the pool | ~39 |
| Gate/up and down interleaved per thread (IQ2_S / IQ3_XXS / IQ3_S pair) | 34.5 / 31.7 / 32.7 |

Gate/up is compute-bound on the i-quant decode (one core: 4-4.6 GB/s, IPC 3.3-3.7 under `perf`), and the down rows run
at RAM speed. With the barrier, each phase leaves the other's resource idle. Using the SMT siblings as well gave
0-8% (14 threads), so more threads are not the fix.

### Measured

`pool_layer_bench` (new, in this PR) drives the real `ExpertPool` the way decode does: one layer's CPU experts per call,
6 or 7 distinct experts, a quarter of them with 2 tokens, drawn from 4 GB of blobs. Pinned like the engine (7 workers
on cores 1-7, the host on core 0). It prints a hash of every output over 300 fixed layers. Ryzen 7 3800X, nothing else
running:

| Gate/up / down | Two phases | Fused | Output hash, both |
| --- | ---: | ---: | --- |
| IQ2_S / IQ4_NL | 0.420 ms a layer | 0.387 (-8%) | `1c64c600a48b3217` |
| IQ3_XXS / IQ4_NL | 0.496-0.504 | 0.456 (-9%) | `fc3f11e3f60dea42` |
| IQ3_S / IQ4_NL | 0.525-0.526 | 0.469 (-11%) | `2f5f9c94d2c599c5` |

Lag 2 measured the same or 1-2% slower; 8/4 parts were as good as 12/6 and better than 4/2 or 16/8.

In the engine (SYCL build on an Arc A770, Coder IQ1_M, 8K code prompt, 3 same-day pairs, reference tokens in all six
runs) the CPU pool call went from 25.45 / 21.83 / 21.62 to 23.01 / 20.42 / 20.50 ms a round and decode from 37.57 /
39.97 / 40.23 to 38.16 / 41.07 / 41.04 tok/s (+2.1%). That A/B also turned on a separate change to the single-token
IQ2_S gate/up rows (+7% on that kernel alone), which is not in this PR, so the engine numbers are not this change alone.

### Also

- `generate.cpp` (CUDA/HIP and SYCL): one stats line saying how many layer calls ran fused, because their down rows
  are then counted in the "gate/up" column.
- `pool_layer_bench` in both CMake files.

Not measured: AVX-512 CPUs, Intel hybrid CPUs, CUDA/HIP engine runs. The CUDA/HIP `generate.cpp` line is the same
three lines as the SYCL one; I could not compile the CUDA build here. `STRATA_POOL_FUSED_PARTS=<gu>,<down>` overrides
the cut for testing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.