#6550 [Engine] STRATA_L3_PF=1: +6% decode by reading the next layer's likely CPU experts into the CPU caches while the GPU works (opt-in)

open · draft · @gcp · 0 コメント · GitHub で見る

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

本文

## Problem and result

**+6% decode speed** on current main with CPU-offloaded experts, output unchanged: +6.0% on a 34-turn agent session with 142K-174K token prompts and +5.8% on chat prompts (RTX 3090 + Ryzen 9 5950X, 3 interleaved rounds). Opt-in, one environment variable: `STRATA_L3_PF=1`. On #5712's base it is also +5.0% faster than #5712 (see Related discussion).

When a decode window misses an expert in VRAM, the CPU pool computes it from RAM. On an RTX 3090 with a Ryzen 9 5950X and DDR4 that costs about 74 us per expert per layer (DRAM-bound, about 47 GB/s), against about 26 us when the expert's rows are already in the CPU caches (`kq_window_bench`). Meanwhile the CPU waits about 200 us per layer for the GPU.

`STRATA_L3_PF=1` (opt-in) uses that wait: after each layer's CPU dispatch the pool workers compute the next layer's router on the layer's input, all pick the same 4 most likely experts that are not in VRAM, and each worker reads its own row slice of those experts. In the next layer, each worker computes exactly the rows it read. The next batch of CPU work stops the reading.

Decode speed, RTX 3090 at PCIe 4.0 x8 + Ryzen 9 5950X, Qwen3.8-Flash-Next UD-Q4_K_XL, 262K context, 3 interleaved rounds, a fresh engine per run:

| Workload | 0.1.64 (main `6e38f588a`) | + `STRATA_L3_PF=1` | Gain |
| --- | --- | --- | --- |
| 34-turn agent session (142K-174K token prompts as token ids), total tok/s | 63.03 / 62.10 / 60.56 | 65.63 / 66.81 / 64.31 | **+6.0%** (Welch t 3.6) |
| 8 chat prompts x 600 tokens, tok/s | 66.76 / 67.98 / 67.37 | 70.54 / 71.58 / 71.79 | **+5.8%** (t 7.5) |

The output does not change (see Validation). The full report with every run's data is in `docs/bench-results/2026-11-22-community-rtx3090-5950x-l3-prefetch/`.

## Related discussion

Scope issue: #6553 (no maintainer agreement yet; feedback on the scope is welcome). Draft #5712 (`STRATA_CPU_PREFETCH`) targets the same idle time with a similar idea. I built it on its base (`87dfca0e7`, 0.1.59.1) next to these same commits rebased there, and measured both on the same machine, same workloads, interleaved:

| Workload | 0.1.59.1 | + #5712 (`STRATA_CPU_PREFETCH=4`) | + this change | This vs #5712 |
| --- | --- | --- | --- | --- |
| Agent session, total tok/s | 60.58 | 62.26 (+2.8%) | 65.34 (+7.9%) | **+5.0%** (t 11.9) |
| Chat prompts, tok/s | 66.70 | 68.50 (+2.7%) | 70.90 (+6.3%) | **+3.5%** (t 4.9) |

(means of 3 rounds; per-round numbers in the report.) This change won every round. The main design difference: #5712 warms each worker's share of the bytes while the compute phases still hand out row ranges dynamically, so a worker often computes rows another worker read (on a two-CCD Ryzen, possibly into the other L3). Here each worker gets fixed row slices for both reading and computing. #5712 also predicts from a Q4 router copy and needs a build flag; this uses the BF16 router and is a runtime switch only. #5712 no longer applies to main (the `engine/` move); this branch is on current main. I posted these numbers in #5712 as well.

Prior art: Basalt's `cpu_warm` (commit `ca7d074`, the origin of #5712, enabled by default there) warms the single likeliest expert in contiguous per-worker byte chunks while its pool claims rows dynamically.

## Approach

- `ExpertPool::set_static_slices(true)` gives worker j the rows `[per*j/n, per*(j+1)/n)` of every expert in the native gate/up and down phases and the host none. The kernels are unchanged; only where a row range ends moves, and the kernels give the same bits for any split. With static slices a native layer call runs as the two phases, not `STRATA_POOL_DEP`'s one batch (whose claimed tasks do not follow the slices); the gain above is net of that.
- `ExpertPool::prefetch_next` publishes an asynchronous batch (mode 8) and returns: each worker computes its share of the next layer's router logits (`bf16_rows_dot_multi`), the workers meet at a barrier, each picks the same top N non-resident experts by rank votes over the tokens' top k, and reads its slices (gate, up and down rows) one byte per cache line. `wait_parked` stops it before the next batch of any kind.
- `generate.cpp`: with `STRATA_L3_PF=1` and an eligible setup (one GPU, a native pack with the experts in RAM, AVX2, no NUMA split), the BF16 routers are copied to RAM at startup (120 MiB for Flash-Next) and the prefetch is published after each layer's CPU dispatch. `STRATA_L3_PF_TOP` (1-16, default 4) sets N; `STRATA_POOL_STATIC=1` turns on the static slices alone. Otherwise it logs why it is off and nothing changes.
- `kq_window_bench`: `--pf`, `--static`, `--min-experts`, `--check-split`, pinned threads, used for the cost numbers above.
- Docs: one row in the opt-in switches and one in the speed tables of `docs/DETAILS.md`, plus the report.

## Validation

**Passed:**
- Default path: a greedy one-shot run (`--tokens-file`, `--max-new 200 --adapt-every 100000 --pcie-frac 0 --window-hashes`) 3 times on `6e38f588a`, twice on this branch with the switch off and once with it on: the same 200 tokens every time. The draft windows' sizes vary between runs on every build (main against itself too); every window with the same position and drafts had the same hash in every pair of runs.
- `pool_tasks_test` (468 bitwise comparisons, including the static slices with 2, 3 and 5 workers, with and without the host, alone and with a prefetch stopped by the batch), `pool_determinism` (218,736 checks, 0 failures), `pool_test` (its AVX-512 VNNI case skipped on this CPU), `pool_cores_test`, `pool_plan_test`, `router_dot_parity`: all pass on this branch. Linux, GCC 13.3, CUDA 13.1, sm_86, Ninja Release.
- Speed: the tables above.

**Failed:** none observed.

**Not run:**
- HIP and SYCL builds: no ROCm or oneAPI here. `pool.cpp` and `pool.hpp` are shared; the new code has no intrinsics of its own and calls `bf16_rows_dot_multi` (declared in `kq_avx2.hpp`, which `expert_source.cpp` and `native_expert.cpp` already include). Left to CI; that is why this is a draft.
- Windows, Metal, NUMA or two-socket machines (the switch stays off with the NUMA split), other GPUs, CPUs, models or packs, the file-backed expert source (all runs used the page-locked RAM arena).

## Limits

- Measured on one machine. The gain depends on the CPU computing missed experts while the GPU works; a machine with a larger hit rate or faster RAM gains less.
- Top 2, 3, 6 and 8 measured within the noise of top 4 here; 4 is the default.
- Off with more than one GPU, with the NUMA split, without AVX2, or without BF16 routers.

関連リンク

インストール・モデル・リリースへのサイト内リンク。