#6325 [Engine] RDNA4 16 GB + 6-core CPU, Gyro-S: the automatic PCIe share (0.90) is ~3 % slower than --pcie-frac 1.0; with --spec 6 --draft-conf 0.6 decode is +7.5 %

open · @ric20527 · 1 comentarios · En GitHub

BenchmarksSetup & installAMD / HIPModels & quantsWindowsLinux

Descripción

### Area

[Engine]  (speed, kernels, models, the server and API)

### What do you want to do?

This is a measured report, a follow-up to #6309. #6309 found that on this PC the decode is set by the CPU's share of
the experts, not by the GPU kernels. This report looks at the settings that move that share. Two settings give a
measurable gain, and the automatic PCIe share stops short of the best value.

**Results.** Decode tok/s is the engine's own count: tokens generated ÷ ms of generation, prefill excluded. One
session is one fresh server process, 2 warm-up answers, then 6 prompts of 400 tokens at temperature 0. The 6 prompts
are Spanish technical prose, LaTeX worked solutions and code, in shuffled order. Arms alternate within each block.
Intervals are 95 % paired t intervals over sessions; windows and answers are not counted as independent. The
confirmation runs used a separate prompt set that was not used to choose the arms, and I fixed the arms before
running them.

| change against the defaults | pairs | decode change | 95 % CI |
| --- | --- | --- | --- |
| A/A (control against control, 6 sessions) | — | SD 1.1 % | — |
| `--pcie-frac 1.0` (auto settles at 0.90) | 10 | **+3.3 %** | +1.8 to +4.7 |
| `--pcie-frac 0.6` | 3 | -6.9 % | -10.5 to -3.2 |
| `--spec 6` (against `--spec 8`), exploration set | 4 | +2.5 % | +1.3 to +3.7 |
| `--spec 6`, confirmation set | 6 | +1.2 % | -0.7 to +3.1 (not confirmed) |
| `--spec 6 --draft-conf 0.6` | 10 | **+3.0 %** | +1.8 to +4.3 |
| all three (`--spec 6 --draft-conf 0.6 --pcie-frac 1.0`), test build | 4 | +5.3 % | +2.0 to +8.7 |
| all three, **official 0.1.62 engine** | 3 | **+7.5 %** (93.4 → 100.4 tok/s) | +5.9 to +9.0 |
| `STRATA_PCIE_BALANCE=1` | 4 | -1.4 % | -4.1 to +1.3 |
| `STRATA_TQ_SUB8=1` (#6309's opt-in patch) | 9 | +1.0 % | -0.2 to +2.3 (inconclusive) |

**Where the gain comes from** (`STRATA_DECODE_TIMING=1`, one session per arm, so this is indicative only):

* The window falls from 33.4 to 28.6 ms and emits 2.72 tokens instead of 3.0.
* The CPU's part of the window falls from 12.5 to 8.0 ms. It computes 1.42 missed experts per layer-window instead of
  2.17: smaller windows route fewer distinct experts, and more misses go over PCIe.
* Draft time does not change.
* The tail-skip count does not rise; it is the same or lower in the faster arms. So the gain does not come from
  skipping more experts.

**The automatic share.** The log line is `PCIe share: the CPU pool takes 119 us per missed expert (8.8 GB/s of
expert bytes), the link 55.0 GB/s -> pcie_frac 0.90`. `pcie_share_for` gives 55 / (55 + 0.8 × 8.8) = 0.887, rounded to
0.90, with the 0.95 ceiling. On this pool (6 Zen 4 cores, compute-bound on the trellis rows at ~9 GB/s), 1.0 is
faster. With 8 GiB of the 16.6 GiB RAM tier page-locked, only about 54 % of the misses can go over PCIe at 0.90.
I measured only 0.6, 0.9 (auto) and 1.0, not 0.95.

**Logits.** Same 6 prompts, 200 tokens, logprobs 5, compared on the common prefix. Two control sessions differ from
each other by a mean |Δlogprob| of 0.0094 for the chosen token (p95 0.057). `--pcie-frac 1.0` gives 0.0099 (p95 0.059)
and all three changes give 0.0101 (p95 0.051) against the control. Top-1 matches on 100 % of the common prefix in all
three comparisons. So the changes move the output no more than the control's own run-to-run variation does.

**What I did not test:**
* `--pcie-frac 0.95`, `--draft-conf` other than 0.4, 0.5 and 0.6, and `--spec` other than 5, 6 and 8.
* Other cards and CPUs, Linux, Gyro-M and Gyro-L, contexts beyond a few hundred tokens, and long prompts.
* Answer quality beyond the logprob comparison above.

**Question:** should the rule's k for weak pools (r ≤ 12 GB/s, k 0.8) be lower, or the ceiling higher, when the GPU
waits on the CPU's rows? I can run more pool sizes on this box (`--pool-workers`) if a sweep would help pick the
constant.

**Setup:**
* Official engine `0.1.62-windows-x86_64-hip-rx9070`. Test build: tag v0.2.16 (64524fb9f) plus the opt-in patches
  from #6309, with those patches off unless stated.
* Gyro-S, `--resident-experts --resident-budget-gib 17 --expert-cache auto --kv int8 --kv-resident 32768
  --max-context 131072`, DFlash2, defaults `--spec 8 --draft-conf 0.5`.
* RX 9070 XT 16 GB, Ryzen 5 7600X3D, 32 GB DDR5-6000, Windows 11 Pro.

### What do you do today instead?

`--spec 6 --draft-conf 0.6 --pcie-frac 1.0` in user args.

En el sitio

Enlaces a install, modelos, releases.