贡献 / #1912

#1912 --pool-affinity measure: on a hybrid CPU, time the all and p-cores pool layouts at start and take the faster (opt-in)

open · @Xiaomi2008 · 0 评论 · 去 GitHub 看

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

说明

## Title
Issue: none (searched the open issues and pull requests; nothing covers this)

## Summary
`--pool-affinity measure` (opt-in; hybrid CPUs, native packs): at start the `all` and `p-cores` pool layouts are timed on the pack's own experts, and the faster one is taken.

The default layout puts a worker on every physical core, and a layer's experts are done when the slowest worker is. On a Core i7-13700K (8 P + 8 E cores) the E-cores make the verify window wait, by an amount that changes from run to run, so a fixed rule did not seem right.

What is timed is the decode's own call (`run_split_multi_native`): two experts of four tokens per layer, every layer in turn, the host spinning ~150 us between two layers as it does on the GPU's flags, three interleaved blocks per layout, about a second in all. `p-cores` is taken unless `all` is more than 10% faster. One line prints both readings and the choice.

## What changed
- src/program/generate.cpp: the `measure` value of `--pool-affinity`, its usage text, `measure_pool_affinity()` and the call before the pool is built.
- docs/DETAILS.md: one entry with the measurements and the limit.

## Extra Notes
### Measured

This branch (0.1.42 + this change), Core i7-13700K, RTX 5090, 64 GB, Windows 11, Swift 1.5 IQ3_XXS, 128K context, one 8K prompt and the same 1,500 forced tokens (`--spec-follow`), three interleaved rounds:

| | decode tok/s | CPU pool per verify window |
|---|---|---|
| default (`all`, 15 workers) | 146.6 / 151.8 / 158.0 | 6.3 - 7.6 ms |
| `--pool-affinity measure` (read `all` at 16.4 - 17.6 ms, took `p-cores`) | 200.0 / 201.5 / 204.8 | 2.1 - 2.2 ms |
| `--pool-affinity p-cores` (7 workers) | 200.3 / 203.9 / 204.9 | 2.1 ms |

### Its limit

It is one reading at start and is not repeated. Over four server starts of one day on 0.1.41 it read `all` at 14.8, 21.9 and 25.3 ms against 6.1 to 6.2 ms for `p-cores` and took `p-cores`; once it read `all` at 6.6 ms against 8.6 ms and took `all`. The same day `all` ran the pool at 2.4 ms per window in one run and 6.7 to 13.2 ms in others. Where the E-cores' speed changes while the engine runs, `--pool-affinity p-cores` is the steady choice, and the docs entry says so. If a fixed `p-cores` default on hybrid CPUs is preferred to this switch, these numbers may still be of use for that.

### What it leaves alone

- Without the value nothing new runs: the option parses as before and the function is not called. I did not run the byte-identity tests; this is from the code path.
- The layout changes no result, only where threads run.
- On a CPU that is not hybrid, or without a native pack with its experts in RAM, it says so and keeps `all`.
- `--pool-affinity auto | p-cores | all` are unchanged.

### Not tested

- Built for CUDA only: sm_120, CUDA 13.0.88, MSVC 14.50 with nvcc's `--allow-unsupported-compiler`. **HIP and SYCL were not compiled**; the change is in `generate.cpp`, which both build (the SYCL tree has its own copy, which this PR does not touch).
- One hybrid CPU. No Linux run, no other Intel hybrid part, no AMD card.
- No unit test: the function needs a loaded pack and a pool.

Docs: one entry in `docs/DETAILS.md`.

### The whole tuned config on this PC, through the server (context, not this change alone)

0.1.42 built from source, the same engine binary in both columns, the Python server, the same requests, two interleaved rounds (each: a start, three 9K prompts, a 54K and an 85K prompt). Core i7-13700K, RTX 5090, 64 GB, Windows 11, Swift 1.5 IQ3_XXS, 128K, image encoder on the GPU.

| request | setup's arguments | tuned config |
|---|---|---|
| decode, 9K prompt, first request after the start | 118 / 144 tok/s | 184 / 177 tok/s |
| decode, 9K prompt, 2nd and 3rd | 173 - 194 tok/s | 195 - 212 tok/s |
| decode, 54K prompt | 153 / 156 tok/s | 217 / 198 tok/s |
| decode, 85K prompt | 149 / 131 tok/s | 179 / 192 tok/s |
| prefill, 9K prompt, 2nd and 3rd | 4,180 - 4,760 tok/s | 6,410 - 6,520 tok/s |
| prefill, 54K and 85K prompts | 5,700 - 5,980 tok/s | 6,880 - 6,900 tok/s |
| VRAM free with everything loaded | 458 / 561 MiB | 1,039 / 1,083 MiB |

The tuned config differs from setup's arguments in more than this PR, so the table does not isolate it:
- `--pool-affinity p-cores` (what this PR's `measure` picked in three of four starts): the largest part of the decode difference. The isolated measurement is the table above (146.6 - 158.0 against 200.0 - 204.9 tok/s).
- `STRATA_GR_V3=1` and `--adapt-every 1 --adapt-swaps 80 --adapt-decay 0.97`, existing switches: on the same forced text, P-cores in both arms, 220 - 223 tok/s without and 236 - 239 with both (two interleaved rounds).
- `STRATA_PF_FUSED=1` and `--prefill auto:16384`, existing switches: the prefill rows. On 0.1.41, a 30K prompt: 5,644 - 5,676 tok/s without, 6,533 - 6,575 with both (two rounds). Quality of the fused kernel here, teacher-forced over 2,000 tokens after an 8K prompt: 17 - 19 tokens differ from the text the default prompt path generated, against 9 for the default path itself.
- `--pcie-frac 0.25`, `--vram-reserve-mib 1500` and `STRATA_VRAM_FREE=device` (#1911): the VRAM row.

Not the released 0.1.42 binary: both columns are this source build.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

本站相关内容

相关页面的快捷入口。