#5540 [Engine] sycl: count helper/peer caches before running the adaptive tier (helper mode decode 32 -> 61 tok/s)
closed · @Captain-Tripps · 0 评论 · 去 GitHub 看
BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
说明
## Problem and result In the SYCL serve path, the adaptive expert tier decides whether to run from `all_experts_resident` (`sycl/src/program/generate.cpp`, v0.2.12 line 8339). That check reads only the primary card's `host_res`. With a helper card (`--expert-cache-device1 auto`), the helper holds the experts the primary does not, so every expert is on some GPU, but the check says no and the tier keeps running its rounds. Measured on two Arc Pro B70 cards in helper mode: decode goes from **32.1 to 61.3 tok/s** (median of 3 fresh starts each). Turning the tier off by hand with `--adapt-every 0` gives 55.0 tok/s on the same setup. Before, the stock engine log showed a primary hit rate of 91-96%, with the rest read over PCIe or from the helper. After, every routed expert is a hit on the primary or the helper (100.0%), and the engine prints one new line at start: `strata generate: every expert is on a GPU (primary + helper/peer caches); adaptive tier off`. ## Related discussion None for this exact check. Nearby: #731 (duplicate slots across helper and primary) and #854 (`RemoteExperts::holds`), whose live `holds()` query this change reuses. ## Approach An expert counts as on a GPU when the primary holds it, a helper cache holds it (`RemoteExperts::holds`), or the peer tier holds it (`peer.has`). When all experts are on a GPU, the adaptive tier's usage tracking is not started, which is the same as `--adapt-every 0`, and one log line says so. - **Without a helper or peer card, the behavior is unchanged:** `all_experts_on_gpus` reduces to the old `all_experts_resident`. - **Scope:** only the adaptive-tier condition changes. The PCIe auto tuning and `--expert-profile-save` keep the old check. - **One-time cost:** a loop over `n_layers * n_expert` entries at start, run only when a helper or peer is configured. ## Validation Environment: Windows 11, two Arc Pro B70 (32 GB each), driver 32.0.101.8976, oneAPI DPC++ JIT build (`-DSTRATA_SYCL_AOT= -DSTRATA_SYCL_XE2_JIT=ON`). - **Source:** tag v0.2.12 (`146df3c4`), the same as `main` when this PR was opened, plus a local Windows build patch (icx flag spellings, `pread`/`memmem` shims; similar to #3607, not part of this PR). That patch is the same for both builds. - **Server:** v0.2.12's own `sycl/serve/server_intel.py`, with config `"gpu": [0]` and both cards visible to the engine. - **Model:** Qwen3.8-Flash-Next GSQ-RCO IQ3_S (ISTA-DASLab). - **Engine arguments:** `--max-context 200000 --kv int8 --spec 2 --mtp <rt> --spec-min-p 0 --ple-io mmap --stream-experts --expert-cache auto --vram-reserve-mib 1024 --prefill 8192 --no-prefill-borrow --conversation-cache-mib 32768 --conversation-cache-slots 4 --suffix-draft 0 --lookup-chain 0 --mtp-max-t 4 --expert-cache-device1 auto`. **Passed:** - The SYCL `strata` target builds with this change (Windows, icx, JIT). - Served A/B: before each start, the Windows standby list was purged and a fresh server process started. Each start ran 4 prompts (code, prose, explanation, summary) of up to 400 tokens, temperature 0, thinking off. Decode tok/s is the server's `timings.predicted_per_second`, median per start: | build | start 1 | start 2 | start 3 | median | |---|---|---|---|---| | v0.2.12 | 33.4 | 32.1 | 31.4 | **32.1** | | v0.2.12 + `--adapt-every 0` | 53.1 | 59.5 | 55.0 | **55.0** | | v0.2.12 + this change | 61.3 | 62.3 | 59.1 | **61.3** | Load time was 33-34 s in every arm. Answers were coherent in all arms. At temperature 0 the wording varies a little from start to start within every arm, and in the same way in all three, not by build. **Failed:** - None observed. **Not run:** - CUDA, HIP and Metal builds: the file changed is SYCL only. - Linux SYCL build. - The peer tier (`--peer-device`): that branch was not exercised, so the `peer.has` part is untested. - The SYCL parity tests and ctest. - Long prompts and multiple parallel requests. - Layer-split runs: not affected, because there is no helper or peer card. ## Limits - **Small sample:** one machine, one model, 3 starts per arm. - **Unexplained gap:** the change is about 11% faster than `--adapt-every 0`. I have not found why. - **CUDA original, unconfirmed static analysis:** `src/program/generate.cpp` (around line 12730 on `main`) has the same `!all_experts_resident` condition for the adaptive tier. I did not build or test a CUDA helper setup, so it is not changed here. - **CLI path not covered:** the CLI `generate` path in the SYCL file (around line 13139) turns the tier on with no residency check, and is not changed. - **Who this reaches:** with two cards in `"gpu"`, the launcher makes a layer split by default, so this only affects users who set up a helper card. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
本站相关内容
相关页面的快捷入口。