#6425 [Engine] q36: card-1 expert tier (--peer-device) for Qwen3.6-35B-A3B, 37.9 → 74.8 tok/s
open · @Mark-1982 · 0 comentarios · En GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quants
Descripción
## Problem The `qwen35moe` (q36) path is single-GPU. Every multi-GPU flag the other families take is refused before any state exists — `spawn`/`run` hands the whole argv to `strata::qwen35::run()`, which parses its own options: ``` strata-q36: ignoring --peer-device 1 (not an option of this engine) strata-q36: ignoring --expert-profile … (not an option of this engine) ``` So on a two-card box the second card sits at ~283 MiB while decode is bound by the CPU expert pool. This is #6156, where I said I would do the implementation if the scope was right; this PR is that implementation, scoped to the expert tier only. ## Result Same box, same model, 400 greedy tokens, `--prefill auto --mtp 2 --expert-vram 0.1 --cpu-threads 6`, 65,536 context: | | decode | GPU0 | GPU1 | |---|---|---|---| | tier off | 37.9 / 38.0 tok/s | ~11.4 GiB | 289 MiB | | `--peer-device 1 --expert-profile …` | **72.3 / 74.8 tok/s** | ~11.4 GiB | 11.36 GiB | Roughly +95%. The generated tokens are unchanged (same output md5 on a fixed prompt, both arms). ## Approach - `--peer-device N`, `--peer-slots`, `--peer-reserve`, `--expert-profile PATH`: an optional arena on device N holding resident experts, one slot table per layer, filled from the same STRP routing profile the hybrid path already uses (`--dump-expert-profile` writes it). - It is a direct loader, not `strata::core::PeerExperts`: that class goes through `ExpertSource`, which refuses every geometry against the blob layout q36 builds, so the tier keeps q36's own layout (one arena, per-layer slot tables, `kMmqTail` padding after each blob). - Its own stream, its own tables and its own launches; `route_group` gains the peer's slot table so a row whose expert lives on the second card is grouped there. The prompt path stays out of the tier (`hy_rows` is false for a chunk), so a chunk's experts never straddle the two cards. - Sizing: the tier takes `--peer-reserve` MiB (default 600) for its scratch and the rest of the card, and reports what it placed at start. ## Validation Built for sm_86 with the release CUDA line (`STRATA_ENABLE_CUDA=ON`, `STRATA_GEOMETRY=flash_next`, `STRATA_NATIVE_EXPERTS=ON`, CUDA 13.1); 0 errors, both binaries relinked. Ran tier-off and tier-on arms at 65,536, 98,304, 131,072 and 262,144 context (the last with a narrower K/V cache from the separate `--kv` work), and a 12-request soak at 131,072: 69.1–82.7 tok/s, VRAM flat, host `memory.current` flat. Not verified, so it should not be assumed: - Only Qwen3.6-35B-A3B-UD-Q4_K_XL on 2x RTX 3060. No 3-card box, no NVLink, no P2P path — the tier copies through a staging buffer when the cards cannot peer-copy. - `--batch N` (several sessions in one window) with the tier on: the row tables carry it, but every run here was one session. - No HIP or SYCL build: the q36 file is CUDA-only in the tree, with no twin for either backend. - A tier-on arm can fail in a plain float `cudaMalloc` when the placement it picks leaves too little for the attention scratch; the fit line before the allocation is honest, but the tier's own sizing is not part of that arithmetic yet. The `--kv` work is a separate branch and a separate PR on purpose (`q36-kv`); this one touches only the tier. Closes #6156.
En el sitio
Enlaces a install, modelos, releases.