#6425 [Engine] q36: card-1 expert tier (--peer-device) for Qwen3.6-35B-A3B, 37.9 → 74.8 tok/s

open · @Mark-1982 · 0 Kommentare · Auf GitHub

BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quants

Beschreibung

## Problem

The `qwen35moe` (q36) path is single-GPU. Every multi-GPU flag the other families take is refused before any state
exists — `spawn`/`run` hands the whole argv to `strata::qwen35::run()`, which parses its own options:

```
strata-q36: ignoring --peer-device 1 (not an option of this engine)
strata-q36: ignoring --expert-profile … (not an option of this engine)
```

So on a two-card box the second card sits at ~283 MiB while decode is bound by the CPU expert pool. This is
#6156, where I said I would do the implementation if the scope was right; this PR is that implementation, scoped to
the expert tier only.

## Result

Same box, same model, 400 greedy tokens, `--prefill auto --mtp 2 --expert-vram 0.1 --cpu-threads 6`, 65,536 context:

| | decode | GPU0 | GPU1 |
|---|---|---|---|
| tier off | 37.9 / 38.0 tok/s | ~11.4 GiB | 289 MiB |
| `--peer-device 1 --expert-profile …` | **72.3 / 74.8 tok/s** | ~11.4 GiB | 11.36 GiB |

Roughly +95%. The generated tokens are unchanged (same output md5 on a fixed prompt, both arms).

## Approach

- `--peer-device N`, `--peer-slots`, `--peer-reserve`, `--expert-profile PATH`: an optional arena on device N holding
  resident experts, one slot table per layer, filled from the same STRP routing profile the hybrid path already
  uses (`--dump-expert-profile` writes it).
- It is a direct loader, not `strata::core::PeerExperts`: that class goes through `ExpertSource`, which refuses
  every geometry against the blob layout q36 builds, so the tier keeps q36's own layout (one arena, per-layer slot
  tables, `kMmqTail` padding after each blob).
- Its own stream, its own tables and its own launches; `route_group` gains the peer's slot table so a row whose
  expert lives on the second card is grouped there. The prompt path stays out of the tier (`hy_rows` is false for a
  chunk), so a chunk's experts never straddle the two cards.
- Sizing: the tier takes `--peer-reserve` MiB (default 600) for its scratch and the rest of the card, and reports
  what it placed at start.

## Validation

Built for sm_86 with the release CUDA line (`STRATA_ENABLE_CUDA=ON`, `STRATA_GEOMETRY=flash_next`,
`STRATA_NATIVE_EXPERTS=ON`, CUDA 13.1); 0 errors, both binaries relinked. Ran tier-off and tier-on arms at 65,536,
98,304, 131,072 and 262,144 context (the last with a narrower K/V cache from the separate `--kv` work), and a
12-request soak at 131,072: 69.1–82.7 tok/s, VRAM flat, host `memory.current` flat.

Not verified, so it should not be assumed:

- Only Qwen3.6-35B-A3B-UD-Q4_K_XL on 2x RTX 3060. No 3-card box, no NVLink, no P2P path — the tier copies through a
  staging buffer when the cards cannot peer-copy.
- `--batch N` (several sessions in one window) with the tier on: the row tables carry it, but every run here was one
  session.
- No HIP or SYCL build: the q36 file is CUDA-only in the tree, with no twin for either backend.
- A tier-on arm can fail in a plain float `cudaMalloc` when the placement it picks leaves too little for the
  attention scratch; the fit line before the allocation is honest, but the tier's own sizing is not part of that
  arithmetic yet.

The `--kv` work is a separate branch and a separate PR on purpose (`q36-kv`); this one touches only the tier.


Closes #6156.

Mehr auf der Site

Links zu Install, Modellen, Releases.