#490 Heterogeneous multi-GPU roles: whole model on one GPU, the MTP draft head on the other (+ test harness)

Beschreibung

Hello Niko — while working with Strata's MTP speculative decoding on a two-GPU box, I noticed the GPUs don't have to be (and rarely are) equal, and today's `--split-device` uses them as one pipeline, which costs capacity on both and doesn't separate the model from the drafter. So I explored a different placement: **the whole model (KV + verify) stays on the main GPU, only the MTP draft head runs on the second one** — no P2P required, no layer split. The point of doing this with two unequal GPUs: the drafter is tiny relative to the 125B main, so moving it (≈870 MiB + the draft head) off the main GPU **gives the main GPU's expert cache that VRAM back**, while the drafter gets a whole GPU's free VRAM to grow its own buffers; and the two directions of "which card is the main" let the user match whatever hardware they actually own.

**What I built on top of 0.1.34 (`sm61-hetero-0134`, ~10 commits, all opt-in flags, default behavior unchanged):**

- `--main-device N --draft-device M` — explicit role placement (both directions work; the drafter's K/V, ring restore, workspace and graph captures all get re-homed on the draft device);
- `--auto-roles` — pick the roles from the capability records (main = device 0, draft = the best free remaining);
- `--draft-prefill-parallel` — the prompt fill overlaps the decode instead of blocking it (works with the roles);
- `--draft-chain-batch` — the draft chain's step graphs launch back to back with **one** wait per round instead of one per step (drafting went 4.63 → 4.14 ms/round same-device, ~11 %);
- a per-role VRAM report after binding, so the trade is visible at start.

**The guarantee I wanted before claiming anything: placement never changes the output, only the measured time.** Every claim below is on greedy token streams compared bit-exactly, across repeated runs, and — this is the part I'd like to stress — the honest subtlety is that **expert residency is not placement**: when the roles split, the drafter's VRAM frees up on the main GPU, the auto expert cache there grows, and a different expert-cache size legitimately decodes differently today. So all bit-exactness comparisons are done **under equal residency** (either the same auto slot count, or a forced `--expert-cache N` shared by both runs). Under that rule, roles 0→1, 1→0, reversed, batched and pipelined all reproduce the same-device stream token for token (e.g. the roles run with a forced cache is bit-identical to the no-roles baseline).

**Where I think more speed is still on the table** (reasons, not results — this is the part I'd most want measured by other people):

- **prefill using both GPUs**: right now the prefill is expert-mmap I/O-bound on my pack (78–98 % of the window is page-cache reads of experts), pipelining the prompt fill into the decode didn't move the needle *here* — but with a cached/pinned expert profile, or a smaller model, an actual two-GPU prefill split may pay off;
- **draft speed**: the chain batching above recovered a measurable slice of the drafter's own latency (4.63→4.14 ms/round on a 1080 Ti; the drafter is ~0.5 % of a ~939 ms decode window on this rig, so it's measured, not *felt* — a faster second GPU should make the batch's per-step wait elimination worth proportionally more).

**How to test it on your own rig (the piece I'd most like to hand over):** I built a portable harness, `tools/hetero-test.sh` — you give it your build dir and your own model-load flags, it runs the variant matrix (baseline / roles both ways / prefill-parallel / chain-batch / both / auto-roles; a single-GPU batch variant falls back automatically), extracts the greedy stream from each log, picks bit-exactness references **by residency** (the rule above, encoded), and writes an `OUT/report.md` table (accepted/of-totals, tok/round, ms/round drafting, expert slots, stream md5) plus the verdicts — ready to paste back. On my rig (2× GTX 1080 Ti, 125B Q2_0 pack, 16 runs) it reported **20 PASS / 0 FAIL**, and the report's md5 clusters reproduce exactly what I measured by hand. **A report from any non-Pascal rig — one GPU or two unequals — would be exactly the evidence both of us need** before I open a clean hetero-only PR (I kept my sm_61 work separate, so the PR branch is easy to cut).

I have not opened the PR yet because I'd rather this land as: issue first, your read on the idea, then a small PR (flags + the drafter device handling + the harness), not a dump.