Pull requests / #2720

#2720 --peer-device: the adaptive tier's copies land while the next window runs (opt-in STRATA_ADAPT_OVERLAP=1); opt-in STRATA_PEER_LAUNCH_IN_POOL=1 launches the peer inside the pool's first phase

open · @bulhenvaster · 0 Kommentare · Auf GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Beschreibung

In peer mode every decode window began with a blocking wait for the previous adapt round's copies (3.0-7.7 ms of a 27-43 ms window, in 94-99% of the windows; --adapt-every 1, 17-38 swaps a window): the overlap gate left --peer-device out.  With STRATA_ADAPT_OVERLAP=1 the peer tier now joins it: PeerExperts::adapt issues its copies in layer order and signals each layer's landing with the primary's kind of signal (a stream-ordered D2H copy of the layer number into a pinned int, or host functions where the primary uses them), its table takes the swaps at the window's start (flip_pending), and a layer waits for a tier's copies only when that tier swapped in it.  Without a peer the per-layer wait is the one it was; without the variable nothing changes.

Separately, STRATA_PEER_LAUNCH_IN_POOL=1 (opt-in): the peer's launch (~30 us of host work a layer) ran before the CPU pool started, with every worker waiting; with the switch it runs on the host thread right after the pool's first phase is published (ExpertPool::run_split_multi_native with `aside`).  The rows are the same either way; without the switch the order is the old one.  STRATA_DECODE_TIMING gains a line with the peer's launch and wait and the CPU pool's own time per expert.

```
RTX 5060 Ti (PCIe 3.0 x8) + RTX 3060 (x16), Xeon E5-2683 v4, IQ3_XXS, 7 greedy requests of 300-600 tokens, means of
two alternating rounds:
  0.1.43, before                  72.2 tok/s   copy wait 3.0-7.7 ms per window
  + overlap for the peer          74.6         0.2 ms
  + STRATA_PEER_LAUNCH_IN_POOL     75.9         0.2 ms   (this step alone is inside the run-to-run spread)
  0.1.43.1 (stream-ordered signal): 75.6 -> 77.1 (inside the spread), code 91.7 -> 96.3; the host-function signal
  on the same build 76.9 / 93.5.
Not kept, measured flat or slower on the same rig: the peer streaming missed experts over its own link (-2.4%),
pacing the copies through the window (-3%), a low-priority refill stream (0), --adapt-min-gain 4 / 8 / 15 (-1 / -5 /
-7%).
```

Not tested: HIP, SYCL, Metal, Windows, a layer split; on 0.1.44.1 (this base) it was rebased and has run in daily use on the rig above since 2026-10-14, but on/off was not measured again.  Tokens are not compared bit for bit: two runs of the unchanged engine already differ (the adaptive tier makes the output depend on timing).

**How it was checked.** Built on its own on top of `main` (296b59d2): CUDA 13.3, sm_86 + sm_120, Linux (Ubuntu), the `strata` target. Without `STRATA_ADAPT_OVERLAP=1` peer mode runs as before, and without `STRATA_PEER_LAUNCH_IN_POOL=1` the peer launches first and the rows are zeroed exactly as before. HIP, SYCL and Metal were not built: I have only the NVIDIA rig.

One change per PR; nothing else from my fork is in it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.