Pull requests / #2721

#2721 prefill: STRATA_PF_PEER_CPU=1 (opt-in) runs the CPU share beside a peer GPU (--peer-device)

open · @bulhenvaster · 0 Kommentare · Auf GitHub

AMD / HIPNVIDIA / CUDAModels & quantsLinux

Beschreibung

The share took nothing there (cpu_maybe required !m.pp).  With the switch it takes only experts that neither card computes (not peer-held, not peer-streamed), and its rows go before the peer's block, so the peer's block stays the tail the peer combine reads from back_at (layout: primary GPU, CPU, peer).  The staged chunks then go up to cpu_share_max()'s 3072 tokens instead of the default share's 1024 (an explicit STRATA_PREFILL_CPU_SHARE_MAX still wins).  Without the switch nothing changes.

RTX 5060 Ti (PCIe 3.0 x8) + RTX 3060 (x16), Xeon E5-2683 v4, IQ3_XXS, a 161-173K-token agent conversation, tool results read one after another (STRATA_PREFILL_TIMING): 117-132 tokens 1.80-1.92 -> 1.16 s, 325-335 2.12-2.89 -> 1.36-1.63 s, 524 3.33 -> 1.99 s, 1067 3.74 -> 2.64 s, 1170 3.81 -> 3.26 s, 2012 4.33 -> 4.22 s, 3.2-3.5K unchanged; the chain of 12 reads 36.0 -> 29.7 s (the base twice: 36.1, 36.0).  A question about the last tool output answered the same in all three runs.  In daily use on that rig since 2026-10-15.

Not tested: HIP, SYCL, a layer split's helper (the change is in the CUDA prompt path's host code; the share stays off on HIP unless set), other peer pairs.

**How it was checked.** Built on its own on top of `main` (296b59d2): CUDA 13.3, sm_86 + sm_120, Linux (Ubuntu), the `strata` target. HIP, SYCL and Metal were not built: I have only the NVIDIA rig.

One change per PR; nothing else from my fork is in it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.