Pull requests / #1983

#1983 Linux CUDA: warm adaptive-overlap callbacks before host pinning (opt-in)

open · draft · @oal · 0 comentarios · En GitHub

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Descripción

Related to #1975.

## Change

Add a Linux/CUDA-only opt-in, `STRATA_ADAPT_CALLBACK_WARMUP=1`: submit and synchronize one no-op CUDA host callback **before constructing ExpertPool**, while the calling thread still has its original allowed CPU set. This prevents the lazily created callback worker from inheriting the decoding thread's single-core affinity.

The engine change is 12 lines. Swap selection, per-layer callbacks/waits, residency updates and GPU math are unchanged. No callback or synchronization is added when the flag is unset or adaptive overlap is disabled; no global default changes. Also includes a standalone reproducer, benchmark report and flag documentation.

## Why

On this RTX 5090 / Ryzen 9 9950X system, the decoding host is pinned to CPU 0. CUDA's callback worker created after that inherits CPU 0, competing with the host spinning for its completion. Subsequent copies on that stream cannot proceed until the callback runs.

Standalone CUDA/Linux reproduction, 200 unprofiled callback/spin handshakes:

| First callback submitted | Callback affinity | Median latency |
|---|---|---:|
| After caller pinning | CPU 0 | 984.51 us |
| Before caller pinning | CPUs 0–31 | 4.01 us |

## Inference results

Qwen3.8-Flash-Next GSQ-RCO IQ3_S, existing `--adapt-every 1 --adapt-swaps 160 --adapt-decay 0.97` calibration. NVIDIA 610.57.04, CUDA 13.3.33/sm_120, Fedora Silverblue / Ubuntu 26.04 NVIDIA distrobox.

Normal/warm-up/warm-up/normal starts, fixed `--pcie-frac 0.4`, temperature 0, 512-token caps, no competing inference workload; warm-up rounds excluded:

| Case | Normal callbacks, mean | Early warm-up, mean | Requests/setting |
|---|---:|---:|---:|
| 4K, three prompts | 142.82 TPS | 207.75 TPS | 12 |
| 100K, warmed code prompt | 232.96 TPS | 252.19 TPS | 4 |

Adaptive-gate waiting, including warm-up, fell from **4.357 ms/window to 0.000343 ms/window**. These are a new paired suite, not replacements for #1975's original 126/170 TPS figures.

Full methodology and reproducer commands: `bench/results/2026-10-11-rtx-5090-callback-affinity/README.md`.

## Validation / draft status

- Linux CUDA Release/sm_120 compile and link passed: changed C++ objects linked against existing same-version CUDA libraries. Standalone reproducer compiled and its original equivalent exercised on this machine.
- The inference A/B used an experimental build with its GPU-signal alternative **disabled in every arm**; only callback warm-up changed. This PR's minimal warm-up-only engine patch compiled separately.
- Some token arrays matched; full-suite bit-exactness is **not claimed**, since auto-sized cache capacities and existing-path outputs also varied between starts.
- Keeping this a draft pending clean minimal-PR inference validation, other GPU/driver checks and the upstream-requested HIP/SYCL build checks. The new code is guarded out on HIP/Windows; SYCL's separate program source is unchanged.
- Production remains on the existing overlap-off workaround; the installed engine and calibration were not changed.

En el sitio

Enlaces a install, modelos, releases.