Pull requests / #2476

#2476 serve: skip file lookahead when experts remain resident

open · @KeyanHu-git · 0 commentaires · Sur GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Description

## Title

serve: skip file lookahead when experts remain resident

Issue: Resolves #1922

## Summary

Once every expert is covered by GPU or RAM, file lookahead still computes routes even though there is nothing left for it to fetch. This takes CPU time from the same machine running the expert workers.

This PR stops submitting those predictions during serial decode when complete expert coverage is established. On a Ryzen 5 7600X / RX 7900 XTX with IQ1_M, 11 matched English pairs increased decode throughput by **1.67%** and reduced request CPU time by **6.10%**. The [full report](https://github.com/KeyanHu-git/Strata/blob/e4821fe7edb330668c139460b0deb1fe2d50a7f1/bench/results/2026-10-15-rx7900xtx-resident-lookahead/README.md) is submitted separately in #2520, following the repository's benchmark-report guidance.

## What changed

The coverage check runs immediately before `READY`, after prompt-buffer reservations, KV planning and prediction consumers have been set up. Some startup reservations remove GPU-only experts after the RAM complement is built, so the check uses the final residency table.

Adaptive exchanges preserve coverage by saving outgoing experts before replacing their GPU slots. Serial prompts return borrowed slots before decode. This lets the policy make one decision at startup. Batch serving, elastic KV, Foresight, lookahead statistics and explicit `STRATA_LOOKAHEAD` settings keep their existing behavior.

The change covers CUDA/HIP and SYCL and adds a CPU test for eligibility, exchanges, duplicate copies, prompt loans and permanent slot removal. The predictor's startup allocations and idle thread remain.

## Extra Notes

The new benchmark uses the same executable for forced ON and automatic mode: 12 balanced, randomly ordered A/B pairs, plus an ON/ON control at each end. Each arm starts fresh, warms up for 600 tokens, then generates 600 tokens each of English exposition and code. Diagnostic timing and statistics switches are off.

| Workload | Matching pairs | Decode throughput change | Request CPU time reduction |
|---|---:|---:|---:|
| English, primary | 11/12 | +1.67% [0.80%, 2.58%] | 6.10% [5.26%, 7.06%] |
| Code, exploratory | 4/12 | +2.70% | 6.89% |

Changes are geometric means of paired ratios; brackets are approximate 95% paired bootstrap intervals. English improved from **40.40 to 41.08 tokens/s**. Nine of the 11 matched pairs were faster, and all 11 used less CPU time. Both run-order groups were positive. Output IDs, MTP work, preceding requests and expert-tier counts matched. Every measured request had zero decode expert-file reads.

The run retained 131072 context capacity, int8 KV, MTP, five pool workers, 5759 GPU expert slots and a 15481 MiB RAM complement. It used GGUF-backed experts because the original Windows flat pack does not start this predictor. CPU time covers the whole request, including prefill. The report contains all observations, exclusion reasons, A/A controls, scripts and the earlier five-pair check as a separate dataset.

Full CUDA, HIP and SYCL builds pass. The CPU test passes on macOS with ASan/UBSan, Windows AMD Clang and Linux CTest, including 20,000 modeled exchanges. Model execution was tested on HIP with short prompts; the report records the tested configuration and coverage.

![English matched-pair effects](https://raw.githubusercontent.com/KeyanHu-git/Strata/e4821fe7edb330668c139460b0deb1fe2d50a7f1/bench/results/2026-10-15-rx7900xtx-resident-lookahead/figures/english.png)

Sur le site

Liens install, modèles, releases.