Pull requests / #2078

#2078 prefill: refill the lent expert slots from the GGUF mapping's pieces (opt-in STRATA_REFILL_PARTS=1; gfx1151 table: 4K +2.3%, 8K +2.3%, 32K +1.1%)

open · @libratechw · 0 commentaires · Sur GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows

Description

## Title
Issue: none (the end-of-prompt gap measured on gfx1151 in #1258)

prefill: refill the lent expert slots from the GGUF mapping's pieces (opt-in STRATA_REFILL_PARTS=1; gfx1151 table: 4K +2.3%, 8K +2.3%, 32K +1.1%)

## Summary

After a prompt, the expert-cache slots it borrowed are refilled from the experts' files. With the GGUF read in place an expert is
three tensors (gate, up, down), so `blob_stable` first assembled each 4 MB blob in a staging buffer (a copy on the calling
thread) and the refill then copied that to the slot. The three pieces are contiguous in the mapping, so they can go straight to
their offsets in the slot: `ExpertSource::blob_parts` hands them out, `ExpertCache::fill_slot_queued_parts` copies them on the
same queue and `sync_queued` waits as before. The bytes in the slot are the same.

## What changed

- `ExpertSource::blob_parts` (default: none), `FileExpertSource::blob_parts` (the GGUF in place, no RAM copy / override / unbuffered
  read / closed mapping; those keep `blob_stable`), the pass-through in `RemoteExperts`.
- `ExpertCache::fill_slot_queued_parts`.
- The `--serve` refill (`refill_issue` in `generate.cpp`) uses them when `STRATA_REFILL_PARTS=1`. The other refill (the single-run path)
  is unchanged.
- `src/core/arch_defaults.cpp` sets `STRATA_REFILL_PARTS=1` on gfx1151 only; `docs/STRIX_HALO.md` lists it.

Ryzen AI Max+ 395 (gfx1151), ROCm 7.14.1, `--prefill 16384`, UD-Q4_K_XL (budget 72 GiB, `STRATA_PF_FUSED_KQ=1`) and UD-IQ4_XS, fresh
server per arm, three interleaved rounds after a discarded one, v0.1.43 against this branch (prompt tokens/s, median):

| | 4K | 8K | 32K |
|---|---:|---:|---:|
| UD-Q4_K_XL v0.1.43 | 1,098 | 1,191 | 1,231 |
| UD-Q4_K_XL this branch | 1,123 (+2.3%) | 1,219 (+2.3%) | 1,245 (+1.1%) |
| UD-IQ4_XS v0.1.43 | 1,064 | 1,188 | 1,254 |
| UD-IQ4_XS this branch | 1,103 (+3.4%) | 1,230 (+3.5%) | 1,273 (+1.2%) |

The generated text is identical in all 3 of 3 rounds for every length and model. Time inside the refill (timers in the engine,
UD-Q4_K_XL): 4K, 672 slots, 217 -> 105 ms; 32K, 2,054 slots, 589 -> 322 ms. Of the 589 ms, 343 ms was `blob_stable`.

Correctness: after the refill the slots were compared with the assembled blob (`verify_slot`, a temporary check, not in the diff): 0 of 162
sampled slots differed, with the switch on and off.

## Extra Notes

Tried and left out because they did nothing: mapping the pages ahead on a helper thread (`MADV_POPULATE_READ` at lend time) and
copying with 2-4 threads on their own streams. What remains of the refill (105 / 322 ms) is the driver's staged copy out of pageable
memory, about 26 GB/s. The page cache holds only about 27 GB of the 99 GB model on this box (`fincore`), so some lent experts are read
from the SSD in any case.

`blob_parts` returns nothing with `STRATA_IO_PREFETCH=1` (that path preads the ranges the page cache lacks), so it keeps its behaviour.

Default path: unchanged (the switch is off everywhere except gfx1151's table; `STRATA_GFX1151_DEFAULTS=0` turns that off).
Built and tested on CUDA (RTX 3060, CUDA 13.0, sm_86, Windows): `file_expert_source_test`, `qsa_topk_parity`, `prefill_fused_iq_test` pass. Not built: SYCL (the SYCL port has its own copies of `expert_source.*`, `expert_cache.*` and `generate.cpp`; none of them is touched).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sur le site

Liens install, modèles, releases.