Pull requests / #1323

#1323 prefill: batched unbuffered stager reads; close the experts.bin view once reads are unbuffered (port of #833)

open · @malloc32 · 0 comentários · No GitHub

BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityWindowsLinux

Descrição

**AI-written.** This PR was prepared by an AI (Claude Opus 5.5, via Claude Code) at the submitter's request; the code, the port and the measurements were done by the AI on the submitter's machine, and the submitter reviewed it before opening.

**Update 2026-10-07:** rebased onto 0.1.40.2 (`e8ca9af`). The one conflict was the Stager's wait in `src/prefill/prefill.cpp`: the release's sleeping wait (`stager_sleep()`, Linux by default) now applies to each job of a claimed run, beside the batched reads.

## What

A port of @adonizm's #833 onto the current `main` (it was closed by the history rewrite, not rejected). Two changes:

1. The prompt path's stager claims runs of consecutive jobs and reads them with one `read_direct` batch (`ExpertSource::copy_blobs`), merging neighbouring blobs into requests of up to 32 MiB. `STRATA_STAGER_BATCH` (default 8 for file-backed blobs, 1 otherwise).
2. Once the file tier reads unbuffered, the mapped view of `experts.bin` is closed (`FileExpertSource::drop_mapping`, Windows): while it is mapped, NTFS serves the unbuffered reads one at a time. `STRATA_KEEP_MAPPING=1` keeps it.

Two fixes on the way: `unmapped_` is declared on every platform (`mapped_blob` reads it outside the Windows block, so Linux did not compile), and `copy_blobs` asks `resident_blob`, which follows the exchange rotation (`STRATA_EXCHANGE_ROTATE`).

## Measured

i7-12700K, 64 GB, 2x RTX 5060 Ti (layer split), IQ3_S, `experts.bin` on a Samsung 990 PRO, `--resident-budget-gib 8`, unbuffered reads: prompt **513 -> 623 tok/s at 8K, 962 -> 1,198 at 32K**. Batching gives most of it; closing the view adds 4-5%. The original author's numbers: 1,261 -> 1,669 tok/s at 47.6K.

## Not tested

Linux and HIP builds (the Linux fix was checked with a syntax-only compile).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

No site

Links install, modelos, releases.