Pull requests / #2623
#2623 PLE: reuse prefetched row vectors across reordered and duplicate gathers
open · draft · @hotschmoe · 0 comentarios · En GitHub
AMD / HIPNVIDIA / CUDAModels & quantsDocumentation
Descripción
DRAFT - NOT READY FOR REVIEW. CUDA/HIP backend builds have not been performed; no reviewers are requested. Bug report: https://github.com/Niko1221/Strata/issues/2622 Prefetching six PLE token row-vectors in one order and gathering those same vectors in another order currently falls through to a second reader ticket. `prefetch_rows` also deduplicates identical vectors, so `n_prefetch >= n_tokens` excludes a gather containing repeated requests even when every requested vector is available. This changes only `PleTable::gather_batch` in `src/kernels/ngram.cpp`: - Match each requested 16-row key against any of the at most 16 prefetch slots, preserving requested output order. Duplicate requests can reuse an immutable slot. - Collect every outstanding prefetch ticket before clearing the slots, including after a collection failure; retain the first error and return before decoding on failure. The existing 16-token limit, missing-key/oversize fallback, mapped path, public API and logical byte accounting stay unchanged. The lookup performs at most 256 fixed-size key comparisons. This does not wire batch-session PLE metadata or change kernels. Reproduction on upstream `296b59d2f73c8f04a805df29c2fa97655adace78`: open a finite tiny IQ4_NL GGUF in Direct mode; prefetch six distinct row-vectors A,B,C,D,E,F; record the requested-row count from `io_report`; call `gather_batch` with D,E,F,A,B,C. The baseline requests another 96 rows; the patch requests zero. With only A prefetched, gathering A,A,A changes 48 additional requested rows to zero. Results remain byte-identical to an independently opened Mmap reference. Validation already executed: a CPU-only harness compiles the actual shared PleTable and the SYCL tree's host PleReader/DirectFile, using g++ 13.3.0, C++17/O2/pthreads. Eleven cases at cache sizes 0/256 and worker thread off/on, both baseline and candidate: **88 cases pass**. This includes ordered/permuted/subset/duplicate/extra/missing keys, duplicate-prefetch suppression, zero/over-limit gathers, prefetch overflow and the pending-token guard. Output guards, finite references, logical bytes and requested-row counts pass. Error injection and out-of-range rows were not tested; drain-all failure handling was reviewed, not fault-injected. Requested-row counts include row-cache hits and do not measure physical disk traffic or latency. [Evidence and exact identities](https://github.com/hotschmoe/b70_ai_things/blob/fff2c71b0a1d806db428b8489b42fd248d50bebd/docs/20261015_strata_ple_prefetch_readiness.md), [executed fixture](https://github.com/hotschmoe/b70_ai_things/blob/fff2c71b0a1d806db428b8489b42fd248d50bebd/strata/flash-next/lowlevel_probe/ple_prefetch_cpu.cpp). The fixture retains its original local include path; reproduction on another checkout needs that path adjusted. The submitted production patch is byte-for-byte the CPU-tested patch (SHA256 `217550f5994f9159c9a443a81bc71429f481fd823493b44f189f11fc241199b5`). No end-to-end throughput gain is claimed. A later local two-B70 C2 serving comparison included this same helper in both arms, so it cannot isolate its benefit. Full CUDA/HIP backend builds remain unperformed for this contribution; this draft must not be presented as fully validated across backends. Related source overlap: #1516 changes the large-prompt reader path and needs a context rebase if merged first; its small-gather fast path still matches positionally. #2073 changes PLE diagnostics; #2326 adds table serialization. Neither implements this key matching. Existing #2511 covers unrelated draft/restore accounting.
En el sitio
Enlaces a install, modelos, releases.