Pull requests / #950

#950 prefill: opt-in pointer-list bypass for gate/up gather (STRATA_PF_PTR_MMQ)

closed · draft · @DuncanBetts · 0 评论 · 在 GitHub 查看

BenchmarksNVIDIA / CUDAModels & quants

描述

## Why

On short prompts a large share of expert-layers are resident (tiny IQ3_XXS prompt: 3910 of ~5k), and each pays a full VRAM-to-VRAM gather copy before its group GEMM. Resident gate/up matrices can be read in place through a pointer table instead: ~1.25 MB skipped per resident expert (~21 GB on a 7.8K-token prompt), at the cost of one pointer-table upload per group.

## What

`STRATA_PF_PTR_MMQ=1` (default off). Gate/up reads via a vendored pointer-list MMQ slice (llama.cpp pin 3cf03257, namespace-isolated, 4 additive hunks over upstream); down stays gathered because its 640-wide rows over-read 72 B past the matrix end, which is only safe against zeroed `MMQ_TAIL` — via pointers that tail is the next slot's gate bytes, and an Inf scale there NaNs column 2559 (found by probe, fixed by construction, documented in-code). Unsupported types, non-native packs, groups over 16, and all non-CUDA backends take the previous path bit-identically.

## Evidence (RTX 5090 32 GB, IQ3_XXS)

- Answers bit-identical both states on 42/1499/7849-token prompts; `STRATA_DBG_NAN=1` clean.
- `prefill_mmq_ptr_test` green (gu/dn x all three instances, production shapes, masked-gather branch); ctest entry, skips without sm_80+.
- Perf: 275 vs 289 ms (tiny, noise), 535 vs 524 ms (medium), 1475 vs 1473 ms long (~5.3K tok/s both) — parity, not a speedup here; the skipped traffic runs overlapped with compute on this card. Kept default-off; slow-link numbers welcome.

## Ask

Deliberately not unified: the per-position blob/slot arrays mirror the stride path selection on purpose, so the two stay diffable side by side. Happy to fuse them into a struct on request.


Slow-link measurements (older discrete NVIDIA, APUs) and a 2-GPU run for the mirrored peer paths (compile-verified only). Happy to split the stack if preferred.

站内延伸阅读

链到安装、模型与版本说明,便于 SEO/GEO,非官方 issue 正文。