#5712 [Engine] Add opt-in NUMA-aware idle CPU expert prefetch
open · draft · @brokenarrow2099 · 0 comentarios · En GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
Descripción
## Problem and result With CPU-offloaded native experts, the expert-pool workers can be idle during GPU work while the next layer's RAM-resident expert bytes remain cold. This adds optional, read-only next-layer CPU cache warming using those workers. A compact Q4 router prediction selects up to four nonresident experts from an owned activation snapshot. Predictions affect speculative reads only; routing and expert arithmetic remain unchanged. This is the focused CPU-prefetch extraction. It contains no CPU expert DAG, cross-window asynchronous refill, arena allocation/placement, GPU L2, or MTP changes. #5099 was an independent asynchronous-refill experiment and has been withdrawn. **Draft:** this feature has no prior maintainer scope agreement. Feedback on the bounded scope is welcome. HIP/Windows execution and model-level parity on this exact extracted revision remain untested. No performance claim or default change is proposed. Related #5711 discusses disk expert prediction; this path reads a stable private RAM arena and does not perform speculative disk I/O or VRAM fills. Submission-time searches of open/closed prefetch issues and PRs found no matching idle CPU-cache warming PR. ## Approach and use - Build with `-DSTRATA_CPU_PREFETCH=ON`, then set runtime `STRATA_CPU_PREFETCH=1..4`; runtime unset/0 is off. Both gates default off. The retained local setting is 4; larger values clamp to 4. - Reuse the pool's epoch queues. Foreground entry cancels new claims, joins claimed work and parks workers before reusing pool metadata. Shutdown also joins before freeing the context. - On the existing two-node NUMA row split, each node claims its own tickets and warms its gate/up/down halves. This does not add or repair NUMA arena placement. - Require x86 AVX2, native experts, BF16 routers, at least two pool workers, a private RAM arena and serialized windows/groups. Read routers from their owning GPU at startup. Exclude mapped/file sources, mutable complement storage, and GPU/peer/remote-resident experts from speculative reads. - Preserve Basalt's MIT attribution and document eligibility, cancellation, counters and limits in [docs/CPU_PREFETCH.md](docs/CPU_PREFETCH.md). The standalone SYCL and Metal projects do not wire this feature; HIP is unvalidated. ## Validation Final tested revision: `eca66e81a2a8620f2b7538bbc152f4957c3f4418`, based on `87dfca0e7c3ee24ad36cc3f69a31b859587245f8`. Environment: offline Debian 12 builder, GCC 12, CUDA 13.1, sm_86, portable AVX2 CPU configuration, cached GGML 0.24.0 checkout supplied through `STRATA_GGML_DIR`. No dependency/model downloads, production restart, or new inference benchmark was performed. Build/test configuration and commands (`GGML_CHECKOUT` names that existing offline source checkout; engine and test targets were built in separate steps as described below): ```sh cmake -S . -B build-prefetch \ -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=ON \ -DSTRATA_CPU_PREFETCH=ON -DSTRATA_NATIVE_EXPERTS=ON -DSTRATA_PORTABLE=ON \ -DSTRATA_GGML_DIR="$GGML_CHECKOUT" \ -DCMAKE_CUDA_COMPILER=/opt/cuda/bin/nvcc -DCUDAToolkit_ROOT=/opt/cuda \ -DCMAKE_CUDA_ARCHITECTURES=86-real -DSTRATA_WINDOW_MAX_T=8 \ -DSTRATA_MMQ_TAB=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON cmake --build build-prefetch --target strata strata_kq -j6 cmake --build build-prefetch --target pool_idle_test idle_prefetch_test numa_idle_test pool_tasks_test \ pool_cores_test pool_plan_test numa_pool_test pool_stress native_arithmetic_parity -j6 ctest --test-dir build-prefetch --output-on-failure -j2 \ -R '^(pool_idle_test|idle_prefetch_test|numa_idle_test|numa_idle_prediction|pool_tasks_test|pool_cores_test|pool_plan_test|numa_pool_test|pool_stress|native_arithmetic_parity.*)$' ``` **Passed:** - Full CUDA engine and K-quant library build/link; focused CPU regressions: **17/17 passed**, no failures or skips (10.78 s). - Cancellation/restart/destructor join; every public foreground entry cancels an idle context, including empty batches. Q4 predictor ordering, resident exclusions, activation snapshot independence, read-only unaligned blobs, 1/2/4 selection and clamp-to-4. Synthetic two-node queue cancellation and warm reads with/without host participation. - The complete `pool.cpp` and `generate.cpp` translation units, preprocessed with the feature build macro absent, match upstream base. Only checkout-root paths and blank lines were normalized; no token/value rewriting. This is source equivalence for the disabled build, not model-level output parity for an enabled build with runtime 0. - `git diff HEAD^ --check`; read-only code review with no Critical/Important finding, two minor test/documentation findings addressed. **Build history:** the first combined build successfully linked `strata` and `strata_kq`, then failed compiling a new test because its `cpu_avx2_ok` declaration header was missing. Added `expert_layout.hpp` to the two new tests and successfully rebuilt all selected test targets before running CTest. Engine sources and CMake are identical between that successful engine build (`064f0edf71f5288180a93b4a3816cdbea59d02f4`) and the final revision; only those two test includes changed. A redundant second full build was stopped after reconfiguration rewrote cached GGML headers; the final test build resumed without reconfiguration. Compiler warnings remain; this was not a warnings-as-errors build. **Not run / limits:** - Model execution or token/logit/semantic parity on this exact revision, with either runtime 0 or 4. Earlier local measurements used a broader private tree; they are not presented as validation or a speedup for this PR. - HIP/gfx906 build or runtime, Windows/MSVC build/runtime; their toolchains are unavailable here. Targeted help with HIP compilation and cancellation is welcome before requesting review. - Real two-socket page-placement/cache/bandwidth instrumentation. Synthetic NUMA tests do not establish physical locality. End-to-end repeated performance testing, sustained soak and fault injection are pending. AI assistance was used to extract and review the change. Builds, static checks, CPU regressions and missing hardware validation are reported separately.
En el sitio
Enlaces a install, modelos, releases.