#6501 [Engine] Give each layer-split stage its own two-pass check buffers
open · @woodsonl · 0 comments · View on GitHub
Multi-GPUAMD / HIPNVIDIA / CUDA
Description
## Problem and result `STRATA_SELECT_TWO_PASS_CHECK=1` compares the two-pass selection with today's 3xTF32 scores and ids. When `m.attn` has no room for them past the two-pass scratch, the check keeps them in buffers of its own, and those buffers are function-static in `Prefill::run_impl` (`engine/src/prefill/prefill.cpp`, lines 5328 to 5330 at 3142b66). On a layer split every stage then uses the pair the first stage allocated on its card, so a later stage's scorer writes to another device's memory. With this change each stage allocates its own pair on its own device. The default path is unchanged, and so is the check on one GPU. Reproduction status on Strata: unconfirmed static analysis. The same code, ported to Basalt, crashed a two-card layer-split run with the check on, and the tester fixed it there with per-stage buffers: https://github.com/jesdga95/basalt/pull/445#issuecomment-7054978140 ## Related discussion None. The change touches only a diagnostic and no default. ## Approach The pair moves into `Prefill::Impl` beside `sel_scores` and is allocated on first use, under the `OnDevice` guard that `run_impl` already holds for the stage. Both pointers go into `m.owned`, which `release()` frees. A larger need allocates a new pair, as the static pair grew before, and the smaller pair is freed at release, so the check no longer synchronizes and frees in the middle of a prompt. The `m.attn` path and the rest of the check are unchanged. The check's counters (`g_tp_stats`) are still one per device, so the comment keeps "one GPU". ## Validation **Passed:** - Target `strata` at 2605372 builds with the `cuda-compile` preset's cache variables (Release, `CMAKE_CUDA_ARCHITECTURES=86-real;89-real;89-virtual`, native and Q6_K experts on), generated for Makefiles. `prefill.cpp` warns twice (`-Wunused-variable` at lines 8277 and 8356), both in code this change does not touch. **Failed:** - The same build with `-DSTRATA_WERROR=ON` stops on warnings in `engine/src/core/layer.cpp` (misleading indentation, unused variables) and `engine/src/core/verify.cpp` (a format mismatch), files this change does not touch. **Not run:** - Any run with `STRATA_SELECT_TWO_PASS_CHECK=1`, on one GPU or on a layer split: no pack or free GPU was available. The fallback path on a split stays unverified. - The HIP build (`hip-compile`): no ROCm toolchain was available. - SYCL: its `prefill.cpp` has no such check. Metal: the block sits under `#if !defined(STRATA_GPU_METAL)`. ## Limits Only the fallback path changes. On a layer split the check's counters stay per device, as before.
Related on this site
Quick links to related pages.