Pull requests / #2683
#2683 tests: cuda_spill_guard fails the CUDA test build when a Strata kernel spills or keeps a per-thread array in local memory
open · @Avicennasis · 0 comentarios · En GitHub
AMD / HIPNVIDIA / CUDAModels & quantsWindows
Descripción
## Title
Issue: Resolves #2682 (re-filed from #1758)
## Summary
A test-build gate that fails when a release CUDA kernel spills registers or keeps a per-thread array in local memory, per `sm_`. Spills are compiler-output facts, like #1522 and #1469: they show up in the resource report and nowhere in the source, they vary by toolkit and architecture, and they cost decode throughput silently. The compiler writes that report into every object, so the gate reads the kernel libraries that were built anyway with `cuobjdump --dump-resource-usage` — no build-flag change, no GPU at test time — and mirrors `tools/check_isa_guard.py` in shape. Its table carries the measured state of `296b59d2` on sm_86 and sm_120, so it is green today and documents what is there; from here it holds the line.
## What changed
- `tools/check_cuda_spills.py` (new, 282 lines). Per library, per entry function, per SASS arch: REG, STACK, LOCAL, SHARED from `cuobjdump --dump-resource-usage` (demangled with `cu++filt` beside it, or binutils `c++filt`). A kernel must have STACK 0 and LOCAL 0 on every `sm_` unless `ALLOWED` carries its allowance for that `sm_`: 37 entries, 53 (kernel, `sm_`) allowances, each the measured maximum with one line above it saying what the array is, or what spilled (stores / loads at the register count ptxas settled on, and the launch bounds). Keyed by the demangled name without template arguments and parameters, so every instantiation of a template kernel shares one entry. The vendored llama.cpp ggml kernels (`mul_mat_q`, `mul_mat_q_stream_k_fixup`, `quantize_mmq_*`, `quantize_q8_1`; instantiated into `strata_mmq`, `strata_mmq_w4a4`, `strata_kq`) are skipped by the one property that singles them out — no namespace at all; every Strata kernel sits in `strata::` or an anonymous namespace — and the report line counts them. `--list` prints every kernel on every `sm_` as CSV, judging nothing; that is how the table's numbers were produced. Exit 0 clean / 1 a violation naming kernel, `sm_`, STACK and LOCAL / 77 when `cuobjdump` or a library is missing, or a library yields no SASS entry function (a pass with nothing counted is not a pass).
- `CMakeLists.txt` (+22): registers `cuda_spill_guard` under `STRATA_ENABLE_CUDA AND STRATA_BUILD_TESTS AND NOT STRATA_ENABLE_HIP AND NOT STRATA_HIP_GFX906` over `$<TARGET_FILE:…>` of `strata_kernels`, `strata_core`, `strata_engine`, `strata_prefill`, `strata_qwen35`, `strata_mmq`, `strata_mmq_w4a4`, `strata_kq` (each guarded by `if(TARGET)`), passing `${CUDAToolkit_BIN_DIR}/cuobjdump`; `find_package(Python3 QUIET)` guarded — no Python, no test, with a status line saying so. `SKIP_RETURN_CODE 77`, `TIMEOUT 600`.
### Measured on `296b59d2` (CUDA 13.1, nvcc/cuobjdump V13.1.80, cmake 3.28.3 + ninja, Release, `CMAKE_CUDA_ARCHITECTURES="86;120"`, Ubuntu 24.04 container, no GPU)
| | kernels | collapsed names | (kernel, sm) entries | with a stack frame | spilling | spill-free arrays |
|---|---|---|---|---|---|---|
| gated (Strata) | 1351 | 359 | 2702 | 150 | 61 | 89 |
| skipped (vendored ggml) | 909 | 6 | 1972 | 538 | — | — |
LOCAL is 0 everywhere on this toolkit: spills and per-thread arrays both land in STACK. `cuobjdump`'s STACK/REG agree with `ptxas -v`'s stack frame / registers on every one of the 4720 joined rows (one TU rebuilt with `-Xptxas -v`: identical SASS, only the `ptxasOptions` metadata differs).
The 21 Strata kernels that actually spill (template arguments collapsed; maxima over their instantiations):
| kernel | library | sm | instantiations spilling | stack frame B | spill stores B | spill loads B | registers |
|---|---|---|---|---|---|---|---|
| `strata::kernels::native_down_amd_kernel` | kernels | sm_120 | 1 | 8 | 4 | 4 | 64 |
| `strata::kernels::native_gu_amd_kernel` | kernels | sm_120 | 3 | 16 | 12 | 12 | 80 |
| `strata::kernels::native_gu_fused_kernel` | kernels | sm_120 | 1 | 8 | 16 | 32 | 64 |
| `strata::kernels::native_gu_fused_kernel` | kernels | sm_86 | 1 | 8 | 12 | 24 | 64 |
| `strata::kernels::native_gu_lds_kernel` | kernels | sm_120 | 2 | 8 | 8 | 8 | 80 |
| `strata::kernels::native_gu_lds_kernel` | kernels | sm_86 | 1 | 8 | 8 | 4 | 48 |
| `strata::kernels::{anonymous}::block_topk_reg_kernel` | kernels | sm_120 | 1 | 112 | 196 | 284 | 64 |
| `strata::kernels::{anonymous}::block_topk_reg_kernel` | kernels | sm_86 | 1 | 88 | 88 | 276 | 64 |
| `strata::kernels::{anonymous}::gdn_step_norm_multi_kernel` | kernels | sm_120 | 1 | 8 | 8 | 8 | 128 |
| `strata::kernels::{anonymous}::gdn_step_norm_multi_kernel` | kernels | sm_86 | 1 | 8 | 4 | 4 | 64 |
| `strata::kernels::{anonymous}::gr_down_q8_fast_kernel` | kernels | sm_86 | 1 | 8 | 4 | 8 | 128 |
| `strata::kernels::{anonymous}::gr_down_q8_kernel` | kernels | sm_86 | 2 | 16 | 12 | 24 | 128 |
| `strata::kernels::{anonymous}::gr_down_staged_kernel` | kernels | sm_120 | 4 | 16 | 16 | 16 | 80 |
| `strata::kernels::{anonymous}::gr_down_staged_kernel` | kernels | sm_86 | 2 | 8 | 8 | 8 | 80 |
| `strata::kernels::{anonymous}::gr_down_v3_kernel` | kernels | sm_86 | 1 | 8 | 12 | 8 | 48 |
| `strata::kernels::{anonymous}::gr_up_q8_fast_kernel` | kernels | sm_86 | 1 | 8 | 8 | 16 | 48 |
| `strata::kernels::{anonymous}::gr_up_v3_kernel` | kernels | sm_120 | 2 | 8 | 4 | 4 | 40 |
| `strata::kernels::{anonymous}::native_down_kernel` | kernels | sm_86 | 1 | 8 | 4 | 4 | 48 |
| `strata::kernels::{anonymous}::native_down_multi_kernel` | kernels | sm_120 | 2 | 24 | 12 | 12 | 64 |
| `strata::kernels::{anonymous}::native_down_multi_kernel` | kernels | sm_86 | 4 | 64 | 16 | 16 | 64 |
| `strata::kernels::{anonymous}::native_gu_kernel` | kernels | sm_120 | 1 | 8 | 4 | 4 | 48 |
| `strata::kernels::{anonymous}::native_gu_kernel` | kernels | sm_86 | 3 | 8 | 4 | 4 | 80 |
| `strata::kernels::{anonymous}::native_gu_multi_kernel` | kernels | sm_120 | 7 | 112 | 52 | 64 | 64 |
| `strata::kernels::{anonymous}::native_gu_multi_kernel` | kernels | sm_86 | 2 | 96 | 56 | 64 | 48 |
| `strata::kernels::{anonymous}::native_mmvq_il_kernel` | kernels | sm_120 | 3 | 24 | 60 | 60 | 56 |
| `strata::kernels::{anonymous}::prompt_attn_kernel` | kernels | sm_86 | 2 | 8 | 4 | 12 | 72 |
| `strata::kernels::{anonymous}::s27_down_kernel` | kernels | sm_86 | 1 | 16 | 12 | 12 | 64 |
| `strata::kernels::{anonymous}::s27_gu_kernel` | kernels | sm_120 | 1 | 16 | 20 | 20 | 64 |
| `strata::prefill::fused::{anonymous}::native_kernel` | mmq | sm_120 | 2 | 32 | 32 | 32 | 128 |
| `strata::prefill::fused::{anonymous}::native_kernel` | mmq | sm_86 | 6 | 32 | 36 | 52 | 128 |
The other 89 framed entries are spill-free per-thread arrays: `route*`'s `values[16]`, `attn_chunk_kernel`'s `float a[3][32]` (384 B, sm_86 only — sm_120 keeps it in 142 registers), `copy_kernel`'s `Runs` parameter copied to the stack to index it, `s26`/`s27`'s `wr[]`/`s[][]`, `gr_write_*`'s `x`/`v[10]`, the prefill `route_kernel`'s `v[16]`, Q3_K's `uint32_t aux[4]`, `gdn_step_commit_kernel<true>`'s `printf` argument buffer, `gr_down_kernel`'s `gw`/`ss`, and 32 bytes each in `rope_kernel` and `k_attn_prep` whose array I did not locate in the source (their entries say so). The spills are findings, not changed here; the table is the current state, so the gate passes on `296b59d2` and any growth from here is a red test.
### The gate, run
`ctest -R cuda_spill_guard` on the branch, both arches in one build:
```
check_cuda_spills: libstrata_kernels.a: 1234 kernels on sm_120, sm_86 (2468 entries; 0 vendored ggml entries skipped), 132 use stack / local memory, 0 not allowed
check_cuda_spills: libstrata_prefill.a: 49 kernels on sm_120, sm_86 (98 entries; 0 vendored ggml entries skipped), 8 use stack / local memory, 0 not allowed
check_cuda_spills: libstrata_qwen35.a: 31 kernels on sm_120, sm_86 (62 entries; 0 vendored ggml entries skipped), 2 use stack / local memory, 0 not allowed
check_cuda_spills: libstrata_mmq.a: 33 kernels on sm_120, sm_86 (66 entries; 1306 vendored ggml entries skipped), 8 use stack / local memory, 0 not allowed
check_cuda_spills: libstrata_mmq_w4a4.a: 1 kernels on sm_120a, sm_86 (2 entries; 154 vendored ggml entries skipped), 0 use stack / local memory, 0 not allowed
check_cuda_spills: libstrata_kq.so: 0 kernels on sm_120, sm_86 (0 entries; 512 vendored ggml entries skipped), 0 use stack / local memory, 0 not allowed
1/1 Test #120: cuda_spill_guard ................. Passed 1.28 sec
```
(`strata_core` 1 kernel and `strata_engine` 2 kernels, 0 framed, 0 not allowed.)
Red, both ways, against the committed script: one allowance lowered by a byte (`attn_chunk_kernel` sm_86 384 → 383) fails with `VIOLATION void strata::kernels::<unnamed>::attn_chunk_kernel<(int)0, (bool)1, (bool)0, (bool)0>(…) sm_86: STACK 384 LOCAL 0 (over its allowance STACK 383 LOCAL 0) [qsa_decode_attn.cu.o]`, one line per instantiation, and is green again on `git checkout`; a scratch kernel with a dynamically indexed `float buf[1024]` built into `strata_kernels` fails with `VIOLATION strata::kernels::<unnamed>::spill_scratch_kernel(const float *, float *, const int *, int) sm_86: STACK 4096 LOCAL 0 (not in ALLOWED) [spill_scratch.cu.o]` (and sm_120), and is green again once removed and rebuilt. A missing library or a missing `cuobjdump` exits 77 (ctest reports the test skipped).
## Extra Notes
- Entries exist only for sm_86 and sm_120 (and `strata_mmq_w4a4`'s sm_120a, which has none): a build for sm_75 or sm_89 will report the kernels that hold a frame there as `not in ALLOWED` on its first run — the numbers for those two arches come from the first box that builds them, and go in the same way.
- Not measured: HIP and Windows paths (the test is not registered under HIP), any timing or GPU behaviour, and why ptxas settles on different register counts per arch (`gr_down_q8_kernel<7>`: 128 registers and a spill on sm_86, 206 registers and clean on sm_120 — there is no register cap in the tree that explains it).
- If you would rather have the capture live elsewhere (a CMake option next to `STRATA_BUILD_TESTS`, a CI step over the build log), say so and I will move it; the table and the numbers stay the same.
En el sitio
Enlaces a install, modelos, releases.