Issues / #2682

#2682 [Feature]: fail the test build when a release-engine CUDA kernel spills or uses local memory (re-file of #1758, with the PR)

open · @Avicennasis · 0 Kommentare · Auf GitHub

AMD / HIPNVIDIA / CUDAModels & quantsWindows

Beschreibung

### What do you want to do?

Re-filing #1758, which the 2026-10-13 sweep closed as older than three days. The ask is unchanged; the difference is that the PR is filed right behind this issue and names it (`Resolves #…`), so both exist inside the three-day window.

A build-time check that fails `STRATA_BUILD_TESTS` when a CUDA kernel in the release engine spills registers to local memory or keeps a per-thread array there, per architecture in `CMAKE_CUDA_ARCHITECTURES`.

**Why.** Strata carries per-architecture kernel tables and has found two defects that were compiler-output facts rather than source facts: #1522 (`gdn_l2_kernel` compiles without the fma the fused kernel spells out on CUDA 13.3 / sm_120) and #1469 (Pascal decode regression). A register spill is the same kind of fact: it shows up in the compiler's resource report and nowhere in the source, it varies by toolkit and `sm_`, and it costs decode throughput silently. Nothing in the tree reads that report today.

**Shape (what the PR does).** `tools/check_cuda_spills.py` reads the built kernel libraries with `cuobjdump --dump-resource-usage` and fails when a kernel's STACK or LOCAL bytes exceed what an exceptions table in the script allows for that (kernel, `sm_`) pair; the message names the kernel, the `sm_` and the two numbers. It is registered as the CTest `cuda_spill_guard` under `STRATA_ENABLE_CUDA AND STRATA_BUILD_TESTS` (not under HIP, where there is no `cuobjdump`), skips with 77 where `cuobjdump` or Python is missing, and mirrors `tools/check_isa_guard.py` in shape. No build-flag change: it reads the objects that were built anyway, and it needs no GPU at test time. The vendored llama.cpp ggml kernels (`mul_mat_q`, `quantize_*`) are skipped — they are the only kernels with no namespace, and they are llama.cpp's to gate. This is how DeepGEMM gates its kernels (`DG_JIT_CHECK_NO_SPILLS` / `DG_JIT_CHECK_NO_LOCAL_MEMORY`).

**Measured on `296b59d2` with CUDA 13.1 (nvcc V13.1.80), Release, `sm_86` and `sm_120`:** 1351 Strata kernels (359 once template instantiations are collapsed) across the eight kernel libraries; 150 (kernel, `sm_`) entries carry a stack frame — 61 are register spills in 21 distinct kernels (the largest `block_topk_reg_kernel`, 196 B of spill stores / 284 B of loads on sm_120 at the 64-register cap of its launch bounds; then `native_mmvq_il_kernel` 60/60 B and `native_gu_multi_kernel` 56/64 B) and 89 are spill-free per-thread arrays (`attn_chunk_kernel`'s 384-byte score tile the largest). LOCAL is 0 everywhere: on CUDA 13.1 both arrays and spills land in STACK. The exceptions table carries exactly those numbers — 37 entries, 53 (kernel, `sm_`) allowances, one line each saying what the array is or what spilled — so the first run documents the current state rather than failing on it; a later change that adds a spill or grows a local array is a red test naming the kernel. The spills themselves are findings, not fixed in the PR.

**Not measured:** sm_75 and sm_89 (the table is per-`sm_`; entries for them come from the first build on such a box), HIP and Windows, any timing or GPU behaviour, and why ptxas settles on different register counts per arch (e.g. `gr_down_q8_kernel<7>`: 128 registers + spills on sm_86, 206 registers and clean on sm_120).

If you would rather have the capture elsewhere (a CMake option, a CI step over the build log), say so on the PR and I will move it.

Mehr auf der Site

Links zu Install, Modellen, Releases.