#4408 [Engine] fix KVA crash with STRATA_GDN_CHUNKED on HIP: the state-only GDN recurrence no longer takes the chunked WMMA scan
closed · @bsvinay · 0 Kommentare · Auf GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quants
Beschreibung
Problem and result: With STRATA_KVA=1 and STRATA_GDN_CHUNKED=1 on gfx11, the first prompt chunk KVA
approximates fails with "prefill: unspecified launch failure" and the engine exits. After the change,
KVA runs with STRATA_GDN_CHUNKED=1; prompts without KVA take exactly the same path as before.
Approach: gdn_recurrence_state_only() passes z, gamma and y16 as null (with_out=false). On HIP,
gdn_recurrence_impl() still sent it to gdn_rec_wmma(), whose scan kernel fuses the output normalization
(reads z/gamma, writes y16). ld16 is defaulted to HV*S before that call, so gdn_rec_wmma's own ld16 check
does not turn it away. The chunked scan is now taken only when with_out is true; a state-only call falls
through to the quad kernels, which already honour with_out. The change is inside the __HIPCC__ block only.
Validation:
Passed:
- HIP build (gfx1100, ROCm 10.1), target strata, v0.2.7 (5e8a4cc) + this commit.
- Hardware, 1x RX 7900 XTX, Swift 1.5 IQ3_XXS, --prefill auto:16384, resident mode. A synthetic
long-document test (24 entries spread through ~13K/25K/47K-token prompts, greedy, the model lists them),
2 rounds, KVA off vs on:
before the fix: STRATA_KVA=1 alone works; STRATA_KVA=1 + STRATA_GDN_CHUNKED=1 crashes on the first
approximated chunk (2 of 2 runs); each of PREFILL_RING, HIP_WMMA, PF_GEMM, SCORES_THREAD, SELECT_SIMT
with KVA works.
after the fix, with GDN_CHUNKED and the other RDNA3 opt-in switches set: no crash;
prompt 25K 2208/2232 -> 2611/2692 tok/s, 47K 2594/2637 -> 3118/3121 tok/s; answer decode
25K 129.0/128.4 -> 126.0/125.1, 47K 128.0/127.4 -> 123.9/123.8; 24/24 entries found, 0 invented,
in every run; ~13K unchanged (no chunk approximated with 16K chunks).
Failed: none observed.
Not run: CUDA and SYCL builds (the changed lines are HIP-only); other AMD architectures; KVA quality on
agent-style long system prompts.
Limits:
- KVA output quality is unmeasured: the long-prompt battery checks recall (24/24 register entries at ~16K and ~32K), not whether the approximated chunks change answers. This fix removes a crash; it does not qualify KVA on gfx11.
- No KVA × GDN_CHUNKED bitwise parity cell was run. By construction the KVA state-only call (`with_out=false`) now takes the same kernels as with STRATA_GDN_CHUNKED unset (see below); end-to-end state still differs by the chunked WMMA scan used in the full calls, which is what the existing hip_gdn_chunked parity covers.Mehr auf der Site
Links zu Install, Modellen, Releases.