#6576 [Engine] fix NVFP4 prefill activation quantization past 65,535 CUDA blocks

open · @li-yifei · 0 comentarios · En GitHub

Server & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Descripción

## Problem and result

On Blackwell (`sm_120`) with the default W4A4 NVFP4 prefill path (`STRATA_PREFILL_NVFP4=w4a4`), long prompt chunks fail once activation quantization needs more than 65,535 CUDA blocks in `grid.x`.

`quant_act_fp4_kernel` in `engine/src/prefill/moe_fp4_tc.cu` assigns one warp per 64-element activation block without a grid-stride loop (`w = blockIdx.x * 8 + warp; if (w >= nblk) return;`). However, `fp4_tc::quantize_act` launched it through `blocks_for(nblk, 8)`, which clamped `grid.x` to `65535` (524,280 blocks). For `cols = 2560` (40 blocks per row), every activation block from row `13,107` (0-based) onward was never written, leaving uninitialized buffer contents in `m.Xq` during large prefill chunks (for example, a 32,768-token chunk).

**Before:**
- In an isolated stock `v0.2.19` (`c6f6427d9`) server run on RTX 5090 with `STRATA_EXP_MODE=0` and default `STRATA_PREFILL_NVFP4=w4a4`, a 19-token prompt returned `"42"`, while a 31,992-token prompt hit `non-finite logits (#879)` at position 31,985 in the prompt window and again at position 31,991 on the batched retry (`0` tokens generated).
- In `prefill_fp4_tc_test`, the new `run_long_quantize_case` regression failed against the unmodified `c6f6427d9` `moe_fp4_tc.cu` object at the exact predicted boundary: `prefill_fp4_tc_test: activation block mismatch at row 13107 block 0`.

**After:**
- `fp4_tc::quantize_act` launches all `(nblk + 7) / 8` CTAs directly so every activation block is quantized across long prefill chunks.
- `prefill_fp4_tc_test` passes `run_long_quantize_case` (matching single-row quantization byte-for-byte via `memcmp` on all 40 written bytes per block for row 0 and row 13,107) and all existing MoE and dense FP4 parity cases.

## Related discussion

none

## Approach

- Remove the `blocks_for(..., 65535)` helper in `engine/src/prefill/moe_fp4_tc.cu` (its only caller was `quantize_act`) and launch `quant_act_fp4_kernel` with `grid = (unsigned) ((nblk + 7) / 8)`.
- Validate `cols < 0 || cols % K_STEP != 0` before computing block counts so negative or unaligned widths fail cleanly instead of producing negative `nblk` or wrapped unsigned grid dimensions.
- Return early when `rows <= 0 || cols == 0` so zero-work calls do not launch a 0-CTA kernel.
- Guard `rows > (kMaxGridX * 8) / blocks_per_row` (`kMaxGridX = 0x7fffffffLL`) before multiplying to prevent `int64_t` overflow or exceeding CUDA's `grid.x` limit (`2^31 - 1`).
- Add `run_long_quantize_case` in `engine/tests/cuda/prefill_fp4_tc_test.cpp`: allocates `rows = (65535 * 8) / (N / 64) + 1` (`13,108` rows at `N = 2560`), poisons the output buffer with `0xff`, runs `fp4_tc::quantize_act`, and compares rows `0` and `rows - 1` (`13,107`) byte-for-byte (`memcmp` over the 40 written bytes of each 80-byte slot: 32 payload bytes, 4 UE4M3 scales, 4-byte outer float scale) against single-row `fp4_tc::quantize_act`.

**Compatibility effects:**
- Leaves default switches (`STRATA_PREFILL_NVFP4`), the W4A8 path (`moe_fused_iq.cu`), and all non-NVFP4 formats untouched.
- For inputs within the previous limit (`nblk <= 524,280`, i.e., `<= 13,107` rows at `cols = 2560`), `quantize_act` launches the exact same CTA count and kernel code as `v0.2.19`. End-to-end byte-identity against `v0.2.19` on W4A8 or other model packs was not separately measured.

## Validation

**Environment:**
- Upstream base: `v0.2.19` (`c6f6427d9de9016fd92d13e29740f65625261df7`, engine `0.1.65`)
- Hardware: NVIDIA GeForce RTX 5090 (`sm_120`, compute capability 12.0, 32 GiB VRAM)
- OS / toolchain: Linux x86_64 (`7.1.5-76070105-generic`), GCC 13.3.0, NVIDIA driver `615.71.09`, CUDA `13.1` (`nvcc` V13.1.115), Release build (`-DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120-real`)
- Model pack (end-to-end checks): `qwen3.8-flash-next-huihui-nvfp4-gptq` (NVFP4 experts, `n_embd = 2560`, `n_ff = 640`, 48 layers, `--prefill auto:32768 --max-context 262144 --kv int8 --spec 6 --spec-min-p 0.7`)

**Passed:**
- *Negative control (reproduction of original bug):* `prefill_fp4_tc_test` linked against the unmodified `c6f6427d9` `moe_fp4_tc.cu` object failed as expected:
  ```text
  prefill_fp4_tc_test: activation block mismatch at row 13107 block 0
  ```
- *Regression test at PR head `c003073ca`:* `./prefill_fp4_tc_test` on RTX 5090 passed `run_long_quantize_case` and all parity cases:
  ```text
  strata prefill: Blackwell mxf4nvf4 Tensor Core dense NVFP4 GEMM active (STRATA_PF_DENSE_FP4=0 disables)
    T=   1 E= 16 shift=0: vs W4A4-FP64 model rel RMS = 0.000000 | vs unquant-FP64 rel RMS = 0.1701
    T=  17 E= 16 shift=2: vs W4A4-FP64 model rel RMS = 0.000000 | vs unquant-FP64 rel RMS = 0.1694
    T=  65 E= 32 shift=0: vs W4A4-FP64 model rel RMS = 0.000000 | vs unquant-FP64 rel RMS = 0.1730
    T= 517 E= 64 shift=2: vs W4A4-FP64 model rel RMS = 0.000000 | vs unquant-FP64 rel RMS = 0.1683
    DENSE T=  33 N= 192 K= 256 beta=0.0: vs W4A4-FP64 rel RMS = 0.000990 | vs unquant-FP64 = 0.1011
    DENSE T=  65 N= 320 K= 512 beta=1.0: vs W4A4-FP64 rel RMS = 0.000000 | vs unquant-FP64 = 0.1023
    DENSE T= 130 N=2560 K=2560 beta=0.0: vs W4A4-FP64 rel RMS = 0.000997 | vs unquant-FP64 = 0.0978
  prefill_fp4_tc_test: ok
  ```
- *Combined end-to-end check (pre-hardening revision, with both this fix and the companion NVFP4 grouped-down-scale fix active under all 13 SM120 defaults on RTX 5090):*
  - Exact needle retrieval passed at 31,992 prompt tokens (`CEDAR7429`) and 199,977 prompt tokens (`CEDAR7429`), plus cached follow-up turns (`7429`), and a 259,880-token near-limit prompt (`KITE8326 MORI5194 ELM2738`).
  - `/v1/responses`, code generation, automatic tool calling (`echo`), and synthetic 3-turn incident handoffs at ~32k and ~200k prompt tokens passed all assertions (6/6 JSON checks).

**Failed:**
- none observed on the patched build (stock `v0.2.19` negative control failed as expected above).

**Not run:**
- `sm_121a` (DGX Spark GB10), Windows CUDA, and non-Blackwell CUDA builds (where `moe_fp4_tc_absent.cpp` is compiled instead of `moe_fp4_tc.cu`).
- Full `STRATA_BUILD_TESTS=ON` CTest suite and `tools/check_cuda_spills.py`.
- HIP, SYCL, and Metal builds (`moe_fp4_tc.cu` is CUDA `sm_120a`/`sm_121a` only; HIP compiles `moe_fp4_tc_absent.cpp`, and SYCL/Metal do not compile this file).
- Other model packs or quantization formats beyond `qwen3.8-flash-next-huihui-nvfp4-gptq`.

## Limits

- **Unconfirmed static analysis (separate out-of-scope issue):** `engine/src/prefill/moe_fp8_mxf_tc.cu` appears to have a parallel `blocks_for(nblk, 8)` 65,535-CTA clamp on `quant_act_mxfp8_kernel` (behind opt-in `STRATA_PF_MXFP8_MOE`). That path was not tested and is intentionally left untouched here to keep one fix per PR.
- Cross-backend and non-`sm_120` CUDA builds remain unverified locally.

En el sitio

Enlaces a install, modelos, releases.