Pull requests / #1659
#1659 prefill gemm: recover the FP16 shapes cuBLAS 12's default algorithm fails on (issue #1650)
open · @W1nge · 0 commentaires · Sur GitHub
NVIDIA / CUDAModels & quantsWindows
Description
Issue #1650: with more than one CUDA device initialized in the process, cuBLAS 12 answers the default algorithm of a few FP16 GEMM shapes (`CUDA_R_16F` operands, `CUBLAS_COMPUTE_32F`) with `CUBLAS_STATUS_INTERNAL_ERROR` (status 14), and the request aborts with `prefill gemm: cublasGemmEx f16: cuBLAS status 14`. Only initializing the other device's context is enough to set the state that does it: on this host, `cudaSetDevice(1); cudaFree(0);` on the P100 makes the default-algorithm FP16 GEMM fail on the 2080 Ti before any work has run there. The failing shapes depend on K as well as N (K = 10240: N a multiple of 8 is safe; K = 2560: N = 8..32 fails too), so a "small N" guard is not enough. The fixed algorithms (ALGO2 and up) run through that state, but they are markedly slower for the wide shapes. So a `Gemm` moves only the shapes that failed: it remembers their `(N, K)` ((K << 32) | N, 32 slots) and runs them on `CUBLAS_GEMM_ALGO2`. The failing call is harmless and is paid once per shape per process. Measured on one Windows host (CUDA 12.4.131, RTX 2080 Ti sm_75 + Tesla P100 sm_60, driver 537.13), the 3-request fixture with `STRATA_BF16_TC=1` (the abort is the stock engine, not a number): | FP16 algorithm | 2K prompt | 5.1K prompt | | --- | --- | --- | | stock (`CUBLAS_GEMM_DEFAULT`) | aborts | aborts | | every shape on ALGO2 | 7113 ms | 11515 ms | | this change | 6261 ms | 7499 ms | For reference, the same fixture with the tensor cores off is 6333 ms / 7899 ms: this change keeps the `STRATA_BF16_TC` win and removes the abort. Tested: the stock engine aborts on `--short-read 0` (40-token prompt, `STRATA_BF16_TC=1`, `cublasGemmEx f16: cuBLAS status 14`) and this change answers it (`[3]`, valid, 3.1 s); the 2K / 5.1K / warm fixture and the daily lend/source regression all pass with the same answers.
Sur le site
Liens install, modèles, releases.