Pull requests / #1659

#1659 prefill gemm: recover the FP16 shapes cuBLAS 12's default algorithm fails on (issue #1650)

open · @W1nge · 0 コメント · GitHub で見る

NVIDIA / CUDAModels & quantsWindows

本文

Issue #1650: with more than one CUDA device initialized in the process, cuBLAS 12 answers the default
algorithm of a few FP16 GEMM shapes (`CUDA_R_16F` operands, `CUBLAS_COMPUTE_32F`) with
`CUBLAS_STATUS_INTERNAL_ERROR` (status 14), and the request aborts with
`prefill gemm: cublasGemmEx f16: cuBLAS status 14`.

Only initializing the other device's context is enough to set the state that does it: on this host,
`cudaSetDevice(1); cudaFree(0);` on the P100 makes the default-algorithm FP16 GEMM fail on the 2080 Ti
before any work has run there.  The failing shapes depend on K as well as N (K = 10240: N a multiple
of 8 is safe; K = 2560: N = 8..32 fails too), so a "small N" guard is not enough.

The fixed algorithms (ALGO2 and up) run through that state, but they are markedly slower for the wide
shapes.  So a `Gemm` moves only the shapes that failed: it remembers their `(N, K)` ((K << 32) | N, 32
slots) and runs them on `CUBLAS_GEMM_ALGO2`.  The failing call is harmless and is paid once per shape
per process.

Measured on one Windows host (CUDA 12.4.131, RTX 2080 Ti sm_75 + Tesla P100 sm_60, driver 537.13), the
3-request fixture with `STRATA_BF16_TC=1` (the abort is the stock engine, not a number):

| FP16 algorithm | 2K prompt | 5.1K prompt |
| --- | --- | --- |
| stock (`CUBLAS_GEMM_DEFAULT`) | aborts | aborts |
| every shape on ALGO2 | 7113 ms | 11515 ms |
| this change | 6261 ms | 7499 ms |

For reference, the same fixture with the tensor cores off is 6333 ms / 7899 ms: this change keeps the
`STRATA_BF16_TC` win and removes the abort.

Tested: the stock engine aborts on `--short-read 0` (40-token prompt, `STRATA_BF16_TC=1`,
`cublasGemmEx f16: cuBLAS status 14`) and this change answers it (`[3]`, valid, 3.1 s); the 2K / 5.1K /
warm fixture and the daily lend/source regression all pass with the same answers.

関連リンク

インストール・モデル・リリースへの站内リンク。