Pull requests / #3453

#3453 [Engine] the sm_89 (Ada) MMVQ-IL table, measured on an RTX 4070

open · @rodi4eg · 0 comentários · No GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentation

Descrição

## Problem and result

`docs/MMVQ_IL_TABLE.md` asks for a table per card and names **sm_89 (Ada)** as the gap: *"sm_89 (Ada) and every other sm_80+ card -> `kIlRowsShared` too. Nobody has measured sm_89 yet (we have no such card)."* So an Ada card takes the shared table, and that table has **no Q3_K row at all**, which means Q3_K shapes run the plain kernel.

This change adds the sm_89 table: `kIlRows_89[]` (2-4 columns), `kIlRows58_89[]` (5-8 columns) and `kIl1Rows_89[]` (one column), with the `{89, ...}` entries in `kIlArch`, `kIl58Arch` and `kIl1Arch`, and the docs row updated.

Every choice in a table is bitwise the same output as `native_mmvq` (the parity test checks that), so this is speed only - the answers do not change.

The table differs from the shared one in every type. Example shapes from the tool's own output on this card (`IQ4_XS 2560x10240`): 4 columns 54.4 us plain vs **35.2 us** with rows 1; 8 columns 71.8 us plain vs **46.5 us** with rows 1.

## Related discussion

none

## Approach

The block the repo's own tool prints, pasted as the three tables and registered for cc 89. The 3% rule is the tool's: a cell is non-zero only if the fastest rows count takes at least 3% off `native_mmvq`'s time in *every* measured shape of that class, else 0; a class with no measured shape copies its neighbour below.

The raw tool output (parity result, every BENCH line and the printed table) is in the report folder, so the numbers can be checked without rerunning anything.

## Validation

**Passed:**

- `mmvq_il_parity --bench --emit-table` ends with
  `mmvq_il_parity: OK (every case, T 1-8, rows 1/2/4 and the table, bitwise)` - the parity gate the docs ask for, including the new table itself.
- The changed translation unit compiles for sm_89 with CUDA 12.0: the project's own nvcc command (from
  `ninja -t commands` in the build folder) re-run against the patched source, producing the object file.
- Without a rebuild the same table can be injected with `STRATA_MMVQ_IL_ROWS="type:ncols:r0,r1,r2,r3,r4;..."` (`src/kernels/cuda/native_mmvq.cu`, `il_env_rows`); the string for this card is in the report folder.

**Failed:**

- none observed

**Not run:**

- HIP, SYCL and Metal builds (no toolchains on this machine; the table is a CUDA file).
- The table was not measured on any other card, and the 5-8 column path is the second such table (after sm_120) and the least exercised one.
- A full engine build in a separate worktree was not run for this change (the translation unit that carries the tables
  was compiled instead, with the project's own nvcc command); the engine itself was not relinked with them.

## Limits

- One card: RTX 4070, 46 SMs, driver 580.126.09, CUDA 12.0, one run of the tool.
- The 5-8 column table is the least exercised path in the engine; the 2-4 column table is the one that runs in every decode step.
- On the model measured here (Coder IQ1_M) the table did not change the decode speed measurably: the bottleneck is the expert path, not the head's dense projections. It should be tried on a card and a model where the dense projections matter.

No site

Links install, modelos, releases.