Pull requests / #1822

#1822 prefill: gate/up MMQ reads the experts in place (CUDA; patched copy of llama.cpp mmq.cuh)

open · @xrip · 0 commentaires · Sur GitHub

Multi-GPUAMD / HIPNVIDIA / CUDAModels & quants

Description

## What

CUDA builds, native i-quant packs, prompt path: the gate/up MMQ product reads each expert's weights where they
already are (expert cache slot or stream ring slot) instead of copying them into a group buffer first. Same bytes into
the same kernel, so the same results. `STRATA_MMQ_INPLACE=0` turns it off. HIP builds keep the old path.

## Why

Before MMQ, the prompt path copied every routed expert's gate/up and down rows into a contiguous group buffer,
resident experts included. An expert blob is already "gate rows then up rows", the exact matrix MMQ wants, so the
gate/up copy is only there because MMQ takes one base pointer and a fixed stride.

## Change

- **llama.cpp is not edited.** CMake writes a copy of the pinned `mmq.cuh` into the build tree with one change in its
  three tile loops: a negative expert stride means `x` is a table of expert pointers. The MMQ instance files (and the
  headers that include `mmq.cuh`) compile against that copy. Configure stops if the patched text is not found exactly
  three times (a different llama.cpp). The changed lines are inside kernel templates, so `quantize.cu`, which sees
  the original header, is unaffected.
- **Prompt path:** a complete group's gate/up MMQ gets a table of expert pointers (`m.grp_ptrs`, written on the
  stream by `group_ptrs`). The group's ring slots are released after the gate/up MMQ instead of after the gather.
- **Down rows keep their copy on purpose:** MMQ reads one tile past a 640-value matrix's last row, and that must be
  the zeroed tail (`MMQ_TAIL`), not the next cache slot.
- A flush forced inside a group (a skipped ring entry) copies both, as before.

## Measured

2x CMP 50HX (sm_75), Flash-Next IQ3_XXS, `--layer-split auto`, one +4.4K-token prompt at 140K context (nsys):

| | before | after |
|---|---:|---:|
| `copy16_group_kernel`, GPU 0 | 75.5 ms | 31.9 ms |
| `copy16_group_kernel`, GPU 1 | 77.0 ms | 46.4 ms |
| `mul_mat_q` | 803.6 ms | 804.8 ms |
| engine read time | 6,792 ms | 6,746 ms (-0.7 %) |

A coding-agent session from 34K to 190K tokens (36 turns) gave word-for-word the same text as before.

The gain on this box is small because it is PCIe Gen2 and the second GPU waits on PCIe; a card where the copies are
a bigger part of the prompt should gain more, but that was not measured.

Note: measured on a fork based on 0.1.38. The patch applies to current `main` without changes; re-measuring on 0.1.41
is still to do. Only sm_75 was tested.

## How to check

Build (configure must pass the patch check), then compare a prompt's output and read time with
`STRATA_MMQ_INPLACE=0` and without it: the output should be the same.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sur le site

Liens install, modèles, releases.