Pull requests / #1822
#1822 prefill: gate/up MMQ reads the experts in place (CUDA; patched copy of llama.cpp mmq.cuh)
open · @xrip · 0 comentários · No GitHub
Multi-GPUAMD / HIPNVIDIA / CUDAModels & quants
Descrição
## What CUDA builds, native i-quant packs, prompt path: the gate/up MMQ product reads each expert's weights where they already are (expert cache slot or stream ring slot) instead of copying them into a group buffer first. Same bytes into the same kernel, so the same results. `STRATA_MMQ_INPLACE=0` turns it off. HIP builds keep the old path. ## Why Before MMQ, the prompt path copied every routed expert's gate/up and down rows into a contiguous group buffer, resident experts included. An expert blob is already "gate rows then up rows", the exact matrix MMQ wants, so the gate/up copy is only there because MMQ takes one base pointer and a fixed stride. ## Change - **llama.cpp is not edited.** CMake writes a copy of the pinned `mmq.cuh` into the build tree with one change in its three tile loops: a negative expert stride means `x` is a table of expert pointers. The MMQ instance files (and the headers that include `mmq.cuh`) compile against that copy. Configure stops if the patched text is not found exactly three times (a different llama.cpp). The changed lines are inside kernel templates, so `quantize.cu`, which sees the original header, is unaffected. - **Prompt path:** a complete group's gate/up MMQ gets a table of expert pointers (`m.grp_ptrs`, written on the stream by `group_ptrs`). The group's ring slots are released after the gate/up MMQ instead of after the gather. - **Down rows keep their copy on purpose:** MMQ reads one tile past a 640-value matrix's last row, and that must be the zeroed tail (`MMQ_TAIL`), not the next cache slot. - A flush forced inside a group (a skipped ring entry) copies both, as before. ## Measured 2x CMP 50HX (sm_75), Flash-Next IQ3_XXS, `--layer-split auto`, one +4.4K-token prompt at 140K context (nsys): | | before | after | |---|---:|---:| | `copy16_group_kernel`, GPU 0 | 75.5 ms | 31.9 ms | | `copy16_group_kernel`, GPU 1 | 77.0 ms | 46.4 ms | | `mul_mat_q` | 803.6 ms | 804.8 ms | | engine read time | 6,792 ms | 6,746 ms (-0.7 %) | A coding-agent session from 34K to 190K tokens (36 turns) gave word-for-word the same text as before. The gain on this box is small because it is PCIe Gen2 and the second GPU waits on PCIe; a card where the copies are a bigger part of the prompt should gain more, but that was not measured. Note: measured on a fork based on 0.1.38. The patch applies to current `main` without changes; re-measuring on 0.1.41 is still to do. Only sm_75 was tested. ## How to check Build (configure must pass the patch check), then compare a prompt's output and read time with `STRATA_MMQ_INPLACE=0` and without it: the output should be the same. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
No site
Links install, modelos, releases.