#6197 [Engine] int8 expert kernels: IQ4_XS as a down format
open · @schmittcn · 0 评论 · 去 GitHub 看
BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quants
说明
## Problem The int8 expert kernels (`native_expert_i8`) take IQ4_XS as a gate/up format but not as a down format. A Qwen3.6-family GGUF whose routed experts are all IQ4_XS (i1-IQ4_XS requants, for example Ornith-1.5-35B-A3B i1-IQ4_XS or Qwen3.6-35B-A3B-heretic IQ4_XS, loadable with #3925) therefore has no int8 route. Since 0.2.16 (3e26b4e15) there is no MMQ build at all, so these layers read the prompt on the native grouped kernels. ## Result Ornith-1.5-35B-A3B i1-IQ4_XS on one RX 6900 XT (gfx1030), on 0.2.16 (`58ee3227`). Cold 40,207-token prompt, from the engine's own `prompt ... read` line. 3 interleaved rounds per arm, a fresh server each, 3 cold prompts per round: | | prompt tok/s (9 requests) | path in the start line | |---|---|---| | before (0.2.16 + #3925) | 329.1–439.0 (the first cold prompt after each start 329–332, then 432–439) | native grouped kernels | | after (+ this PR) | 715.9–721.1 | 41 of 41 layers on our int8 kernels | That is about 1.65x on the steady requests and 2.2x on the first one. It's the same as the first measurement on 0.2.15 (`c7757280`), which was 327.5–436.4 before and 718.2–720.1 after over 2 rounds. ## Approach The decoder is the one the kernel already has for IQ4_XS gate/up. As in f66a48b9d (Q5_K / Q6_K as down formats), the change is: - the format predicate `i8_d_type`; - the shared-memory size table; - the down dispatch; - the `n_ff % 256` condition of `native_expert_i8_supported`. `native_i8_parity` gets a fifth wave: IQ4_XS / IQ4_XS, Q4_K / IQ4_XS and Q6_K / IQ4_XS at 2048 x 512 (the Qwen3.6 shape) and 512 x 256. ## Validation **Passed** - HIP build (ROCm 7.2.4, gfx1030, 0.2.16 setup's flags) of `strata`, `strata-q36` and `native_i8_parity`, before and after the change. The warning count is the same in both build logs. - `native_i8_parity` on the RX 6900 XT, on 0.2.16: `all ok`, before and after the change. The 6 new rows (IQ4_XS / IQ4_XS, Q4_K / IQ4_XS, Q6_K / IQ4_XS at 512 x 256 and 2048 x 512) have a worst relative RMS against the grouped kernels of 1.97e-3 to 2.66e-3. The existing rows are 1.58e-3 to 2.77e-3, the same in both builds. - Default path unchanged for setup's files: no file in setup's catalog has IQ4_XS downs. - Checked from the GGUF headers on this PC: Qwen3.6 UD-Q4_K_XL (Q5_K / Q6_K downs); Qwen3.8-Flash-Next GSQ-RCO IQ2_XS (Q2_0), Coder IQ1_M and Swift IQ3_S (IQ4_NL / Q2_0 downs; their one IQ4_XS layer is gate/up, which was already covered). - The other catalog files (UD-Q6_K_XL, Q8_0, Flash-Next UD-IQ4_XS) are from setup's comments, not checked from their files here. - Teacher-forced full logits after a 30K-token prompt, 201 steps (the #3926 sequence), KL against the native grouped kernels. The first row was re-run on 0.2.16 and gave the same figures as on 0.2.15, to four digits. The MMQ row is from 0.2.15, since 0.2.16 has no MMQ build: | | mean KL | p99 | max | top-1 same | |---|---|---|---|---| | Ornith IQ4_XS, int8 (this PR) | 0.0167 | 0.1673 | 0.1986 | 94.0% | | Ornith IQ4_XS, MMQ (same file, `-DSTRATA_MMQ=ON` build) | 0.0128 | 0.1229 | 0.1908 | 95.0% | | Qwen3.6 UD-Q4_K_XL, int8 vs `STRATA_EXPERT_I8=0`, for scale | 0.0020 | 0.0133 | 0.0356 | 97.0% | On this file, the int8 path is in the same range as MMQ against the grouped kernels. The model itself moves more under int8 activations than UD-Q4_K_XL does. **Not run** - CUDA build: no CUDA toolkit on this machine. The change is in the shared `.cuh` / `.cu` code, so CUDA needs a build, and CI or a maintainer would have to run it. SYCL does not compile these files. - Any model from setup's catalog with IQ4_XS downs: there is none. ## Limits - Decode on short prompts was 1–10% lower in the "after" arm. On 0.2.16, 400-token stories after a 32-token prompt, the 1st / 2nd / 3rd request per round ran at 69.6–71.3 / 60.4–62.2 / 60.5–62.2 tok/s after, and 71.9–72.3 / 66.6–67.4 / 62.6–64.7 tok/s before. All 3 rounds were alike, and 0.2.15 showed a similar gap (3–8%). After the 40K prompts, decode was level at 27.2–27.9 against 27.5–28.2 tok/s. The change only touches the prompt path, and the generated text differs between the arms (see the KL), so I could not separate a real cost from different text. It is stated here, not explained. - Measured on one model and one card. The file needs #3925 to load on main, so the arms ran on main + #3925 (0.2.16 `58ee3227` + `eac0ae3e`; before that, c77572806 + 31f7de16d). This PR itself is on main. - Ornith is not in setup's catalog. It is the same architecture as Qwen3.6-35B-A3B (`qwen35moe`) and runs on the same engine path; any all-IQ4_XS Qwen3.6 file takes the same route. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
本站相关内容
相关页面的快捷入口。