#6228 [Engine] split: Gyro trellis experts (TQ2_T, TQK6, TQK7) on Turing (sm_75), opt-in
open · draft · @grogroningem-code · 0 commentaires · Sur GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentation
Description
## Problem and result The all-cards split engine couldn't load the Gyro files of Qwen3.8-Flash-Next, whose routed experts are trellis types (TQ2_T, TQK6, TQK7) with a Hadamard rotation folded into the weights. On Turing it didn't even get that far: the prompt's recurrent scan wants more than the 64 KB of shared memory a block gets on sm_75, so the engine stopped at start and the server fell back to the layer split. With this PR, `--split-engine on` runs Gyro-S and Gyro-M on two RTX 2080 Ti 22 GB, and the greedy answers match the layer split's (qcheck: Gyro-S 11/11 on both engines; Gyro-M 10/11 on both, with the same miss). Without the explicit "on" nothing changes, because Auto keeps trellis files on the layer split on every card. Is it worth turning on? For one user, yes. I ran both engines on the same two cards in one session, with the same benchmark and the same drafting (MTP, up to 4 tokens ahead). Steady decode in tok/s, new chats / follow-up turn: | model, chats at once | layer split | split engine | |---|---|---| | Gyro-S, 1 | 75.3 / 71.9 | 78.9 / 83.1 | | Gyro-S, 2 | 126.2 / 127.3 | 102.8 / 104.0 | | Gyro-S, 4 | 164.4 / 163.2 | 135.1 / 130.8 | | Gyro-M, 1 | 65.4 / 61.9 | 74.5 / 75.3 | | Gyro-M, 2 | 112.7 / 109.8 | 103.9 / 102.8 | | Gyro-M, 4 | 147.1 / 146.2 | 132.2 / 130.0 | One chat gains 5 to 22%; two or four chats lose 6 to 20%, which is why I left it opt-in and didn't touch Auto. The doc adds totals, time to first token, draft acceptance and a run with drafting off. ## Related discussion Stacked on #5119 (split engine on sm_70, opt-in). The branch is #5119 merged into main `c77572806` plus one commit, and only that last commit is new. There's no scope issue yet; I opened this as a draft so you can tell me whether you want it. ## Approach The engine rotates the routed experts' inputs the same way the main engine's `hadamard_rows` does: blocks of 128, explicit signs from the `prism.hadamard.*` keys. The shared expert isn't rotated, so it keeps its own copy of the unrotated input. For the weights themselves I reused the main engine's trellis decode (`tq_load` / `tq_apply`, int8 codebook, dp4a), and prompts dequantize trellis rows to f16 tiles for the tensor-core product. Down was the awkward part. Its 640-wide input splits 320/320 between the cards, but rotation block [256, 384) sits across both of them. Each card now keeps the whole blocks around its slice (card 0 columns [0, 384), card 1 [256, 640)) and feeds zeros for the columns the other card computes; the product is linear, so the two partial sums still add up to the full result. That costs 20% more down-matrix memory. Gyro-S keeps its PLE table in a separate GGUF (`FNX_PLE_GGUF`, from `--ple-gguf`). The engine accepts that file only if it holds `per_layer_token_embd.weight` and nothing else, and only when the model has no table of its own. On sm_75 the recurrent prompt scan takes the 16-column layout the HIP build already uses for the same 64 KB limit. Compatibility. No main-engine file changes: the diff stays inside `src/split`, `strata_control/split_engine.py`, `strata_cli/drafting_cmd.py`, `tests/split` and one new doc, and trellis files only reach the split engine through an explicit "on". The one behaviour change outside Gyro comes from the RAM rule, which under-counted. With the host expert cache (sm_70, sm_75) the engine keeps a page-locked copy of every routed expert, and the loader keeps the dense tensors it converts on its heap; the rule counted neither. A 2 x V100 32 GB host with UD-Q4_K_XL at 393216 context now asks 106.2 GiB plus the 8 GiB spare instead of 60.2 plus the spare, so a V100 box with roughly 68 to 114 GiB free that used to get the split engine now stays on the layer split. The heap term also adds a few GiB in Auto for packs the loader converts. I only worked that out from the rule (about 42 to 46 GiB for the IQ3_S set) and haven't run such a host. On AMD, "on" refuses trellis experts outright, since nobody has built the HIP version of these kernels; on an NVIDIA card other than sm_75 it starts them with a "not tested" note. A review of the first version turned up these defects, all fixed here: - Prompt chunks over 6553 rows broke down. The kernel that builds the rotated h put rows x 10 experts on the grid's y axis, which stops at 65535, so with `FNX_PF_ROWS=8192` a 10215-token prompt got HTTP 400 `swiglu_h16_had: invalid argument`. It answers with HTTP 200 now, and `split_trellis_test` has a 70000-pair case that failed before the fix. - The RAM rule asked 60.8 GiB (plus spare) for Gyro-S, while three loads took 84.3 to 86.9 GiB of `MemAvailable` (26.7 GiB of that was the page-locked expert copy in `Shmem`, 5.6 GiB heap). Now it asks 88.1 GiB, and 90.2 for Gyro-M against a measured 86.0. - Auto accepted trellis files on sm_86 and sm_120 pairs that nobody has run them on. - The separate PLE file could add or replace any tensor in the model, and an empty one got through. - `expected()`, which answers before a download, only knew about sm_70 and ignored `--ple-gguf`, so on sm_75 it disagreed with the actual launch; `drafting plan|fetch` didn't pass it the engine arguments either. - `FNX_PF_MOE_Q8` sends experts to big-job kernels that apply no rotation. It was refused for trellis experts but not for rotated experts of other types; it's now refused for any rotated expert and read with `env_flag`. - The new kernel test read past one of its buffers (7870 `compute-sanitizer` errors, 0 now), and a NaN difference slipped past its bound. ## Validation Tested on Ubuntu with 2 x RTX 2080 Ti 22 GB (sm_75), CUDA 13.5 and g++-13, on this PR's last commit. The code is exactly what these runs used; I only edited the doc text afterwards. **Passed:** - Build for sm_75 (`-DSTRATA_SPLIT_ARCH=75 -DSTRATA_SPLIT_CARDS=2`, full command in the doc): no errors, and the warnings are ones the tree already had. - Compile-only checks of the same tree: sm_86 with 2 and 4 cards (`-DSTRATA_SPLIT_ARCH=86 -DSTRATA_SPLIT_CARDS="2;4"`, CUDA 13.5) and sm_70 with 2 cards (`-DSTRATA_SPLIT_ARCH=70 -DSTRATA_EXPERIMENTAL_SM60=ON`, CUDA 12.4, because CUDA 13 dropped sm_70). Both built `strata-split-*`, `split_trellis_test`, `split_gguf_test`, `split_qdot_test` and `split_moe_tile_test` without errors. `git blame` puts every warning on a line older than this PR, and `ctest -R split_gguf_test` passes on both. - `split_trellis_test` on both real model files: `PASS: 0 check(s) failed`, 203 checks, exit 0. It compares the new kernels with the main engine's CUDA kernels (`hadamard_rows`, `iq_dequant_f16`, `iq_mmvq`) or a double-precision reference. Rotation, dequantized tiles and decode gate/up come out bit-equal; down summed over both cards lands within the main engine's own error (relative L2 1.063e-2, against 1.067e-2 for `iq_mmvq` on the whole matrix). - `compute-sanitizer --tool memcheck` on the two new h kernels: `ERROR SUMMARY: 0 errors`. - `split_gguf_test` (ctest): all passed, PLE-file cases included (3 of them failed before the check existed). - `split_qdot_test`: PASS. - `python -m unittest tests.split.test_split_engine tests.split.test_split_gen_keys tests.split.test_split_hcsplit tests.terminal.test_split_parallel tests.terminal.test_drafting_slot_default tests.terminal.test_app_first_load_drafter_0214`: 152 tests OK. - `FNX_PF_GDN_CHECK=1`, which runs the 16-column scan beside the row-by-row one: over 2304 calls of 31 to 4096 rows the largest difference was 5.9e-5 of the largest value. - qcheck (greedy, 11 questions) on both engines, plus the speed and RAM runs written up in `docs/SPLIT_TURING_TRELLIS.md`. **Failed:** none observed. **Not run:** - HIP. There's no ROCm on this host, which is why the rule refuses trellis experts on AMD; if someone can try a HIP build, I'd be glad of the help. - Anything on cards other than the 2080 Ti, on 3 or 4 cards, or with non-Gyro packs on sm_75. - Contexts past roughly 60k tokens on the split engine. - `split_moe_tile_test` (built, never run). - The full `python -m pytest tests/terminal` suite; I ran only the modules above, with unittest. ## Limits With two or more chats at once the split engine is slower than the layer split (see the table). I haven't profiled why, though my guess is the exchange of partial sums between the cards after every layer, while the layer split keeps both cards busy with different chats. It also keeps int8 K/V, so a layer-split config with `--kv k8v4` needs that flag dropped. With the host expert cache, a Gyro load takes about 86 GiB of host RAM. And the RAM rule still misses the loader's Q2_0 to Q4_0 repack copies (about 4 GiB, IQ3_S set on AMD), a gap that was there before this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sur le site
Liens install, modèles, releases.