#6228 [Engine] split: Gyro trellis experts (TQ2_T, TQK6, TQK7) on Turing (sm_75), opt-in

open · draft · @grogroningem-code · 0 Kommentare · Auf GitHub

BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentation

Beschreibung

## Problem and result

The all-cards split engine couldn't load the Gyro files of Qwen3.8-Flash-Next, whose routed experts are trellis types
(TQ2_T, TQK6, TQK7) with a Hadamard rotation folded into the weights. On Turing it didn't even get that far: the
prompt's recurrent scan wants more than the 64 KB of shared memory a block gets on sm_75, so the engine stopped at
start and the server fell back to the layer split.

With this PR, `--split-engine on` runs Gyro-S and Gyro-M on two RTX 2080 Ti 22 GB, and the greedy answers match the
layer split's (qcheck: Gyro-S 11/11 on both engines; Gyro-M 10/11 on both, with the same miss). Without the explicit
"on" nothing changes, because Auto keeps trellis files on the layer split on every card.

Is it worth turning on? For one user, yes. I ran both engines on the same two cards in one session, with the same
benchmark and the same drafting (MTP, up to 4 tokens ahead). Steady decode in tok/s, new chats / follow-up turn:

| model, chats at once | layer split | split engine |
|---|---|---|
| Gyro-S, 1 | 75.3 / 71.9 | 78.9 / 83.1 |
| Gyro-S, 2 | 126.2 / 127.3 | 102.8 / 104.0 |
| Gyro-S, 4 | 164.4 / 163.2 | 135.1 / 130.8 |
| Gyro-M, 1 | 65.4 / 61.9 | 74.5 / 75.3 |
| Gyro-M, 2 | 112.7 / 109.8 | 103.9 / 102.8 |
| Gyro-M, 4 | 147.1 / 146.2 | 132.2 / 130.0 |

One chat gains 5 to 22%; two or four chats lose 6 to 20%, which is why I left it opt-in and didn't touch Auto. The doc
adds totals, time to first token, draft acceptance and a run with drafting off.

## Related discussion

Stacked on #5119 (split engine on sm_70, opt-in). The branch is #5119 merged into main `c77572806` plus one commit, and
only that last commit is new. There's no scope issue yet; I opened this as a draft so you can tell me whether you want it.

## Approach

The engine rotates the routed experts' inputs the same way the main engine's `hadamard_rows` does: blocks of 128,
explicit signs from the `prism.hadamard.*` keys. The shared expert isn't rotated, so it keeps its own copy of the
unrotated input. For the weights themselves I reused the main engine's trellis decode (`tq_load` / `tq_apply`, int8
codebook, dp4a), and prompts dequantize trellis rows to f16 tiles for the tensor-core product.

Down was the awkward part. Its 640-wide input splits 320/320 between the cards, but rotation block [256, 384) sits
across both of them. Each card now keeps the whole blocks around its slice (card 0 columns [0, 384), card 1
[256, 640)) and feeds zeros for the columns the other card computes; the product is linear, so the two partial sums
still add up to the full result. That costs 20% more down-matrix memory.

Gyro-S keeps its PLE table in a separate GGUF (`FNX_PLE_GGUF`, from `--ple-gguf`). The engine accepts that file only
if it holds `per_layer_token_embd.weight` and nothing else, and only when the model has no table of its own. On sm_75
the recurrent prompt scan takes the 16-column layout the HIP build already uses for the same 64 KB limit.

Compatibility. No main-engine file changes: the diff stays inside `src/split`, `strata_control/split_engine.py`,
`strata_cli/drafting_cmd.py`, `tests/split` and one new doc, and trellis files only reach the split engine through an
explicit "on". The one behaviour change outside Gyro comes from the RAM rule, which under-counted. With the host expert
cache (sm_70, sm_75) the engine keeps a page-locked copy of every routed expert, and the loader keeps the dense tensors
it converts on its heap; the rule counted neither. A 2 x V100 32 GB host with UD-Q4_K_XL at 393216 context now asks
106.2 GiB plus the 8 GiB spare instead of 60.2 plus the spare, so a V100 box with roughly 68 to 114 GiB free that used to
get the split engine now stays on the layer split. The heap term also adds a few GiB in Auto for packs the loader
converts. I only worked that out from the rule (about 42 to 46 GiB for the IQ3_S set) and haven't run such a host. On AMD,
"on" refuses trellis experts outright, since nobody has built the HIP version of these kernels; on an NVIDIA card other
than sm_75 it starts them with a "not tested" note.

A review of the first version turned up these defects, all fixed here:

- Prompt chunks over 6553 rows broke down. The kernel that builds the rotated h put rows x 10 experts on the grid's y
  axis, which stops at 65535, so with `FNX_PF_ROWS=8192` a 10215-token prompt got HTTP 400
  `swiglu_h16_had: invalid argument`. It answers with HTTP 200 now, and `split_trellis_test` has a 70000-pair case that
  failed before the fix.
- The RAM rule asked 60.8 GiB (plus spare) for Gyro-S, while three loads took 84.3 to 86.9 GiB of `MemAvailable`
  (26.7 GiB of that was the page-locked expert copy in `Shmem`, 5.6 GiB heap). Now it asks 88.1 GiB, and 90.2 for
  Gyro-M against a measured 86.0.
- Auto accepted trellis files on sm_86 and sm_120 pairs that nobody has run them on.
- The separate PLE file could add or replace any tensor in the model, and an empty one got through.
- `expected()`, which answers before a download, only knew about sm_70 and ignored `--ple-gguf`, so on sm_75 it
  disagreed with the actual launch; `drafting plan|fetch` didn't pass it the engine arguments either.
- `FNX_PF_MOE_Q8` sends experts to big-job kernels that apply no rotation. It was refused for trellis experts but not for
  rotated experts of other types; it's now refused for any rotated expert and read with `env_flag`.
- The new kernel test read past one of its buffers (7870 `compute-sanitizer` errors, 0 now), and a NaN difference
  slipped past its bound.

## Validation

Tested on Ubuntu with 2 x RTX 2080 Ti 22 GB (sm_75), CUDA 13.5 and g++-13, on this PR's last commit. The code is
exactly what these runs used; I only edited the doc text afterwards.

**Passed:**

- Build for sm_75 (`-DSTRATA_SPLIT_ARCH=75 -DSTRATA_SPLIT_CARDS=2`, full command in the doc): no errors, and the
  warnings are ones the tree already had.
- Compile-only checks of the same tree: sm_86 with 2 and 4 cards (`-DSTRATA_SPLIT_ARCH=86 -DSTRATA_SPLIT_CARDS="2;4"`,
  CUDA 13.5) and sm_70 with 2 cards (`-DSTRATA_SPLIT_ARCH=70 -DSTRATA_EXPERIMENTAL_SM60=ON`, CUDA 12.4, because CUDA 13
  dropped sm_70). Both built `strata-split-*`, `split_trellis_test`, `split_gguf_test`, `split_qdot_test` and
  `split_moe_tile_test` without errors. `git blame` puts every warning on a line older than this PR, and
  `ctest -R split_gguf_test` passes on both.
- `split_trellis_test` on both real model files: `PASS: 0 check(s) failed`, 203 checks, exit 0. It compares the new
  kernels with the main engine's CUDA kernels (`hadamard_rows`, `iq_dequant_f16`, `iq_mmvq`) or a double-precision
  reference. Rotation, dequantized tiles and decode gate/up come out bit-equal; down summed over both cards lands within
  the main engine's own error (relative L2 1.063e-2, against 1.067e-2 for `iq_mmvq` on the whole matrix).
- `compute-sanitizer --tool memcheck` on the two new h kernels: `ERROR SUMMARY: 0 errors`.
- `split_gguf_test` (ctest): all passed, PLE-file cases included (3 of them failed before the check existed).
- `split_qdot_test`: PASS.
- `python -m unittest tests.split.test_split_engine tests.split.test_split_gen_keys tests.split.test_split_hcsplit
  tests.terminal.test_split_parallel tests.terminal.test_drafting_slot_default
  tests.terminal.test_app_first_load_drafter_0214`: 152 tests OK.
- `FNX_PF_GDN_CHECK=1`, which runs the 16-column scan beside the row-by-row one: over 2304 calls of 31 to 4096 rows the
  largest difference was 5.9e-5 of the largest value.
- qcheck (greedy, 11 questions) on both engines, plus the speed and RAM runs written up in
  `docs/SPLIT_TURING_TRELLIS.md`.

**Failed:** none observed.

**Not run:**

- HIP. There's no ROCm on this host, which is why the rule refuses trellis experts on AMD; if someone can try a HIP
  build, I'd be glad of the help.
- Anything on cards other than the 2080 Ti, on 3 or 4 cards, or with non-Gyro packs on sm_75.
- Contexts past roughly 60k tokens on the split engine.
- `split_moe_tile_test` (built, never run).
- The full `python -m pytest tests/terminal` suite; I ran only the modules above, with unittest.

## Limits

With two or more chats at once the split engine is slower than the layer split (see the table). I haven't profiled
why, though my guess is the exchange of partial sums between the cards after every layer, while the layer split keeps
both cards busy with different chats.

It also keeps int8 K/V, so a layer-split config with `--kv k8v4` needs that flag dropped. With the host expert cache, a
Gyro load takes about 86 GiB of host RAM. And the RAM rule still misses the loader's Q2_0 to Q4_0 repack copies (about
4 GiB, IQ3_S set on AMD), a gap that was there before this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.