#5873 [Engine] Gyro-L: 2.3x faster AVX2 CPU rows for the pair trellis (TQP5/6/7), bit-exact

closed · @Ante042 · 0 commentaires · Sur GitHub

BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Description

## Problem and result
Gyro-L's pair-trellis experts (TQP5/6/7, ggml 147-149) are slow on the CPU. With the default (auto) settings, Strata measured the CPU pool at **874 us per missed expert (2.3 GB/s)** on a Ryzen 7 5800X. The same pool does Gyro-M's TQ2_T experts at ~120 us. Gyro-L does not fit in 12 GB of VRAM, so decode is limited by this CPU path.

This PR makes the AVX2 TQP rows **2.3-2.8x faster per thread** with bit-identical results. In the engine, Strata's own calibration then reports **362 us per missed expert (5.4 GB/s)**, and Gyro-L decode with default settings goes from **22.2 to 34.4 tok/s** on the same PC, same day.

## Related discussion
none. This is a focused speed fix in one kernel and does not change defaults or behaviour.

## Approach
There are three changes. Each keeps the same integer math, so the output is bitwise equal:

1. `tq_states_tqp` (`tq_states.hpp`, shared by the scalar and AVX2 kernels): before, each of the 64 states needed three byte reads and two `%` operations for the circular wrap. Now the 8*K stream bytes are copied once into a small buffer with their first 4 bytes appended, so each state is one 32-bit read, a shift and a mask. This is the same idea `tq_states` already uses for the 4-weight layout. `tq_state_tqp` (one state) is kept unchanged as the readable definition.
2. `rows_tqp_t` (`tq_avx2.cpp`): a step's two weights are the first int32 of its 8-byte table row. The 8 first-pairs of a YMM are now loaded with `_mm256_setr_epi32` from the table viewed as int32, replacing `pairs8`'s 8 loads + 6 unpacks. I also measured `_mm256_i32gather_epi32` (scale 8): same speed on Zen 3, so I kept the plain loads.
3. The four group sums use one `hadd` tree, the same one `rows_t` uses, instead of per group `extract + 2x hadd + movd` and an `_mm_set_epi32`. The int32 sums are the same values, so the float part is unchanged.

There are no compatibility effects. The block layout, table, ISA dispatch and AVX-512 routing are unchanged (AVX-512 still takes this AVX2 pair kernel).

## Validation
**Passed:**
- `tqp_cpu_test`, MSVC 19.44 (VS 2022 Build Tools 17.14), `/O2 /arch:AVX2`, Ryzen 7 5800X, Windows 11: `tqp_cpu_test: 0 failures`. This covers the AVX2 TQP rows (1..8 tokens, row ranges, fused gate/up) bitwise equal to scalar, and scalar against the format definition.
- `tqp_cpu_test`, GCC 14.2 (Debian), `-O2 -std=c++20 -Wall -Wextra -Werror`, on a Xeon E5-2667 v2 (no AVX2, so only the scalar path runs, and it uses the changed `tq_states_tqp`): `0 failures`. For 147/148/149, `scalar vs the definition (double): 0 of 296 outside 2e-5`, worst 2.5e-08.
- Single-thread micro-benchmark on the 5800X (1536 rows x 2048, random blocks, 10 iterations each; a small harness that times `tq_rows_avx2_tqp` before/after and memcmp's the outputs for nt = 1, 2, 4, 8, all bit-exact):

  | type | nt | before | after | speedup |
  |---|---|---|---|---|
  | 147 (K=5) | 1 | 3678 us | 1546 us | 2.38x |
  | 147 (K=5) | 4 | 4055 us | 1756 us | 2.31x |
  | 148 (K=6) | 1 | 3781 us | 1645 us | 2.30x |
  | 148 (K=6) | 4 | 4174 us | 1892 us | 2.21x |
  | 149 (K=7) | 1 | 4177 us | 1586 us | 2.63x |
  | 149 (K=7) | 4 | 4652 us | 1828 us | 2.54x |

  For reference, TQ2_T (144) takes 570 us for the same shape. TQP needs twice the table lookups per weight, so ~2x TQ2_T is roughly its floor.
- Full CUDA engine build (Windows, CUDA 13.4, sm_86, Ninja, `-DSTRATA_ENABLE_CUDA=ON -DSTRATA_Q6K_EXPERTS=ON -DSTRATA_MMQ_KQUANTS=ON`) of commit 712fec6b4297 (engine 0.1.59) with this change; both files are identical on that commit and on current `main`. End-to-end on RTX 3080 Ti 12 GB (PCIe 3.0 x16), Ryzen 7 5800X, 64 GB DDR4-3600, Windows 11, `Qwen3.8-Flash-Next-Gyro-L-TQ3_0.gguf`, 256K context, `--vram-reserve-mib 1216`, otherwise the default profile. Same day and same settings; the only difference is the engine binary. Decode is from a ~5K-token prompt, 3 runs each:

  | engine | CPU pool (Strata's calibration) | auto pcie_frac | decode runs (tok/s) | median | prefill 31K |
  |---|---|---|---|---|---|
  | official 0.1.59 | 874 us/expert, 2.3 GB/s | 0.90 | 18.2, 23.3, 22.2 | 22.2 | 1066 tok/s |
  | 0.1.59 + this PR | 362 us/expert, 5.4 GB/s | 0.75 | 30.0, 36.7, 34.4 | 34.4 | 1077 tok/s |

  For comparison, `--pcie-frac 1.0` (no CPU experts at all) gave 31.5 tok/s with the same setup, so the faster CPU path now adds to what the GPU does.
- Answers were checked through the OpenAI API after the change: normal Swedish/English text, no change visible.

**Failed:**
none observed

**Not run:**
- An AVX-512 machine (I have none; it takes the same AVX2 pair kernel, so the AVX2 result applies, but it was not executed there).
- HIP, SYCL and Metal builds. `tq_states.hpp` is also compiled into the scalar path that arm64/Metal uses; that scalar path was only tested with GCC on x86-64.
- A Linux end-to-end run, and a build of current `main` with CUDA (the change was built on 712fec6b4297; the two files have not changed since).
- Gyro-M / TQ2_T are not touched by this change and were not re-benchmarked.

## Limits
- Prompt reading (prefill) is unchanged. On this PC it is limited by PCIe 3.0.
- The gain depends on how much of the model runs on the CPU. It matters for setups where Gyro-L does not fit in VRAM.
- AI-assisted: the change was prepared with an AI agent. I reviewed the diff, and every number above comes from runs on my own hardware.

Sur le site

Liens install, modèles, releases.