#6111 [Engine] STRATA_OWN_CPU_QUANT=ON (0.2.15 x86_64 default) is ~4% slower than ggml-cpu on AVX2 CPU expert pool decode (IQ3_XXS)

open · @wronglebowsk · 5 comentarios · En GitHub

BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quantsWindowsLinux

Descripción

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

In 0.2.15 (`CMakeLists.txt:38-45`), `STRATA_OWN_CPU_QUANT` defaults to `ON` on `x86_64` (whereas commit `ead5a32ae` reverted the `arm64` default back to `OFF` after DGX Spark pool decode measured ~1.5% slower with `STRATA_OWN_CPU_QUANT=ON`).

On a dual-socket Broadwell AVX2 host (`2x Xeon E5-2683 v4`, 32C/64T, 30 CPU expert-pool workers) running `Qwen3.8-Flash-Next UD-IQ3_XXS` across a 3-GPU layer split (10,575 of 24,576 expert pairs cached in VRAM; ~10-16% of routed experts served by the CPU pool), building with `-DSTRATA_OWN_CPU_QUANT=ON` (`src/quant/x86/`) measures 2.6% to 4.3% slower on decode in `tools/strata_bench.py --long` than building the identical source tree and flags with `-DSTRATA_OWN_CPU_QUANT=OFF` (`ggml-cpu`):

| Metric (`strata_bench.py --long`) | `STRATA_OWN_CPU_QUANT=ON` (default) | `STRATA_OWN_CPU_QUANT=OFF` (`ggml-cpu`) | Change |
|---|---:|---:|---:|
| `chat decode tok/s` (median, n=9) | 84.00 (77.80 - 94.00) | 86.20 (76.60 - 94.00) | +2.6% with OFF |
| `agent decode tok/s` (median, n=15) | 97.10 (84.20 - 115.60) | 101.30 (84.20 - 122.70) | +4.3% with OFF |
| `agent turn first token (s)` | 5.21 s | 5.12 s | -1.7% with OFF |

Expected behavior: `STRATA_OWN_CPU_QUANT=ON` on AVX2 should match or beat `ggml-cpu` on `IQ3_XXS` CPU expert pool decode, or default to `OFF` on AVX2 until the `src/quant/x86` `iq3_xxs` dot matches `ggml-cpu` (as `ead5a32ae` did on `arm64`).

### Smallest steps or command that reproduce it

```bash
cmake -B build-on -DSTRATA_ENABLE_CUDA=ON -DSTRATA_MMQ=ON -DSTRATA_OWN_CPU_QUANT=ON
cmake --build build-on --target strata -j32

cmake -B build-off -DSTRATA_ENABLE_CUDA=ON -DSTRATA_MMQ=ON -DSTRATA_OWN_CPU_QUANT=OFF
cmake --build build-off --target strata -j32

python3 tools/strata_bench.py --url http://127.0.0.1:8081 --long
```

### Reproduction status

Reproduced locally

### Strata version

Strata 0.2.15 · engine 0.1.61 (commit c77572806)

### Graphics card, memory and system

RTX 5060 Ti 16 GB + RTX 2080 Ti 11 GB + RTX 3080 Ti 12 GB · 2x Xeon E5-2683 v4 (AVX2, 32C/64T) · 188 GiB DDR4 RAM · Ubuntu Linux 7.3.4

### Model, size and settings

Qwen3.8-Flash-Next UD-IQ3_XXS · 512K context (--kv k8v4) · --layer-split 18,33 --pipeline-windows 2 --spec 4 --mtp

### Logs

```text
strata generate: CPU expert kernels: AVX-2
strata cpu pool: NUMA pool on: 4 NUMA nodes found, one slice of the pool per node (STRATA_NUMA_POOL=0 turns it off)
strata cpu pool: NUMA split on: slice 0 = node 0 (8 threads), slice 1 = node 1 (7 threads), slice 2 = node 2 (8 threads), slice 3 = node 3 (8 threads)
strata generate: CPU pool tasks/phase: 93 (automatic), participating threads: 31
strata generate: CPU pool: 30 workers by default (32 physical cores, 64 logical CPUs)
strata generate: token graph hit path: 10575 resident experts, decided on the device
```

En el sitio

Enlaces a install, modelos, releases.