#6111 [Engine] STRATA_OWN_CPU_QUANT=ON (0.2.15 x86_64 default) is ~4% slower than ggml-cpu on AVX2 CPU expert pool decode (IQ3_XXS)
open · @wronglebowsk · 5 comments · View on GitHub
BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quantsWindowsLinux
Description
### Area [Engine] (speed, kernels, models, the server and API) ### Desktop app, Terminal or source? Source (START-HERE.bat, setup.sh or a build) ### What happened In 0.2.15 (`CMakeLists.txt:38-45`), `STRATA_OWN_CPU_QUANT` defaults to `ON` on `x86_64` (whereas commit `ead5a32ae` reverted the `arm64` default back to `OFF` after DGX Spark pool decode measured ~1.5% slower with `STRATA_OWN_CPU_QUANT=ON`). On a dual-socket Broadwell AVX2 host (`2x Xeon E5-2683 v4`, 32C/64T, 30 CPU expert-pool workers) running `Qwen3.8-Flash-Next UD-IQ3_XXS` across a 3-GPU layer split (10,575 of 24,576 expert pairs cached in VRAM; ~10-16% of routed experts served by the CPU pool), building with `-DSTRATA_OWN_CPU_QUANT=ON` (`src/quant/x86/`) measures 2.6% to 4.3% slower on decode in `tools/strata_bench.py --long` than building the identical source tree and flags with `-DSTRATA_OWN_CPU_QUANT=OFF` (`ggml-cpu`): | Metric (`strata_bench.py --long`) | `STRATA_OWN_CPU_QUANT=ON` (default) | `STRATA_OWN_CPU_QUANT=OFF` (`ggml-cpu`) | Change | |---|---:|---:|---:| | `chat decode tok/s` (median, n=9) | 84.00 (77.80 - 94.00) | 86.20 (76.60 - 94.00) | +2.6% with OFF | | `agent decode tok/s` (median, n=15) | 97.10 (84.20 - 115.60) | 101.30 (84.20 - 122.70) | +4.3% with OFF | | `agent turn first token (s)` | 5.21 s | 5.12 s | -1.7% with OFF | Expected behavior: `STRATA_OWN_CPU_QUANT=ON` on AVX2 should match or beat `ggml-cpu` on `IQ3_XXS` CPU expert pool decode, or default to `OFF` on AVX2 until the `src/quant/x86` `iq3_xxs` dot matches `ggml-cpu` (as `ead5a32ae` did on `arm64`). ### Smallest steps or command that reproduce it ```bash cmake -B build-on -DSTRATA_ENABLE_CUDA=ON -DSTRATA_MMQ=ON -DSTRATA_OWN_CPU_QUANT=ON cmake --build build-on --target strata -j32 cmake -B build-off -DSTRATA_ENABLE_CUDA=ON -DSTRATA_MMQ=ON -DSTRATA_OWN_CPU_QUANT=OFF cmake --build build-off --target strata -j32 python3 tools/strata_bench.py --url http://127.0.0.1:8081 --long ``` ### Reproduction status Reproduced locally ### Strata version Strata 0.2.15 · engine 0.1.61 (commit c77572806) ### Graphics card, memory and system RTX 5060 Ti 16 GB + RTX 2080 Ti 11 GB + RTX 3080 Ti 12 GB · 2x Xeon E5-2683 v4 (AVX2, 32C/64T) · 188 GiB DDR4 RAM · Ubuntu Linux 7.3.4 ### Model, size and settings Qwen3.8-Flash-Next UD-IQ3_XXS · 512K context (--kv k8v4) · --layer-split 18,33 --pipeline-windows 2 --spec 4 --mtp ### Logs ```text strata generate: CPU expert kernels: AVX-2 strata cpu pool: NUMA pool on: 4 NUMA nodes found, one slice of the pool per node (STRATA_NUMA_POOL=0 turns it off) strata cpu pool: NUMA split on: slice 0 = node 0 (8 threads), slice 1 = node 1 (7 threads), slice 2 = node 2 (8 threads), slice 3 = node 3 (8 threads) strata generate: CPU pool tasks/phase: 93 (automatic), participating threads: 31 strata generate: CPU pool: 30 workers by default (32 physical cores, 64 logical CPUs) strata generate: token graph hit path: 10575 resident experts, decided on the device ```
Related on this site
Quick links to related pages.