#6354 [Benchmark] RTX 3060 12 GB on an AVX-only Xeon E5-2630 v2 (no AVX2/FMA): 0.2.16 (engine 0.1.62) against the 0.2.0 report, IQ2_XS and NVFP4
open · @Norhiro · 0 comentários · No GitHub
BenchmarksServer & APIMulti-GPUNVIDIA / CUDAModels & quantsDocumentation
Descrição
A re-run of my 2026-10-20 report (PR #2793, 0.2.0-beta) on 0.2.16, which now labels this build `EXPERIMENTAL older-CPU build (ggml-cpu for avx)`. Same machine, same PCIe slot, same IQ2_XS files, same tool (`tools/strata_bench.py --long`, 3 runs per test). I also ran the NVFP4-GPTQ pack. **Summary** - 0.2.16 builds from source with `STRATA_ISA_FLOOR=avx` and runs on this CPU. No SIGILL. At start-up the engine prints `this CPU has no AVX2, so the CPU's experts run on ggml-cpu's kernels - expect it to be slow` and `CPU expert kernels: AVX1 (older-CPU build)`. - With the 0.2.0 report's exact configuration, 0.2.16 is a little faster than 0.2.0 on every test: short chat 25.8 → 27.8 tok/s, prompt reading +21% at 4K and +3-4% at 16K/32K (32K TTFT 32.06 → 30.98 s). - NVFP4 fits in 12 GB, but on this CPU it is about 2.3-2.9x slower than IQ2_XS with the same configuration: decode 18.1 against 31.8 tok/s, 32K prompt 253 against 724 tok/s. NVFP4 gets fewer expert slots (935 against 3,263), and the rest are computed on the AVX1 kernels. - With my daily configuration, IQ2_XS reads prompts about 30% slower than with the 0.2.0 configuration (32K: 724 against 1,037 tok/s). Changing one setting at a time, almost all of it is the KV type: without `--kv int8`, 16K/32K prompts read 21-23% slower, although the expert cache and chunk shrink less than with a 65,536 context and int8 KV, which costs only 1%. Not binding to one NUMA node gives +24% decode and leaves prompt reading unchanged. **Hardware and software** - GPU: RTX 3060 12 GB, PCIe Gen3 x8 (the slot of the 0.2.0 report), driver 615.78.08, CUDA 13 (pip wheels). - CPU: 2x Xeon E5-2630 v2 (Ivy Bridge-EP: AVX and F16C, no AVX2/FMA), 2 NUMA nodes; 251 GiB DDR3. Models on a SATA SSD. - OS: Fedora 45, kernel 7.2.15. - Strata 0.2.16 (`64524fb9`), engine 0.1.62, source build: `cmake -S engine -DSTRATA_ENABLE_CUDA=ON -DSTRATA_ISA_FLOOR=avx -DCMAKE_CUDA_ARCHITECTURES="86;120"`, gcc 15. Server from `terminal/serve/server.py`, Python 3.15. - Models: IQ2_XS from ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF `ed59f92` (the same files and pack as the 0.2.0 report). NVFP4-GPTQ pack (`--native-dense-gguf`, `--embd-gguf`, and `--ple-gguf` from UD-Q4_K_XL part 2; the tool names it `nvfp4-UD-Q4_K_XL`). - Nothing else used the GPU (40 MiB before each run). **A. The 0.2.0 report's configuration** (40,960 context, `--kv int8`, expert cache auto, prefill auto, MTP spec 4 / min-p 0.5, no vision, no environment variables, bound to NUMA node 1, model files in the page cache) | IQ2_XS, median (range) | 0.2.0-beta, 2026-10-20 (warm) | 0.2.16, today | |---|---:|---:| | Short chat decode, tok/s | 25.8 (22.0-29.6) | 27.8 (22.4-28.9) | | Short chat TTFT, s | 1.68 | 1.27 | | 4K prompt, tok/s (TTFT s) | 633.1 (6.39) | 765.0 (5.30) | | 16K prompt, tok/s (TTFT s) | 983.2 (16.39) | 1024.9 (15.71) | | 32K prompt, tok/s (TTFT s) | 1002.3 (32.06) | 1036.8 (30.98) | | Expert cache | - | 4,126 slots, 5.54 GiB | The agent-session test changed between the two tool versions (the last turn is 14.6K tokens in the 0.2.0 run and 9.0K now), so I leave it out of this comparison. Today: cache hit 71.0%, follow-up turn TTFT 3.60 s, decode 42.0 tok/s. **B. My daily configuration, IQ2_XS against NVFP4** (65,536 context, default KV, `--vision` with the CPU picture reader, `STRATA_PREFILL_AUTO_MAX=16384`, `STRATA_PREFILL_CPU_SHARE=auto`, no NUMA binding; only the model differs) | median (range) | IQ2_XS | NVFP4-GPTQ | |---|---:|---:| | Short chat decode, tok/s | 31.8 (29.9-34.3) | 18.1 (15.5-19.9) | | Short chat TTFT, s | 1.19 | 2.34 | | 4K prompt, tok/s (TTFT s) | 560.3 (7.22) | 258.1 (15.64) | | 16K prompt, tok/s (TTFT s) | 682.6 (23.58) | 254.4 (63.20) | | 32K prompt, tok/s (TTFT s) | 723.9 (44.35) | 252.5 (127.01) | | Agent follow-up turns: TTFT s / decode tok/s | 3.80 / 40.0 | 5.90 / 16.2 | | Agent cache hit | 71.0% | 70.9% | | Expert cache; prompt chunk | 3,263 slots (4.39 GiB); 7,680 | 935 slots (2.41 GiB); 3,328 | | VRAM free with everything loaded | 699 MiB | 675 MiB | The prompt-reading ranges are within 1-2% for every size. **C. From A towards B, one setting at a time** (IQ2_XS, `strata_bench.py --long`, 3 runs each; every row is A with one change) | Change from A | Chat decode tok/s | 4K tok/s | 16K tok/s | 32K tok/s | Expert cache; prompt chunk | |---|---:|---:|---:|---:|---| | none (A) | 27.8 | 765.0 | 1024.9 | 1036.8 | 4,126; 9,984 | | `--max-context 65536` | 26.8 | 762.6 | 1013.6 | 1027.2 | 3,855; 9,216 | | default KV (no `--kv int8`) | 26.6 | **658.3** | **804.6** | **794.6** | 3,757; 8,960 | | `--vision` (CPU picture reader) | 27.9 | 763.5 | 1022.1 | 1034.8 | 4,126; 9,984 | | `STRATA_PREFILL_AUTO_MAX=16384` + `STRATA_PREFILL_CPU_SHARE=auto` | 27.8 | **621.8** | 1023.5 | 1037.1 | 4,126; 9,984 | | no NUMA binding | **34.6** | 786.6 | 1011.8 | 1025.5 | 4,126; 9,984 | | all of the above (B) | 31.8 | 560.3 | 682.6 | 723.9 | 3,263; 7,680 | - **The KV type is most of the gap.** Without `--kv int8`, 16K/32K prompts read 21-23% slower. It is not only the smaller cache: the 65,536 context with int8 KV leaves fewer slots (3,855) and a smaller chunk (9,216) than the 40,960 context with the default KV (3,757; 8,960) does, yet costs only 1%. I have not looked at why the default KV is slower to prefill here. - The two environment variables cost 19% at 4K only; I did not split them. - Without NUMA binding, short-chat decode is 24% faster and prompt reading is the same. - B loses a little more than the single changes add up to (30% at 32K, against 23% + 1%). With both the default KV and the 65,536 context, the cache shrinks to 3,263 slots and the chunk to 7,680 tokens; I did not run that pair alone. **Our own answer checks (not part of the tool, not a scored benchmark)** - RAG over Japanese railway tariff tables (our own pipeline; 16 questions, no thinking), NVFP4 daily config: 16/16 answered with the same numbers as on an RTX 5060 Ti machine; about 1.6x the time per question. - 11 mixed questions (explanation, arithmetic, logic with and without thinking, Python, history, summary, translation, answering from given material, a fictional person), IQ2_XS against NVFP4 with the same configuration: the grades differed only on the Python question. NVFP4 stripped non-ASCII characters, so its palindrome check called "あいう" a palindrome. IQ2_XS handled Japanese correctly. **Not measured:** cold starts (the model files were in the page cache), the standard benchmark (`strata bench run`), AVX2/AVX-512 builds of 0.2.16 on this host (it has only AVX), and why the default KV prefills slower than int8 KV here. Per-request data (`results.json`), the tool's README and the engine log excerpts are available for all eight runs. (AI-assisted; numbers taken from the tool's output and the engine logs.)
No site
Links install, modelos, releases.