#6354 [Benchmark] RTX 3060 12 GB on an AVX-only Xeon E5-2630 v2 (no AVX2/FMA): 0.2.16 (engine 0.1.62) against the 0.2.0 report, IQ2_XS and NVFP4

open · @Norhiro · 0 commentaires · Sur GitHub

BenchmarksServer & APIMulti-GPUNVIDIA / CUDAModels & quantsDocumentation

Description

A re-run of my 2026-10-20 report (PR #2793, 0.2.0-beta) on 0.2.16, which now labels this build `EXPERIMENTAL older-CPU build (ggml-cpu for avx)`. Same machine, same PCIe slot, same IQ2_XS files, same tool (`tools/strata_bench.py --long`, 3 runs per test). I also ran the NVFP4-GPTQ pack.

**Summary**
- 0.2.16 builds from source with `STRATA_ISA_FLOOR=avx` and runs on this CPU. No SIGILL. At start-up the engine prints `this CPU has no AVX2, so the CPU's experts run on ggml-cpu's kernels - expect it to be slow` and `CPU expert kernels: AVX1 (older-CPU build)`.
- With the 0.2.0 report's exact configuration, 0.2.16 is a little faster than 0.2.0 on every test: short chat 25.8 → 27.8 tok/s, prompt reading +21% at 4K and +3-4% at 16K/32K (32K TTFT 32.06 → 30.98 s).
- NVFP4 fits in 12 GB, but on this CPU it is about 2.3-2.9x slower than IQ2_XS with the same configuration: decode 18.1 against 31.8 tok/s, 32K prompt 253 against 724 tok/s. NVFP4 gets fewer expert slots (935 against 3,263), and the rest are computed on the AVX1 kernels.
- With my daily configuration, IQ2_XS reads prompts about 30% slower than with the 0.2.0 configuration (32K: 724 against 1,037 tok/s). Changing one setting at a time, almost all of it is the KV type: without `--kv int8`, 16K/32K prompts read 21-23% slower, although the expert cache and chunk shrink less than with a 65,536 context and int8 KV, which costs only 1%. Not binding to one NUMA node gives +24% decode and leaves prompt reading unchanged.

**Hardware and software**
- GPU: RTX 3060 12 GB, PCIe Gen3 x8 (the slot of the 0.2.0 report), driver 615.78.08, CUDA 13 (pip wheels).
- CPU: 2x Xeon E5-2630 v2 (Ivy Bridge-EP: AVX and F16C, no AVX2/FMA), 2 NUMA nodes; 251 GiB DDR3. Models on a SATA SSD.
- OS: Fedora 45, kernel 7.2.15.
- Strata 0.2.16 (`64524fb9`), engine 0.1.62, source build: `cmake -S engine -DSTRATA_ENABLE_CUDA=ON -DSTRATA_ISA_FLOOR=avx -DCMAKE_CUDA_ARCHITECTURES="86;120"`, gcc 15. Server from `terminal/serve/server.py`, Python 3.15.
- Models: IQ2_XS from ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF `ed59f92` (the same files and pack as the 0.2.0 report). NVFP4-GPTQ pack (`--native-dense-gguf`, `--embd-gguf`, and `--ple-gguf` from UD-Q4_K_XL part 2; the tool names it `nvfp4-UD-Q4_K_XL`).
- Nothing else used the GPU (40 MiB before each run).

**A. The 0.2.0 report's configuration** (40,960 context, `--kv int8`, expert cache auto, prefill auto, MTP spec 4 / min-p 0.5, no vision, no environment variables, bound to NUMA node 1, model files in the page cache)

| IQ2_XS, median (range) | 0.2.0-beta, 2026-10-20 (warm) | 0.2.16, today |
|---|---:|---:|
| Short chat decode, tok/s | 25.8 (22.0-29.6) | 27.8 (22.4-28.9) |
| Short chat TTFT, s | 1.68 | 1.27 |
| 4K prompt, tok/s (TTFT s) | 633.1 (6.39) | 765.0 (5.30) |
| 16K prompt, tok/s (TTFT s) | 983.2 (16.39) | 1024.9 (15.71) |
| 32K prompt, tok/s (TTFT s) | 1002.3 (32.06) | 1036.8 (30.98) |
| Expert cache | - | 4,126 slots, 5.54 GiB |

The agent-session test changed between the two tool versions (the last turn is 14.6K tokens in the 0.2.0 run and 9.0K now), so I leave it out of this comparison. Today: cache hit 71.0%, follow-up turn TTFT 3.60 s, decode 42.0 tok/s.

**B. My daily configuration, IQ2_XS against NVFP4** (65,536 context, default KV, `--vision` with the CPU picture reader, `STRATA_PREFILL_AUTO_MAX=16384`, `STRATA_PREFILL_CPU_SHARE=auto`, no NUMA binding; only the model differs)

| median (range) | IQ2_XS | NVFP4-GPTQ |
|---|---:|---:|
| Short chat decode, tok/s | 31.8 (29.9-34.3) | 18.1 (15.5-19.9) |
| Short chat TTFT, s | 1.19 | 2.34 |
| 4K prompt, tok/s (TTFT s) | 560.3 (7.22) | 258.1 (15.64) |
| 16K prompt, tok/s (TTFT s) | 682.6 (23.58) | 254.4 (63.20) |
| 32K prompt, tok/s (TTFT s) | 723.9 (44.35) | 252.5 (127.01) |
| Agent follow-up turns: TTFT s / decode tok/s | 3.80 / 40.0 | 5.90 / 16.2 |
| Agent cache hit | 71.0% | 70.9% |
| Expert cache; prompt chunk | 3,263 slots (4.39 GiB); 7,680 | 935 slots (2.41 GiB); 3,328 |
| VRAM free with everything loaded | 699 MiB | 675 MiB |

The prompt-reading ranges are within 1-2% for every size.

**C. From A towards B, one setting at a time** (IQ2_XS, `strata_bench.py --long`, 3 runs each; every row is A with one change)

| Change from A | Chat decode tok/s | 4K tok/s | 16K tok/s | 32K tok/s | Expert cache; prompt chunk |
|---|---:|---:|---:|---:|---|
| none (A) | 27.8 | 765.0 | 1024.9 | 1036.8 | 4,126; 9,984 |
| `--max-context 65536` | 26.8 | 762.6 | 1013.6 | 1027.2 | 3,855; 9,216 |
| default KV (no `--kv int8`) | 26.6 | **658.3** | **804.6** | **794.6** | 3,757; 8,960 |
| `--vision` (CPU picture reader) | 27.9 | 763.5 | 1022.1 | 1034.8 | 4,126; 9,984 |
| `STRATA_PREFILL_AUTO_MAX=16384` + `STRATA_PREFILL_CPU_SHARE=auto` | 27.8 | **621.8** | 1023.5 | 1037.1 | 4,126; 9,984 |
| no NUMA binding | **34.6** | 786.6 | 1011.8 | 1025.5 | 4,126; 9,984 |
| all of the above (B) | 31.8 | 560.3 | 682.6 | 723.9 | 3,263; 7,680 |

- **The KV type is most of the gap.** Without `--kv int8`, 16K/32K prompts read 21-23% slower. It is not only the smaller cache: the 65,536 context with int8 KV leaves fewer slots (3,855) and a smaller chunk (9,216) than the 40,960 context with the default KV (3,757; 8,960) does, yet costs only 1%. I have not looked at why the default KV is slower to prefill here.
- The two environment variables cost 19% at 4K only; I did not split them.
- Without NUMA binding, short-chat decode is 24% faster and prompt reading is the same.
- B loses a little more than the single changes add up to (30% at 32K, against 23% + 1%). With both the default KV and the 65,536 context, the cache shrinks to 3,263 slots and the chunk to 7,680 tokens; I did not run that pair alone.

**Our own answer checks (not part of the tool, not a scored benchmark)**
- RAG over Japanese railway tariff tables (our own pipeline; 16 questions, no thinking), NVFP4 daily config: 16/16 answered with the same numbers as on an RTX 5060 Ti machine; about 1.6x the time per question.
- 11 mixed questions (explanation, arithmetic, logic with and without thinking, Python, history, summary, translation, answering from given material, a fictional person), IQ2_XS against NVFP4 with the same configuration: the grades differed only on the Python question. NVFP4 stripped non-ASCII characters, so its palindrome check called "あいう" a palindrome. IQ2_XS handled Japanese correctly.

**Not measured:** cold starts (the model files were in the page cache), the standard benchmark (`strata bench run`), AVX2/AVX-512 builds of 0.2.16 on this host (it has only AVX), and why the default KV prefills slower than int8 KV here.

Per-request data (`results.json`), the tool's README and the engine log excerpts are available for all eight runs.

(AI-assisted; numbers taken from the tool's output and the engine logs.)

Sur le site

Liens install, modèles, releases.