#2585 Report: UD-IQ4_XS and IQ3_XXS on RTX 3090 + Haswell-EP Xeon (AVX2, no AVX-512), NVMe vs HDD

Description

Thanks for Strata. You asked for UD-IQ4_XS numbers on NVIDIA, so here are ours, plus IQ3_XXS for reference.

**Setup**
- HP Z440 workstation, Xeon E5-2683 v3 (14 cores, Haswell-EP, AVX2, no AVX-512), 62 GB DDR4-2133 ECC (4 channels)
- 1× RTX 3090 24 GB, PCIe 3.0 x16 (engine's probe: 11.2–11.9 GB/s), driver 595.58.03, CUDA 12.8
- Model files on a Samsung PM981 NVMe, ext4; Linux (Proxmox LXC, Debian 13)
- Engine 0.1.39 (6f32ec0), built from source via the UnbiasedStrata fork (engine sources identical to yours), ctx 65536, `--threads 14` unless noted (8 turned out best — see below)
- Card at 220 W with clocks locked at 1600 MHz (`nvidia-smi -lgc 1600,1600`) unless noted
- Benchmark: 2,847-token Polish prompt with a unique tag (no cache reuse), up to 600 output tokens, temperature 0.6, 1/3/5/8 concurrent requests (served one at a time), second of two repeats

**Results (single user unless noted)**

| Variant | aggregate tok/s | decode tok/s | TTFT | prefill tok/s | experts on GPU | experts in RAM | from files |
|---|---|---|---|---|---|---|---|
| **IQ3_XXS, `--threads 8`**, `--ple-io direct --resident-experts` | **39.8** | **56.5** | 3.0 s | 948 | 10,326 slots (16.75 GiB) | 26.6 GiB pinned | 0 |
| IQ3_XXS, same, `--threads 14` | 32.4 | 40.2 | 3.1 s | 930 | 10,326 slots (16.75 GiB) | 26.6 GiB pinned | 0 |
| IQ3_XXS, same, ctx 131072 | 28.9 | 38.0 | 3.1 s | 908 | 9,744 (15.79 GiB) | 27.5 GiB | 0 |
| IQ3_XXS, unlocked clocks (1920 MHz) | 34.3 | 41.9 | 3.0 s | 944 | — | — | — |
| **UD-IQ4_XS** (pack `--compat-bf16`), `--ple-io direct --resident-budget-gib 36` | **16.6** | **20.9** | 7.4 s | 387 | 6,911 (15.58 GiB) | 36 GiB of 39.85 | ~3.9 GiB (773 GB read over 27 min) |
| IQ3_XXS, "RTX 3060-like" (`--vram-reserve-mib 12288`, core 600 MHz, mem 5001 MHz on the 3090) | 14.0 | 17.4 | 5.5 s | 517 | 3,327 (5.43 GiB) | 37.9 GiB | 0 |

- UD-IQ4_XS: packs and runs fine on NVIDIA (Ampere). Tool calls and reasoning work. Decode expert-cache hit rate 84.4 %. Aggregate stays ~14 tok/s at 3/5/8 users (one request at a time).
- Long prompts (47k tokens, single request): IQ3_XXS TTFT 28.9 s (prefill 1,628 tok/s), decode 22.6 tok/s; UD-IQ4_XS TTFT 60 s (prefill 784 tok/s), decode 12.2 tok/s. IQ3_XXS at 118k tokens (ctx 131072): TTFT 75 s, decode 38.7 tok/s.
- MTP draft acceptance on Polish text: ~41–45 %; on tool calls / numbers: ~80 %.
- On this box the CPU side is the limit: 1920 vs 1600 MHz on the GPU gives only ~6 %.
- **Thread count matters a lot** (IQ3_XXS, 64k, single user, aggregate / decode): 4 threads 28.0 / 34.4, **8 threads 39.8 / 56.5**, 14 threads (= physical cores) 32.4 / 40.2, 28 threads (Hyper-Threading) 29.5 / 37.3 tok/s. 28 threads were also 7–22 % slower than 14 on 128k, UD-IQ4_XS and the 3060-like run. With 8 threads: 3 users 39.2, 8 users 39.5 tok/s aggregate; CPU package 25 W idle, 62–64 W under load (RAPL). Looks memory-bandwidth bound (4 × DDR4-2133): beyond ~8 workers the expert pool only adds contention. Maybe worth a hint in the docs that the default (all physical cores) is not always best on older many-core Xeons.

Full tables (all variants, power draw, long documents, thread counts) and a comparison with vLLM on 4× RTX 3090: https://www.local-ai.company/geeks-only.html#strata-engine — benchmark page: https://www.local-ai.company/benchmark.html

**HDD confirmation of #605**
Before the NVMe drive, the model sat on a 7200 rpm HDD with `--ple-io ram` + `--mmap-experts` (62 GB RAM, so experts could not be resident): 0.13 tok/s, and the default 60 s watchdog killed the engine on the first request (stall report: threads in state D, millions of major faults). `STRATA_WATCHDOG_S=900` let it finish. The warning in the docs is right — an SSD is effectively required when RAM cannot hold the experts.