#402 Benchmark contribution: RTX PRO 4500 Blackwell (32 GB) + Threadripper PRO 3975WX, 512 GB RAM — engine 0.1.31, UD-Q4_K_XL / UD-IQ4_XS / IQ3_S at 8K-128K

Description

Benchmark report from a 32 GB card with a large-RAM host, on engine 0.1.31 (source build, commit `9259cad`), three packs on the same binary and the same GPU: **Unsloth UD-Q4_K_XL** (imported per `docs/UNSLOTH_Q4.md`, no `experts.bin`, GGUF read in place), **Unsloth UD-IQ4_XS** and **ISTA-DASLab GSQ-RCO IQ3_S**. Posted as an issue rather than a PR, as earlier reports were; the full folder in the `docs/COMMUNITY_BENCHMARKS.md` format (README, per-run JSON, server configs, engine logs, BUILD.json, needles.json) is ready and I can attach it or turn it into a PR if you prefer.

## Hardware and software

- NVIDIA RTX PRO 4500 Blackwell, 32 GB, 200 W default limit, PCIe Gen 4 x16 slot; engine's startup probe 28.2-28.3 GB/s host-to-device. Single GPU (`CUDA_VISIBLE_DEVICES` to its UUID); a second card drives the desktop.
- AMD Threadripper PRO 3975WX, 32 cores / 64 threads, **AVX2 only (no AVX-512)**; the engine chose AVX2 and 31 expert-pool workers.
- 512 GB DDR4 ECC, 8 channels; models on a 4 TB NVMe.
- Omarchy (Arch-based) Linux, kernel 7.2.5, NVIDIA driver 615.71.09, CUDA 13.3 toolkit, GCC 15 host compiler, `CMAKE_CUDA_ARCHITECTURES=120`, Python 3.13.
- Background: a hardware sampler (~2 % of one core) and the desktop compositor on the other card; nothing else on the test GPU.

## Configuration

Same arguments for every pack, only `--pack`/`--native`/`--mtp` and `--max-context` change:
`--resident-budget-gib 100 --expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp mtp/rt --kv int8 --vram-reserve-mib 1536 --serve`, one server process per context limit (8,192 for short/codegen, 81,920 for the 64K prompt, 139,264 for the 128K prompt). With the 100 GiB budget every expert of UD-Q4_K_XL stayed resident (7,982 in VRAM + 16,594 page-locked in RAM at 8K context; 7,614 / 7,329 in VRAM at 64K / 128K), and the engine reported no file reads. Greedy decoding (temperature 0, top_k 1), thinking off, output cap 256-1,024 tokens, 1 warm-up + 3 measured runs (short, codegen) or 1 + 2 (64K, 128K). Prompts: a 2,700-token code review, a 370-token code generation, and the same two padded to 63,997 and 127,999 tokens.

## Results (median of the measured runs)

Decode tokens/s:

| Prompt | UD-Q4_K_XL | UD-IQ4_XS | IQ3_S |
|---|---:|---:|---:|
| short (2,700 tok) | 61.5 | 82.0 | 106.3 |
| codegen (370 tok) | 72.4 | 98.0 | 123.8 |
| long 64K | 72.1 | 95.8 | 120.3 |
| long 128K | 70.2 | 91.4 | 116.5 |

Prefill tokens/s and time to first token:

| Prompt | UD-Q4_K_XL | UD-IQ4_XS | IQ3_S |
|---|---:|---:|---:|
| short | 1,149 (2.35 s) | 1,533 (1.76 s) | 1,855 (1.46 s) |
| codegen | 330 (1.12 s) | 484 (0.76 s) | 667 (0.55 s) |
| long 64K | 2,454 (26.1 s) | 3,053 (21.0 s) | 3,281 (19.5 s) |
| long 128K | 2,351 (54.4 s) | 2,893 (44.2 s) | 3,102 (41.3 s) |

Peak VRAM 31.2-31.4 GB for every pack and context. Peak host RAM in use (MemTotal - MemAvailable): 101-104 GiB for UD-Q4_K_XL, 85-88 GiB for UD-IQ4_XS, 74-78 GiB for IQ3_S. GPU power 140-185 W (UD-Q4_K_XL) and 181-195 W (IQ3_S). Decode expert-cache hit rate 81-94 % (UD-Q4_K_XL), lowest on the first 128K request.

`STRATA_KQ256=1` on UD-Q4_K_XL: short 67.6 vs 61.5 tok/s, codegen 71.7 vs 72.4, prefill unchanged. Within run-to-run spread, consistent with the note in `docs/UNSLOTH_Q4.md`.

Recall: `tools/needle_bench.py --lengths 32k,128k --depths 10,50,90` on the UD-Q4_K_XL 139,264-context server found the planted word in all six cases (14-52 s per check).

## Observations

- On this CPU the Q4_K/Q5_K/Q5_1 experts of UD-Q4_K_XL go through ggml's per-token path, which explains its decode at roughly two-thirds of the IQ packs; its long-prompt prefill is nonetheless the fastest we have measured on this card relative to our production stack (a 128K prompt in 54 s).
- UD-IQ4_XS imports and runs on the unpatched 0.1.31 (on 0.1.18 it needed a local patch for the Q8_0 down experts and the three-shard layout).
- `docs/UNSLOTH_Q4.md`'s serve example omits `--serve`; the 0.1.31 binary exits asking for `--tokens` without it. `--threads` from older recipes no longer exists.
- The 100 GiB resident budget is what makes the "no file reads" case possible; the doc's 40 GiB example is sized for a 64 GB PC.

Limitations: single GPU, one request at a time, greedy decoding, two runs for the long prompts, no quality suite beyond the needle check.

Measurements and this write-up were assembled with Claude Code on the owner's machine and checked by the owner.