#1986 Add community benchmark: Quadro RTX 8000 48 GB, UD-Q4_K_XL, 131K (v0.…

Description

# Community benchmark: Quadro RTX 8000 48 GB, Xeon E5-2687W 0, Unsloth UD-Q4_K_XL, 131K context (v0.1.42)

Results only — one new folder under `bench/results/`, no engine, server or default changes.

**Measured on 2026-10-11 by [aspineux](https://github.com/aspineux).**

## Hardware

- **GPU** Quadro RTX 8000, 49,152 MiB (Turing, sm_75) — **power-limited to 200 W against a
  260 W default**, observed peak 205-224 W, 87-98% utilisation
- **PCIe** gen3 x16 at max; engine's startup probe reports 11.5 GB/s host-to-device
- **CPU** Intel Xeon E5-2687W 0 @ 3.10 GHz, 8c/16t — **no AVX2 and no AVX-512**
  (`-march=native` resolves to `sandybridge`; engine `BUILD.json` says `isa_floor: "avx"`), so
  the CPU expert pool runs the AVX1 path
- **RAM** 125.72 GiB, no hugepages · **OS** Ubuntu 24.04.4, kernel 6.8.0-146-generic
- **Driver** 580.178.04, engine built against CUDA 12.2
- **Strata** tag `v0.1.42`, commit `61b3fb5d`, source build (`source: local`, `archs: [75]`)

## Model and configuration tested

Unsloth **Qwen3.8-Flash-Next UD-Q4_K_XL** (4 GGUF shards, ~111 GB), native-experts pack,
`--expert-profile data/expert-profile.bin`, MTP draft head, `--max-context 131072`, `--kv int8`,
`--kv-resident 32768`, `--resident-budget-gib 71`, `--expert-cache auto`, `--prefill auto`,
`--spec 4`, `--spec-min-p 0.70`, `--pcie-frac 0.32`, `--vram-reserve-mib 700`, `--vision`.
One slot (no `--batch`). Server on `127.0.0.1:8080`, no API key.

Engine at startup: `expert cache 13099 slots, 38.26 GiB of VRAM`, `R4 hit path ON`,
`resident 33.47 GiB, pinned 33.47 GiB in 37 s`, `335 MiB of VRAM free with everything loaded`.

## What was tested
# Community benchmark: Quadro RTX 8000 48 GB, Xeon E5-2687W 0, Unsloth UD-Q4_K_XL, 131K context (v0.1.42)

Results only — one new folder under `bench/results/`, no engine, server or default changes.

**Measured on 2026-10-11 by [aspineux](https://github.com/aspineux).**

## Hardware

- **GPU** Quadro RTX 8000, 49,152 MiB (Turing, sm_75) — **power-limited to 200 W against a
  260 W default**, observed peak 205-224 W, 87-98% utilisation
- **PCIe** gen3 x16 at max; engine's startup probe reports 11.5 GB/s host-to-device
- **CPU** Intel Xeon E5-2687W 0 @ 3.10 GHz, 8c/16t — **no AVX2 and no AVX-512**
  (`-march=native` resolves to `sandybridge`; engine `BUILD.json` says `isa_floor: "avx"`), so
  the CPU expert pool runs the AVX1 path
- **RAM** 125.72 GiB, no hugepages · **OS** Ubuntu 24.04.4, kernel 6.8.0-146-generic
- **Driver** 580.178.04, engine built against CUDA 12.2
- **Strata** tag `v0.1.42`, commit `61b3fb5d`, source build (`source: local`, `archs: [75]`)

## Model and configuration tested

Unsloth **Qwen3.8-Flash-Next UD-Q4_K_XL** (4 GGUF shards, ~111 GB), native-experts pack,
`--expert-profile data/expert-profile.bin`, MTP draft head, `--max-context 131072`, `--kv int8`,
`--kv-resident 32768`, `--resident-budget-gib 71`, `--expert-cache auto`, `--prefill auto`,
`--spec 4`, `--spec-min-p 0.70`, `--pcie-frac 0.32`, `--vram-reserve-mib 700`, `--vision`.
One slot (no `--batch`). Server on `127.0.0.1:8080`, no API key.

Engine at startup: `expert cache 13099 slots, 38.26 GiB of VRAM`, `R4 hit path ON`,
`resident 33.47 GiB, pinned 33.47 GiB in 37 s`, `335 MiB of VRAM free with everything loaded`.

## What was tested

All prompt and decode throughput comes from the engine's own `timings`, never from dividing
generated tokens by request time. Every prompt is byte-fresh with a nonce; **`reused 0` on all
twelve sweep runs**.

| Test | Result |
| --- | --- |
| Fresh prompt / decode, 3 runs each | 4K: **728.4** / **61.5** tok/s · 32K: **1,012.8** / **53.1** · 128K: **896.5** / **53.2** |
| Needle recall (`needle_bench`) | **6 of 6 found** — 32K (32,804 tok) and 128K (125,536 tok), depths 10/50/90 |
| Concurrency (`serve_load`, 1/2/4 clients) | 45.0 / 52.7 / 50.8 total tok/s, TTFT p50 0.72 / 3.08 / 7.99 s, 0 errors — flat throughput, one slot queues |
| **Soak, 74.9 min, 708 requests** | **708 ok, 0 failures, 0 timeouts.** Decode 73.95 → 75.0 tok/s (first → last decile), quarters 73.9 / 74.3 / 74.4 / 74.2. RAM +0.06 GiB. VRAM free **335 MiB at all 142 samples**. Peak 69 °C |
| **Cold engine restart** | `POST /unload` 6.79 s (GPU back to 1 MiB used), `POST /load` **65.7 s**. First touch: 32K 1,019.0 prompt / 59.6 decode tok/s, 128K 899.1 / 59.5 |

This answers the open row in `docs/TEST_REQUESTS.md` for a long soak on a resident setup.

## Limitations, stated plainly

1. **The measurement agent shared the engine.** This box serves the agent that ran the scripts
   (`PI_PROVIDER=strata`, `PI_MODEL=qwen3.8-flash-next-unsloth-ud-q4_k_xl`), so client and
   subject are the same server. Controls: every measurement in a blocking foreground call, one
   at a time, plus a per-run check that the engine was idle before the request and that the
   newest `/metrics` record matches it. **0 of 9 warm sweep runs flagged contaminated.** The
   soak ran with no other traffic. Please weigh this accordingly.
2. **Warm expert cache for the main sweep.** All nine sweep runs share one long-lived session.
   Run 1 of each length is clearly slower (32K: 37.7 tok/s at 92.6% hit) than runs 2-3 (55.6
   median at ~98% hit); both are reported separately in the README.
3. **The cold arm is a cold *engine*, not a cold boot.** `POST /unload` + `POST /load` restarts
   the engine process (PID 27516 → 106524, GPU 48,180 → 1 MiB) but the shards stay in the OS
   page cache, so file reads are warm. 37 s of the 65.7 s is re-creating the page-locked,
   mapped complement. A reboot number is **not** measured.
4. **200 W power cap** — compare against uncapped Turing cards with that in mind.
5. **Not tested:** cold boot, `--batch` slots and true parallel decode, multi-GPU /
   `--layer-split`, `--calibrate`, vision requests (encoder configured, no image sent), low-RAM
   mode, contexts beyond 131,072, quality benchmarks, and any A/B against another build.

## Two observations, as questions rather than findings

- **`335 MiB of VRAM free with everything loaded`** with `--expert-cache auto` and
  `--vram-reserve-mib 700`. `docs/DETAILS.md` names this line as the tell for a cache that leaves
  no free VRAM and cost 7x decode elsewhere. Here decode is healthy (53-61 tok/s on the sweep,
  74.2 tok/s steady across the soak) and the free figure **never moved in 75 minutes**, so I am
  **not** reporting a degradation — but it sits inside the margin the docs warn about. Would a
  `--vram-reserve-mib` sweep say the auto sizer is leaving too little on a 48 GB card?
- **A freshly restarted engine beat the first run inside the long session.** First touch after
  the cold restart: 59.6 / 59.5 decode tok/s at 32K / 128K with 98.9% / 99.3% expert hit. First
  run of each length in the long session: 37.7 / 44.8 tok/s at 92.6% / 96.2% hit. Both are
  first-touch-for-that-prompt. **n = 1 per cell and prompt lengths differ slightly**, so this is
  a question: does the resident complement settle differently after ~1,400 mixed requests than
  after a clean profile-driven start?

## Reproducing

`README.md` in the folder lists every command, the exact engine arguments, and the field names
and units of each JSON file. Folder is 448 KB — no model files, no packs, no API keys, and the
logs contain token counts and timings only, no prompt or answer text.