#5928 [Benchmark] CMP 170HX (sm_80, modified 64 GB) / Qwen3.8-Flash-Next UD-Q4_K_XL / 256K / 0.2.13.1
closed · @30crows · 1 Kommentare · Auf GitHub
BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quantsDocumentationLinux
Beschreibung
### What did you measure, or what went wrong
A report from an sm_80 card ([DETAILS.md](https://github.com/Niko1221/Strata/blob/main/docs/DETAILS.md) notes that the maintainers have none): Qwen3.8-Flash-Next, Unsloth UD-Q4_K_XL, 256K context, on a modified NVIDIA CMP 170HX. Measured on 2026-11-13 with three tools on one source checkout and one run config: the community script from `bench/results/2026-09-30-community-rtx-5090/` (prompts up to 250K), the draft standard benchmark (`strata bench run`, standard-v1) and the community suite (`tools/strata_bench.py --long`). Speeds are the engine's own timings.
**Hardware**
- GPU: NVIDIA CMP 170HX (GA100, sm_80), modified: 64 GB HBM2e (stock: 8 GB), PCIe 2.0 x16 (6.2 GB/s host-to-device)
- CPU: 2x Xeon E5-2640 v3 (16 cores, AVX2, no AVX-512; two NUMA nodes, the GPU on node 1). RAM: 251 GiB DDR4 (speed not read). Model files on 3 SATA SSDs (RAID0).
- OS: Gentoo Linux, kernel 6.18.63. NVIDIA driver 610.43.03. CPU governor schedutil, turbo on. Nothing else on the GPU during the runs.
**Software and configuration**
- Strata v0.2.13.1-160, source checkout `13eedf55`, engine 0.1.59. Setup installed `0.1.59-linux-x86_64-cuda-cuda12` from the signed feed by itself (CUDA 12.9, real sm_80 code; the cuda13 packages have none for sm_80). No local build was needed.
- Model: `unsloth/Qwen3.8-Flash-Next-GGUF@38bb39ee`, UD-Q4_K_XL, setup's pack, MTP draft layer, pictures on.
- The run config setup wrote: `--max-context 262144 --kv int8` (KV streaming, 32,768 cells in VRAM), `--ple-io ram`, `--spec 4`, expert cache auto (about 18,950 of 24,576 experts, 55 GiB of VRAM; every expert in RAM), 15 pool workers.
- `./setup.sh --calibrate` chose `--pcie-frac 0.55 --spec-min-p 0.70`. The gain is the draft floor (84.3 -> 89.3 tok/s median). The PCIe share was within noise from 0 to 0.9 (78.6-82.4 tok/s; 1.0: 71.8), and 4 to 15 pool workers were within 1%.
- Changed from setup's config (custom track): the config's `env` turns off the engine defaults that change answers: `STRATA_ROUTE_TAIL_SKIP=0 STRATA_PROMPT_ATTN_IMMA=0 STRATA_ROUTE_PRIOR=0 STRATA_NV_KQ_DEFAULT=0 STRATA_MMQ_KQ=0 STRATA_PREFILL_CPU_SHARE=0 STRATA_GDN_TC=0 STRATA_GDN_CONVL2=0`. So these are not the default configuration's numbers.
**1. Community script from 2026-09-30** (fresh prompts, temperature 0, thinking off, 256 new tokens, 3 runs per size; median and range)
| Prompt tokens | Prompt tok/s | Decode tok/s | TTFT s |
|---:|---:|---:|---:|
| 4,096 | 808 (797-815) | 73.0 (55.4-74.7) | 5.1 |
| 32,768 | 2,462 (2,421-2,518) | 77.7 (77.1-82.4) | 13.4 |
| 128,000 | 2,711 (2,670-2,790) | 74.9 (74.6-77.8) | 47.4 |
| 250,000 | 2,594 (2,571-2,604) | 81.4 (72.8-81.7) | 96.7 |
Every run generated 256 tokens with 0 reused. Draft acceptance 54-66%, decode expert hit rate 98.5-99.9%.
**2. Standard benchmark, draft standard-v1** (`strata bench run --profile <run config> --track custom`, fresh engine): status complete, 21 of 21 required requests, 0 failed, 2 min 33 s after the load. Thinking on (xhigh), temperature 1.0.
| Case | Prompt tok/s | Decode tok/s | TTFT s |
|---|---:|---:|---:|
| short (~80 tokens) | n/a | 73.6 (64.0-78.2) | 1.25 |
| prompt-4k | 823 (815-874) | 71.8 (70.1-79.9) | 5.01 |
| prompt-16k | 2,151 (2,122-2,158) | 72.9 (60.8-77.2) | 7.68 |
Sanity checks 12 of 12 (instruction 4/4, retrieval 4/4, tool calls 4/4). 2 of 3 4K and 3 of 3 16K requests reached the 512-token cap.
**3. Community suite** (`tools/strata_bench.py --long`, fresh engine, 3 runs)
| Test | Median (range) |
|---|---|
| chat decode | 66.3 tok/s (38.0-109.4, n=9) |
| 4K prompt | 788 tok/s (786-797) |
| 16K prompt | 2,257 tok/s (2,205-2,290) |
| 32K prompt | 2,553 tok/s (2,549-2,569) |
| agent session decode | 73.1 tok/s (52.2-84.7, n=15); cache hit 69.9% over the follow-up turns, TTFT about 5 s |
**What the numbers show**
- Decode stays at 72-82 tok/s from 4K to 250K: about 99.7% of the experts a token routes to are in the VRAM cache, and the decode share sent over PCIe is about 0.
- Short prompts are bound by the PCIe 2.0 link (each chunk streams the experts that are not in VRAM): about 800 tok/s at 4K, 2,460-2,710 tok/s from 32K.
- During long prompts the GPU sits at its 250 W limit ("SW Power Cap" in about 60% of the samples), with an SM clock of about 1,380 of 1,410 MHz. There was no thermal slowdown and no CPU thermal throttling (hottest CPU 77 C during calibration, 63-68 C in these runs).
- On a cold engine, the first three short chats decoded at 38-53 tok/s, and the same questions at 69-109 tok/s two rounds later.
**Not tested**
- The speed of the default configuration (the eight switches above left at their defaults) on this card.
- The sm_80 opt-in switches in DETAILS.md (from #3779), DFlash2, parallel requests, and a local CUDA 13 build.
- Answer quality beyond the 12 sanity checks.
Related: #3779, the earlier CMP 170HX measurements on a PCIe 2.0 x4 link. They are not directly comparable with these: there IQ3_S with every expert in VRAM, a 1,650 MHz clock ceiling and 0.2.1.4/0.2.2; here UD-Q4_K_XL with about 77% of the experts in VRAM, stock clocks and the answer-changing defaults off.
### Reproduction status
Measured locally
Mehr auf der Site
Links zu Install, Modellen, Releases.