改动 / #2795

#2795 bench: RTX 4090 community benchmarks for UD-Q4_K_XL, IQ3_XXS and IQ3_S (262K, one machine)

open · @hongxianzhi · 0 评论 · 去 GitHub 看

BenchmarksSetup & installNVIDIA / CUDAModels & quantsDocumentationWindows

说明

Adds four folders under `bench/results/`: one run folder per quantization, plus a summary folder that compares the three. Results only — no engine, setup or docs changes.

## Hardware

RTX 4090 24 GiB (driver 610.88, PCIe 4.0 x16), i7-13700K (16 cores/24 threads, no AVX-512), 127.8 GiB RAM, Windows 10 (10.0.26200). Strata 0.1.44.1 prebuilt engine (`bd18453f…`, unchanged across the runs), project commit 296b59d2.

## What was tested

Qwen3.8-Flash-Next at 262,144 tokens of context, `--kv int8 --kv-resident 32768`, `--vision`, one request at a time (`settings.parallel` not set), each after `setup.py --calibrate` tuned it for this PC:

| Quantization | Experts in RAM | Tuned settings |
| --- | ---: | --- |
| IQ3_XXS (`ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF`) | 39.97 GiB | `--pcie-frac 0.35 --spec-min-p 0.70 --pool-workers 7 --adapt-swaps 80` |
| IQ3_S (same repo) | 46.84 GiB | `--pcie-frac 0.20 --spec-min-p 0.70 --pool-workers 7 --adapt-swaps 160` |
| Unsloth UD-Q4_K_XL (`unsloth/Qwen3.8-Flash-Next-GGUF`) | 71.73 GiB | `--pcie-frac 0.20 --spec-min-p 0.70 --pool-workers 7 --adapt-swaps 160` |

Measured with `python tools/strata_bench.py --runs 3 --clients 4 --long` against an already-running server; model load is not timed.

## Numbers (median of 3 runs)

| Quantization | chat decode tok/s | 4K prompt | 16K prompt | 32K prompt | agent decode | 4 clients total |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| IQ3_XXS | 109.2 | 2634.9 | 4622.9 | 4771.2 | 132.3 | 95.3 |
| IQ3_S | 92.2 | 2158.7 | 4161.7 | 4228.6 | 119.8 | 74.8 |
| UD-Q4_K_XL | 62.8 | 1310.9 | 2510.6 | 2498.1 | 80.7 | 47.8 |

IQ3_S is 9-22 % slower than IQ3_XXS at every length here, with 6.9 GiB more experts in RAM — consistent with the registry note in `setup.py` ("about 15% slower than IQ3_XXS and 7 GB more RAM"), which had not been measured on this hardware. Needle recall: found at 128k and at 258,266 prompt tokens, depths 10 % and 90 % (IQ3_XXS 4/4, IQ3_S 2/2).

## Limitations

Speed only: no quality, coding or long-conversation evaluation, and the two quality columns quoted in the summary come from `docs/QUANTS.md` on a different machine. One session, one GPU, warm expert cache, background load not recorded, model-load time excluded. The three use different calibrated settings, so these are not kernel-only differences. UD-Q4_K_XL was not needle- or vision-checked in this session.

本站相关内容

相关页面的快捷入口。