贡献 / #1834

#1834 Community benchmark: 2x RTX PRO 4500 Blackwell, Swift IQ3_XXS, engine 0.1.41 against 0.1.40.1 (single stream to 248k, --batch 8 sweep)

open · @qni-live · 0 评论 · 去 GitHub 看

BenchmarksMulti-GPUNVIDIA / CUDAModels & quantsWindowsLinux

说明

Adds one results folder, `bench/results/2026-10-10-community-2x-rtx-pro-4500-engine-0.1.41/`, and nothing else. No engine changes.

- Hardware: 2x RTX PRO 4500 Blackwell 32 GB (GB203, sm_120), Threadripper 7960X, 61 GiB RAM, Linux, driver 595.91.07, CUDA 13.x. Source builds of v0.1.41 (fb58e0d) and v0.1.40.1.
- Model: Swift 1.5 IQ3_XXS, layer split 25 over both cards, 262,144 context, `--pipeline-windows 2`, German draft vocabulary, memory clock 14,000 MHz in every sample.
- Single stream at 4k, 32k, 128k and 248k prompt tokens (3 runs each, 0.1.41 against 0.1.40.1, order ABA): prompt reading is the same (128k: 6,148 and 6,146 tok/s against 6,142); the decode difference is inside the run-to-run spread (up to 15 % at 32k).
- `--batch 8` client sweep (`tools/serve_load.py`, 1, 2, 4, 8 clients, 3 rounds, 256 tokens): `--batch-groups auto` gives 115 / 152 / 208 / 256 total tok/s, against 109 / 126 / 159 / 175 for groups 1. That is +46 % at 8 clients, inside the +32 % to +68 % of the release notes. First token at 8 clients 1.6 s, against 7.0 s for the same requests queued on a single stream. Eight slot sessions take 12.1 GiB of VRAM per card (71 % of the experts stay resident) and `--pipeline-windows` is off.
- Needle recall 6 of 6, tool calls 9 of 9 valid. `layer_split: auto` chose layer 24 and did not reorder the two identical cards (#1760 not reproduced here).
- Not tested: `--batch 4` in this folder, long prompts, tool calls or vision in batch mode, Windows, cards of different speed.

Written with AI assistance (Claude); numbers reviewed by me.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

本站相关内容

相关页面的快捷入口。