#6389 [Benchmark] 5 x Quadro RTX 8000 (sm_75), DeepSeek-V4.1-Flash IST 2.25 bpw: decode 13.5 to 29.3 tok/s by expert placement and --threads 1
open · @Maziar123 · 1 コメント · GitHub で見る
BenchmarksServer & APIMulti-GPUNVIDIA / CUDAModels & quantsLinux
本文
### What did you measure, or what went wrong For when DeepSeek comes back (#6072), and as a companion to #6298 and #5447: the same Turing card, five of them. On this box, decode depends mostly on where the experts live: on the cards or on the CPU lane. ### Hardware and software - 5 x Quadro RTX 8000 48 GB, PCIe Gen3. Threadripper PRO 9975WX (32 cores), 503 GiB RAM, Arch Linux (kernel 7.2), NVIDIA driver 615.71.09. - `strata-dsv41` 0.1.54, built from v0.2.8 (870aedd) with `-DCMAKE_CUDA_ARCHITECTURES=75 -DSTRATA_BUILD_DSV41=ON`. It runs in the `0.2.8-cuda13` image. Pack ISTA-DAS IST 2.25 bpw, `--max-context 262144`, `--layer-split auto`, `--cpu-threads 0` (32 threads), prompt chunk 4096. - Each figure is one request from an OpenAI-compatible client at top reasoning effort, after load, warm-up and a 1+1 check. Rates are the engine's own per-request line. From the fifth run on, the client's own count was checked and agreed within 0.01 tok/s. Read figures come only from cold reads, apart from the client's fixed 6,705-token prefix. The `32k` request is a 39,3xx-token prompt; the `128k` request has 148,6xx tokens. ### Results (output tok/s; read tok/s in brackets) | Layout | 32k | 128k | ~9.7K prompt | |---|---|---|---| | 4 cards shared with two other engines, `--reserve-mib 12288`: 9,884 experts on the CPU lane, DSpark not loaded | 13.45 (520) | 12.80 (734) | 12.82 (214) | | same, `--threads 1` | **16.16** (592) | **15.43** (767) | **15.47** (209) | | all 5 cards alone, `--reserve-mib 600`: 0 experts on the CPU lane, DSpark loads | 26.74 mean of 4 runs (634) | 23.93 (864) | 23.71 mean of 4 runs (288) | | same, `--threads 1` | **29.31** mean of 2 | — | **28.87** mean of 2 | | all 5 cards alone, `--dspark none` | 21.68 | — | 21.71 | - `--reserve-mib 600` puts every routed expert on the cards (CPU lane 9,884 -> 0), which gives about 1.9x decode. It also lets DSpark load: the engine's auto rule no longer refuses it. The drafter adds about +3 to +4 tok/s on top, and 75-85 % of its drafts were kept. The drafter's layer takes about 7.4 GiB on the first card. - `--threads 1` (parallel forward) adds +2.6 to +5.2 tok/s on both layouts and at every context we tried. The answer text changes with it, likely from a different summation order. - No effect: CPU pinning (`--cpuset-cpus`). The 32k answer was byte-identical with and without it. - `--experts 5`, one run each: +0.40 tok/s at 32k, which we count as no change, and +1.62 on the short prompt. Its answer quality was not checked. Not measured: newer engines than 0.1.54 on these cards (no sm_75 DeepSeek package exists), and concurrent requests. ### Reproduction status Measured locally
関連リンク
インストール・モデル・リリースへのサイト内リンク。