#6389 [Benchmark] 5 x Quadro RTX 8000 (sm_75), DeepSeek-V4.1-Flash IST 2.25 bpw: decode 13.5 to 29.3 tok/s by expert placement and --threads 1

open · @Maziar123 · 1 コメント · GitHub で見る

BenchmarksServer & APIMulti-GPUNVIDIA / CUDAModels & quantsLinux

本文

### What did you measure, or what went wrong

For when DeepSeek comes back (#6072), and as a companion to #6298 and #5447: the same Turing card, five of them.
On this box, decode depends mostly on where the experts live: on the cards or on the CPU lane.

### Hardware and software

- 5 x Quadro RTX 8000 48 GB, PCIe Gen3. Threadripper PRO 9975WX (32 cores), 503 GiB RAM, Arch Linux (kernel 7.2),
  NVIDIA driver 615.71.09.
- `strata-dsv41` 0.1.54, built from v0.2.8 (870aedd) with `-DCMAKE_CUDA_ARCHITECTURES=75 -DSTRATA_BUILD_DSV41=ON`.
  It runs in the `0.2.8-cuda13` image. Pack ISTA-DAS IST 2.25 bpw, `--max-context 262144`, `--layer-split auto`,
  `--cpu-threads 0` (32 threads), prompt chunk 4096.
- Each figure is one request from an OpenAI-compatible client at top reasoning effort, after load, warm-up and a
  1+1 check. Rates are the engine's own per-request line. From the fifth run on, the client's own count was checked
  and agreed within 0.01 tok/s. Read figures come only from cold reads, apart from the client's fixed 6,705-token
  prefix. The `32k` request is a 39,3xx-token prompt; the `128k` request has 148,6xx tokens.

### Results (output tok/s; read tok/s in brackets)

| Layout | 32k | 128k | ~9.7K prompt |
|---|---|---|---|
| 4 cards shared with two other engines, `--reserve-mib 12288`: 9,884 experts on the CPU lane, DSpark not loaded | 13.45 (520) | 12.80 (734) | 12.82 (214) |
| same, `--threads 1` | **16.16** (592) | **15.43** (767) | **15.47** (209) |
| all 5 cards alone, `--reserve-mib 600`: 0 experts on the CPU lane, DSpark loads | 26.74 mean of 4 runs (634) | 23.93 (864) | 23.71 mean of 4 runs (288) |
| same, `--threads 1` | **29.31** mean of 2 | — | **28.87** mean of 2 |
| all 5 cards alone, `--dspark none` | 21.68 | — | 21.71 |

- `--reserve-mib 600` puts every routed expert on the cards (CPU lane 9,884 -> 0), which gives about 1.9x decode.
  It also lets DSpark load: the engine's auto rule no longer refuses it. The drafter adds about +3 to +4 tok/s on
  top, and 75-85 % of its drafts were kept. The drafter's layer takes about 7.4 GiB on the first card.
- `--threads 1` (parallel forward) adds +2.6 to +5.2 tok/s on both layouts and at every context we tried. The
  answer text changes with it, likely from a different summation order.
- No effect: CPU pinning (`--cpuset-cpus`). The 32k answer was byte-identical with and without it.
- `--experts 5`, one run each: +0.40 tok/s at 32k, which we count as no change, and +1.62 on the short prompt. Its
  answer quality was not checked.

Not measured: newer engines than 0.1.54 on these cards (no sm_75 DeepSeek package exists), and concurrent requests.

### Reproduction status

Measured locally

関連リンク

インストール・モデル・リリースへのサイト内リンク。