#5624 [Engine] DeepSeek-V4.1-Flash 2.25 bpw on one RTX 4090: two settings worth productizing, the DS4_* knobs, and one start-up peak we cannot explain
closed · @wangminle · 1 Kommentare · Auf GitHub
BenchmarksAMD / HIPNVIDIA / CUDADocumentationWindows
Beschreibung
Rig: RTX 4090 48 GB (sm_89), AMD EPYC 9B14 (96C/192T), 282 GiB RAM, Ubuntu 24.04, driver 595.99.02. Model: DeepSeek-V4.1-Flash IST-DAS 2.25 bpw (274.8 GB), engine `strata-dsv41 0.1.58`. This is the tuning round behind issue #5589 (the follow-up comments there hold the raw protocol and all arms). Two settings cleared our >5% bar; the rest of the sweep (CPU threads, prefill chunk, borrow, pinned RAM, async window, Engram cache/threads, parallel requests, `--vram-experts`) did not. Everything below is one machine, same engine build, single-variable A/Bs with 2 prefill warm-ups, 3 decode warm-ups and 6 measured requests. ## 1. Two settings worth productizing | setting | before | after | effect (engine aggregate) | |---|---|---|---| | `--dspark-conf` | 0.65 (pack default) | **0.45** | decode 67.8 -> **74.7 / 74.2 / 73.8** tok/s (+7-10%), reproduced three times | | `DS4_FAT_ROWS` | 0 (off) | **64** | prefill 1,590 -> **1,678** tok/s (+5.5% vs the contemporaneous control), decode unchanged | Together: prefill 1,517 -> **1,687 tok/s** (+11.2%), decode 67.8 -> **72.4-74.7**, and a fixed 10-item sanity probe stays at 10/10. The cost is one environment variable and a ~77 MB VRAM buffer for `FAT_ROWS`. Suggestion: have `strata tune` suggest these two for the deepseek41 family on a 48 GB card (or at least document them in the DeepSeek launch guide). `DS4_FAT_ROWS` in particular looks like a good default candidate: the source describes it as the same products with a different summation order, and our sweep found 32/64/128 all within noise of each other. ## 2. The `DS4_*` knobs are only discoverable from the source We tested these by reading `src/dsv41/*`; none are in the docs or in `--help` beyond a bare flag name. Measured here (RTX 4090, 96 threads, 2.25 bpw pack): - `DS4_ASYNC_T=4` decode -10%; `16` ties the default 8 (and the source notes a logits window dependence). - `DS4_COPY_PIN=1`: `cudaHostRegister` pins all 116.3 GB successfully in 9.3 s despite `ulimit -l` = 35.35 GiB, but prefill does not improve and free VRAM drops 565 -> 347 MiB. - `DS4_STREAM=1` + `STRATA_DS4_PIN_BY_NEED=1` + `STRATA_DS4_PIN_GB=16`: the one-GPU path measures `0 layers ... stream` (the CPU lane is too fast here) and turns itself off; pinning-by-need works (13.6 GB) but is a no-op. - `DS4_ENGRAM_CACHE_MB=32768` and `DS4_ENGRAM_THREADS=32`: no measurable change (separate report: #5563). A short table of these knobs, with what they do and what they cost, would save users a lot of source reading. ## 3. One start-up peak we cannot explain (an engine question, not a tuning knob) During the sweep, the first start with `DS4_STREAM=1` + `STRATA_DS4_PIN_BY_NEED=1` + `STRATA_DS4_PIN_GB=16` placed **2,462/15,744 experts in VRAM** (40.17 GB, 737 MiB free; 13,282 experts / 128.9 GB in RAM) and measured **1,821 tok/s prefill** and **75.9 tok/s decode** - the best numbers of the whole round. The log then said `0 layers ... stream` and the path disabled itself; `lane-times.txt` recorded `lane off`. A repeat with the same settings placed 3,759 experts and measured 1,687 / 70.7, and the peak never came back. We wanted to know whether "fewer VRAM experts" was the cause, so we swept `--vram-experts` (the budget multiplier, default 1.0) with everything else fixed: | `--vram-experts` | experts in VRAM | VRAM free | prefill | decode | |---|---|---|---|---| | 1.00 | 3,759 | 565-688 MiB | 1,687 / 1,686 | 72.5 / 70.8 | | 0.85 | 3,138 | 6,329 MiB | 1,666 | 67.9 | | 0.75 | 2,720 | 10,181 MiB | 1,705 | 64.5 | | 0.65 | 2,305 | 14,049 MiB | 1,675 | 61.3 | Fewer VRAM experts is monotonically **slower** (the CPU lane's share rises 3.7% -> 5.1%, DSpark acceptance is flat), so the 2,462-expert placement itself is not the explanation. Question for the engine: could that one-off come from the start-up buffer layout - the streaming path reserving windows before deciding nothing streams, and the resources released after it disables itself? Is that a known transient, or is there a state we could deliberately reproduce (for example a documented way to reserve the window buffers without enabling the path)? We are not asking for a fix based on one run; we would like to know whether it is worth a profiling look. Full raw data: `ds41-2-25bpw-tuning-results-2026-11-08.md` on the reporter's machine, summarized in #5589.
Mehr auf der Site
Links zu Install, Modellen, Releases.