#5922 [Engine] DeepSeek V4.1 on a 12 GB card: --dspark on gives the drafter every VRAM slot and the prompt path falls to windows of 16; a cap on the drafter's VRAM experts makes it a net gain
closed · @midokino · 1 Kommentare · Auf GitHub
BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quantsWindowsLinux
Beschreibung
### Area
Engine
### What do you want to do?
Hey Niko, I'm new to this and I'm not that good at English so it might be awkward (actually I'm letting Claude write issue, too) but hope this helps you even a little.
>>>
I would like to use the DSpark drafter on a 12 GB card. Today, `--dspark auto` leaves it unloaded on this card, which is reasonable given what `--dspark on` does here. With a cap on how much VRAM the drafter's experts may take, it turned into a clear gain on this machine. I'm sharing the measurements and the small patch I used, in case they help.
I have run it this way every day since 2026-11-10 (three days so far, with an agent harness and chat), with no problems; draft acceptance stays around 86–87%. The code around the drafter's VRAM split is unchanged in `main` as of 2026-11-13, so the patch below still applies.
**Setup**
- RTX 5070 12 GB (sm_120)
- Ryzen 7 9700X, 64 GB DDR5-5600 (dual channel)
- One NVMe (Crucial P310 1 TB)
- Windows 11
- Engine: strata-dsv41 0.1.58 (`0fe4ae509`). I rebuilt it from source with CUDA 13.0, `DSV41_ARCH=120-real`.
- Model: DeepSeek-V4.1-Flash 1017, IST-DAS 2.25 bpw stock pack
- `--max-context 131072`
- 5 of 6 experts (faster mode)
- RAM store set by `DS4_STORE_GB`
**What happens with `--dspark on`**
The VRAM split gives the drafter's blocks their experts first, all of them (ds4_model.cu, around line 5446, "the drafter's experts first, all of them"). On 12 GB, that leaves the trunk's layers 0 slots:
```
ds4: CUDA0 keeps too few experts in VRAM (0 a layer) to lend the prompt path its room: prompts in windows of 16
```
The effect on this machine:
- Decode: code +30%, but Japanese prose −15 to −25%.
- Prompts:
- A 4.6K-token prompt took 546 s, against about 38 s without the drafter.
- A 30K-token prompt did not finish within the timeout.
- Requantizing the drafter (Q3_K, Q2_K, IQ2_XXS) did not change this, because a drafter expert is about the size of a trunk expert.
**The patch: `DS4_MTP_VRAM_GB`**
This environment variable caps the drafter's VRAM experts. The cap is split evenly over its blocks, and the rest are read like the trunk's (the CPU lane / the files). The trunk keeps the room it had without the drafter (205 of 230 experts at 0.5 GB). The diff is at the end.
**Measurements**
Conditions:
- Same machine, nothing else using the server.
- Four short prompts (Japanese prose, Japanese explanation, code, thinking), 2 rounds each, median.
- A 4.6K-token prompt and a 31.5K-token prompt.
- About 54 GB of RAM free before start.
- Unit: tok/s.
The cap I use is 0.5 GB. Its rows come first, against stock with DSpark off; the other cap values follow.
| Engine, DSpark, RAM store | code | think | JA explain | JA prose | 4.6K prefill | 30K prefill / decode |
|---|---|---|---|---|---|---|
| Stock 0.1.58, DSpark off, 38 GB | 7.8 | 10.1 | 11.8 | 13.8 | 120 | 401 / 8.5 |
| **DSpark on, `DS4_MTP_VRAM_GB=0.5`, 38 GB** | **10.0 (+28%)** | **11.3 (+12%)** | 11.4 | 13.6 | 120 | 306 / 9.2 |
| **DSpark on, `DS4_MTP_VRAM_GB=0.5`, 34 GB** | **9.4 (+21%)** | **11.5 (+14%)** | 10.7 | 12.3 | 118 | 397 / 8.9 |
| **DSpark on, `DS4_MTP_VRAM_GB=0.5`, 36 GB, store slots per layer (see below)** | **10.1 (+29%)** | **12.3 (+22%)** | 11.8 | 13.8 | 123 | 405 / 9.7 |
| DSpark on, `DS4_MTP_VRAM_GB=0`, 38 GB | 9.7 | 11.5 | 11.2 | 12.7 | 120 | 265 / 10.1 |
| DSpark on, `DS4_MTP_VRAM_GB=1`, 38 GB | 9.3 | 11.1 | 11.0 | 13.0 | 120 | 321 / 10.0 |
| DSpark on, `DS4_MTP_VRAM_GB=2`, 38 GB | 10.1 | 10.9 | 10.1 | 11.6 | (windows of 16 again: 1 slot a layer) | – |
| DSpark on, no cap (`--dspark on` as it is) | | | | | 4.6K prompt: 546 s (about 38 s without the drafter) | did not finish |
| Own build, DSpark off, 38 GB (same source as stock; checks the build) | 7.9 | 9.9 | 11.4 | 14.0 | 120 | 402 / 9.0 |
- The drafter pays off on code and thinking; Japanese prose and explanation stay about level.
- Draft acceptance was 80–87% throughout (`--dspark-conf 0.65`).
- The 30K prefill drop at a 38 GB store looks like the same edge I see without the drafter: a store of 41 GB or more gives 30K prefill of 223–238. My guess is that the drafter's experts, now read in place, push it over that edge. Lowering the store to 34 GB restores the prefill, but costs some decode.
**A second observation (less sure of the cause)**
`ExpertStore` sizes every slot to the largest layer's blob. In this mixed-bpw pack, blobs range from 9.16 to 11.37 MB (mean 9.74). I tried slots sized per layer: seven size classes, a chunk holds one class, and an expert takes a slot of its layer's class.
- The same `DS4_STORE_GB` then holds about 16% more experts.
- With DSpark at 0.5 GB and a 36 GB store, I get: prose 13.8, explanation 11.8, code 10.1, thinking 12.3, 4.6K prefill 123, 30K prefill / decode 405 / 9.7.
- That is the stock numbers or better on every row: code +29%, thinking +22%.
- Without DSpark, a 38 GB store of this kind gives 30K prefill 408 and decode 9.7.
I don't understand why this version avoids the 30K prefill edge:
- The unused tail of the uniform slots is never touched, so it should not cost physical RAM.
- The size-class store at 38 GB really uses more RAM (16.5 GB free after load) than the uniform one at 41 GB.
- Yet only the uniform one drops.
I'm mentioning it as a measurement, not a proposal. I can share that patch too if it's of interest.
**A third note: the drafter requantized to IQ2_XXS**
I also requantized the official 1017 drafter with `tools/dsv41/drafter_pack.py --requant` to IQ2_XXS (4.25 GB). Decode and acceptance stayed the same as the stock drafter. It leaves 1–2 GB more RAM free, and with it the 30K prefill no longer drops when free RAM gets tight: with the stock drafter, 30K prefill fell from about 420 to 240–300 tok/s on runs where free RAM was lower; with IQ2_XXS it held. That is the setup I use now (cap 0.5 GB, store 36 GB).
### What do you do today instead?
I run my own build with the two changes above, on branch `local-dspark-vram`. I would need to re-apply and rebuild them on every update. A built-in option, or `--dspark auto` sizing the drafter's VRAM share on small cards, would let 12 GB users get the drafter's gain.
### Patch (`DS4_MTP_VRAM_GB`, against 0fe4ae509)
```diff
--- a/src/dsv41/ds4_model.cu
+++ b/src/dsv41/ds4_model.cu
@@ -5444,10 +5444,23 @@ struct Model::Impl {
room = std::max(0.0, room - std::min(budget, all) * nm);
}
double budget_t = budget; // the drafter's experts first, all of them (every draft uses its blocks)
+ // DS4_MTP_VRAM_GB caps the drafter's VRAM experts, split evenly over its blocks; the rest stream
+ // like the trunk's. On a 12 GB card the drafter otherwise takes every slot and the prompt path falls to
+ // windows of 16
+ double mtp_cap = 1e300;
+ if (const char* v = std::getenv("DS4_MTP_VRAM_GB")) {
+ int n_m = 0;
+ for (const Layer& l : L)
+ if (l.grp == (int) gi && l.mtp) ++n_m;
+ if (n_m > 0) mtp_cap = std::max(0.0, std::atof(v)) * 1e9 / n_m;
+ }
for (Layer& l : L)
if (l.grp == (int) gi && l.mtp) {
- l.n_vram = std::min(l.n_exp, (int) std::floor(budget_t / (double) ucount(l)));
+ l.n_vram = std::min(l.n_exp, (int) std::floor(std::min(budget_t, mtp_cap) / (double) ucount(l)));
budget_t -= (double) l.n_vram * ucount(l);
+ if (mtp_cap < 1e300 && o.verbose)
+ std::fprintf(stderr, "ds4: DS4_MTP_VRAM_GB: drafter %s keeps %d of %d experts in VRAM\n", l.name.c_str(),
+ l.n_vram, l.n_exp);
}
double want = 0;
for (const Layer& l : L)
@@ -5468,7 +5481,7 @@ struct Model::Impl {
if (l.grp == (int) gi) used += (double) ucount(l) * l.n_vram;
// what the floors left: whole experts, layer by layer
for (Layer& l : L) {
- if (l.grp != (int) gi || l.cpu_only) continue;
+ if (l.grp != (int) gi || l.cpu_only || (l.mtp && mtp_cap < 1e300)) continue;
while (l.n_vram < l.n_exp && used + (double) ucount(l) <= budget) {
++l.n_vram;
used += (double) ucount(l);
```
Not checked:
- Multi-GPU
- Linux
- Metal
- The interaction with `DS4_STORE_ZC` (off here)
The drafter blocks' prompt-path streaming (`plan_streams` skips `l.mtp`) did not cause visible problems in these runs. I have not looked deeper at it.
Mehr auf der Site
Links zu Install, Modellen, Releases.