#5570 [Engine] DeepSeek-V4.1 has no --no-prefill-borrow: on a 16 GB card the prompt loan costs half the expert cache

closed · @mazzyler · 3 comentarios · En GitHub

BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows

Descripción

### Area

[Engine]

### Environment

| | |
|---|---|
| GPU | one GeForce RTX 5080, 16,303 MiB, driver 621.01 |
| CPU / RAM | AMD Ryzen 9 7950X3D, 127 GB |
| OS | Windows 10 Pro 22H2 (build 19045) |
| Strata / engine | 0.2.11 / **0.1.57**, the prebuilt `0.1.57-windows-x86_64-cuda-cuda13` from the release |
| Model | ISTA-DASLab **DeepSeek-V4.1-Flash 2.25bpw** (GGUF pack `deepseek-v4.1-flash-ist-das-2.25bpw`), 15,360 experts in the layers |
| Args | `--max-context 32768 --vision on --dspark auto --reserve-mib 400 --experts 6`, one model slot |

### What I am asking for

The Qwen engine has a documented switch for exactly this: `--prefill-borrow` / `--no-prefill-borrow`
(`src/program/generate.cpp:835-837`, parsed at `:3552`; `docs/INTEL.md:113`), with its trade-off measured in
`docs/INTEL.md:184-186`, `docs/BATCHING.md:613` and `docs/STRIX_HALO_WINDOWS.md:96`. The DeepSeek-V4.1 engine has
the same idea and no switch: the loan is decided by `one_gpu_or("DS4_BORROW")` in `src/dsv41/ds4_model.cu:1993`,
read by `chunk_default` (`:3993-4025`) and `alloc_chunks` (`:4046`). That variable is in no document and in no
`--help`, and `src/dsv41/ds4_main.cpp` has no option for it.

I would like `--no-prefill-borrow` on the DeepSeek engine as well - the same name, so a person who learned it on
the Qwen engine keeps it. If a flag is out of scope, documenting the variable wherever a DeepSeek user looks would
already help. The default can stay exactly as it is; I am not asking for a change to it.

If this belongs upstream in Project Maya rather than here (`third_party/project-maya/NOTICE.md` says `src/dsv41`
comes from there), say so and I will bring it there instead.

### Why this report

On a card where the model does not fit, the loan is memory the expert cache cannot use. On one 16 GB card it is
about 2.7 GB of video memory: 297 experts against 612, and generation is 9.6% faster without it. The price is
prompt throughput on prompts longer than about 4K tokens. Measured here, one run per row, same box, minutes apart;
the two configurations differ in one line of the model JSON and nothing else.

| | loan on (default) | `DS4_BORROW=0` |
|---|---:|---:|
| experts in VRAM (of 15,360) | 297 | **612** |
| decode, one request, 384 tokens | 12.37 tok/s | **13.56 tok/s** |
| prefill, ~17,000-token prompt | **752 tok/s** (22.65 s) | 663 tok/s (25.68 s) |
| four requests at once (`--parallel 4`): each / in total | 6.20 / 24.81 tok/s | **7.09 / 28.35 tok/s** |
| same, experts in VRAM | 288 | **601** |

### Where the video memory goes

`nvidia-smi` was read after the model answered one request; the engine's own `ds4: CUDA0 layers` line is beside
it. The line reports only part of the card in both runs, which is why both columns are here:

| run | experts | the engine's line (weights + experts + caches) | `nvidia-smi`, in use | not on the line |
|---|---:|---:|---:|---:|
| default (loan on, vision on) | 297 | 8,736 MiB | 14,647 MiB | 5,911 MiB |
| `DS4_BORROW=0` | **612** | 11,663 MiB | 14,857 MiB | 3,194 MiB |
| `--vision off` | 351 | 9,232 MiB | 14,869 MiB | 5,638 MiB |

The card holds about 14,650 MiB in both runs. The 2,927 MiB the engine's line gained in experts came out of the
5,911 MiB it does not account for, which fell to 3,194 MiB: the loan was holding memory the line cannot see, and
turning it off put 315 more expert slots there. I did not decompose the 3,194 MiB that remain - that needs a build
that prints the scratch region, and this machine has no MSVC toolchain to rebuild the engine.

### Two other variants are fine as they are

Tested and rejected, which is why I think the loan is the outlier rather than the cache design being loose:

- `DS4_RING_MB=0` with `DS4_BORROW=0`: 696 experts, but 12.07 tok/s decode and 595 tok/s prefill - the prompt ring
  refills experts during generation too, so it earns its room.
- `--no-expert-cache` with `DS4_BORROW=0`: 652 experts (+40 = the spare slot of each of the 40 layers), but
  12.93 tok/s decode - the spare slot is what makes a cache swap legal, and it earns its room.

### Smallest steps to reproduce

Two copies of the same model JSON that differ in one line, `"env": {"DS4_BORROW": "0"}`; start each, then read the
log line and time one request.

```sh
python serve/server.py --engine strata --config cfg-borrow-on.json --port 8090
# strata-deepseek-v4.1-flash-ist-das-2.25bpw.log:
#   ds4: CUDA0 layers [0, 39] + head: weights 5.86 GB, experts 3.27 GB (297 of 15360 in VRAM), caches 0.03 GB, 441 MiB free
curl -s http://127.0.0.1:8090/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4.1-flash","messages":[{"role":"user","content":"Explain a hash table."}],"max_tokens":384}'
```

Repeat with `cfg-borrow-off.json`, whose line reads `experts 6.34 GB (612 of 15360 in VRAM)`. Everything else in the
log is the same; only the expert count and the timing move.

### Not run

- HIP and Metal, and any card but this one: one RTX 5080, one pack, one context (32K). The same 2.7 GB is a much
  smaller share of a 24 or 48 GB card, so the tradeoff may not be worth having there at all.
- Multi-GPU: with two or more cards the loan is already off unless `DS4_BORROW=1`, so I expect no change, but I did
  not run it.
- `--parallel` other than 1 and 4, prompts past the 32K context, and other DeepSeek packs.
- I did not rebuild the engine: it is the prebuilt `0.1.57-windows-x86_64-cuda-cuda13`. The line numbers above come
  from `src/dsv41` in the tree as I read it, not from a build I made, so treat them as **unconfirmed static
  analysis**; the measured rows are what I ran.

Related: #3310 chooses how many expert slots a loan keeps on the Qwen path. This asks for the same kind of choice on
the DeepSeek engine, where there is none.

The text was prepared with an AI agent; the numbers were measured on the machine described above.

En el sitio

Enlaces a install, modelos, releases.