Pull requests / #3589

#3589 [Engine] DeepSeek V4-Flash (deepseek4) as a second model family

open · draft · @JUNQINGV587 · 0 Kommentare · Auf GitHub

AMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows

Beschreibung

## Problem

A DeepSeek V4-Flash checkpoint cannot be loaded at all today: `gguf_reader.hpp` accepts only `qwen4exp`, so
the file is rejected before anything else happens. The model differs from Qwen3.8-Flash-Next in the parts
that decide the engine's shape — hash routing on the first layers, MLA attention with a 128 window plus
compressed slots picked by the DSA indexer, 256 experts top-6 on a 4096 hidden / 2048 intermediate geometry —
so it needs its own loader, geometry and kernels rather than a new branch in the Qwen path.

Issue: #479 (V4-Flash support request). This follows the reopen instructions from #1518 being closed by the
0.2.0-beta tracker reset: it is rebased onto `main` (`011c9e60`), its checks were re-run on this tree, and the
title carries the area prefix.

## What this adds

A second model family, **additive to the default path**:

- `src/dsv4/` — engine, MoE path, attention + compressor kernels, and a CPU reference forward
  (`dsv4ref.cpp`, binary `strata-dsv4ref`) that the engine's dumps are compared against.
- `include/strata/dsv4/` — geometry, layout, attention/compressor interfaces.
- `generate.cpp` — one switch (`STRATA_HAVE_DSV4`) that reads the GGUF header's architecture and hands
  `deepseek4` files to `dsv4::run`; `strata-dsv4` is the same entry point on its own command line.
- `tools/nvfp4_convert_dsv4.py` — conversion through the pinned llama.cpp (it already knows `deepseek4` and
  writes `compress_ratios`; the script patches the ModelOpt NVFP4 naming and merges per-expert tensors).
  `tools/iq_pack.py` packs the result as usual; the pack format is unchanged (36-byte NVFP4 blobs).
- `docs/DSV4.md` — what is supported, the conversion commands, and the limits.

The Qwen gate and its kernels are untouched. The only shared-file edits are the family switch, two widened
activation constants in `native_expert.hpp` for the wider V4 geometry, and one extra overload in
`expert_layout.hpp`; all three keep the Qwen path's behaviour.

## Limits (deliberate)

- **No MTP/DSpark draft head.** Measured on this pool: the block verify costs `c(5) ≈ 2.76` forwards per
  round for ~2.4 accepted tokens, and the draft layer is another 2.7-10.9 GiB of resident experts.
- **v5 (Hadamard-folded) packs are refused at start-up** — the CPU native path rotates activations, this
  engine's CUDA GEMV does not, so such a pack would silently compute wrong numbers.
- Context capped at 65536 (compressor slots are sized from `ctx/128`); prompts longer than the 128-token
  window are prefill-chunked by the caller.
- NVFP4 only; IQ2/IQ4 packs of this family are not converted or tested.

## Validation

Environment: this branch on `011c9e60`, containers with CUDA 13.0.2, sm_89 (2×L20, 96 GiB VRAM, 125 GiB RAM),
Release build, model `DeepSeek-V4-Flash-0731-NVFP4` (164 GB GGUF → 145.13 GiB expert pool, 57.5 % resident).

Passed (commands and logs available on request; dumps are compared file by file):

| check | result |
|---|---|
| golden prompts, prefill argmax + 3 decode steps vs CPU reference | MATCH (9544 / 734 / 260 / 4654) |
| multi-row verify R=2..6 vs single-row decode, dumps | 0 files differ (bit-identical) |
| long context, start_pos=130, window wrapped | 0 files differ |
| `compute-sanitizer --tool memcheck` | 0 errors |
| Qwen3.8-Flash-Next NVFP4 on the same binary | 109.24 / 96.99 / 116.94 t/s (this box's band: 88.5-117.1) |

Throughput on the same machine: ~30 t/s for one decode stream (33 ms/pass), ~47 t/s with six rows in one
forward (127 ms/pass).

Not run: GPUs other than sm_89 (no sm_120/HIP/SYCL), Windows, single-GPU 24 GB, IQ2/IQ4 packs, MTP.

## Size

~11 k lines, mostly new files under `src/dsv4` (`dsv4_moe.cu` 2051, `dsv4ref.cpp` 1575, `dsv4_comp.cu`
1193, `dsv4_attn.cu` 1119). Happy to split it into loader/geometry, attention and MoE PRs if that is
easier to review.

Mehr auf der Site

Links zu Install, Modellen, Releases.