Pull requests / #3589
#3589 [Engine] DeepSeek V4-Flash (deepseek4) as a second model family
open · draft · @JUNQINGV587 · 0 comentários · No GitHub
AMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
Descrição
## Problem A DeepSeek V4-Flash checkpoint cannot be loaded at all today: `gguf_reader.hpp` accepts only `qwen4exp`, so the file is rejected before anything else happens. The model differs from Qwen3.8-Flash-Next in the parts that decide the engine's shape — hash routing on the first layers, MLA attention with a 128 window plus compressed slots picked by the DSA indexer, 256 experts top-6 on a 4096 hidden / 2048 intermediate geometry — so it needs its own loader, geometry and kernels rather than a new branch in the Qwen path. Issue: #479 (V4-Flash support request). This follows the reopen instructions from #1518 being closed by the 0.2.0-beta tracker reset: it is rebased onto `main` (`011c9e60`), its checks were re-run on this tree, and the title carries the area prefix. ## What this adds A second model family, **additive to the default path**: - `src/dsv4/` — engine, MoE path, attention + compressor kernels, and a CPU reference forward (`dsv4ref.cpp`, binary `strata-dsv4ref`) that the engine's dumps are compared against. - `include/strata/dsv4/` — geometry, layout, attention/compressor interfaces. - `generate.cpp` — one switch (`STRATA_HAVE_DSV4`) that reads the GGUF header's architecture and hands `deepseek4` files to `dsv4::run`; `strata-dsv4` is the same entry point on its own command line. - `tools/nvfp4_convert_dsv4.py` — conversion through the pinned llama.cpp (it already knows `deepseek4` and writes `compress_ratios`; the script patches the ModelOpt NVFP4 naming and merges per-expert tensors). `tools/iq_pack.py` packs the result as usual; the pack format is unchanged (36-byte NVFP4 blobs). - `docs/DSV4.md` — what is supported, the conversion commands, and the limits. The Qwen gate and its kernels are untouched. The only shared-file edits are the family switch, two widened activation constants in `native_expert.hpp` for the wider V4 geometry, and one extra overload in `expert_layout.hpp`; all three keep the Qwen path's behaviour. ## Limits (deliberate) - **No MTP/DSpark draft head.** Measured on this pool: the block verify costs `c(5) ≈ 2.76` forwards per round for ~2.4 accepted tokens, and the draft layer is another 2.7-10.9 GiB of resident experts. - **v5 (Hadamard-folded) packs are refused at start-up** — the CPU native path rotates activations, this engine's CUDA GEMV does not, so such a pack would silently compute wrong numbers. - Context capped at 65536 (compressor slots are sized from `ctx/128`); prompts longer than the 128-token window are prefill-chunked by the caller. - NVFP4 only; IQ2/IQ4 packs of this family are not converted or tested. ## Validation Environment: this branch on `011c9e60`, containers with CUDA 13.0.2, sm_89 (2×L20, 96 GiB VRAM, 125 GiB RAM), Release build, model `DeepSeek-V4-Flash-0731-NVFP4` (164 GB GGUF → 145.13 GiB expert pool, 57.5 % resident). Passed (commands and logs available on request; dumps are compared file by file): | check | result | |---|---| | golden prompts, prefill argmax + 3 decode steps vs CPU reference | MATCH (9544 / 734 / 260 / 4654) | | multi-row verify R=2..6 vs single-row decode, dumps | 0 files differ (bit-identical) | | long context, start_pos=130, window wrapped | 0 files differ | | `compute-sanitizer --tool memcheck` | 0 errors | | Qwen3.8-Flash-Next NVFP4 on the same binary | 109.24 / 96.99 / 116.94 t/s (this box's band: 88.5-117.1) | Throughput on the same machine: ~30 t/s for one decode stream (33 ms/pass), ~47 t/s with six rows in one forward (127 ms/pass). Not run: GPUs other than sm_89 (no sm_120/HIP/SYCL), Windows, single-GPU 24 GB, IQ2/IQ4 packs, MTP. ## Size ~11 k lines, mostly new files under `src/dsv4` (`dsv4_moe.cu` 2051, `dsv4ref.cpp` 1575, `dsv4_comp.cu` 1193, `dsv4_attn.cu` 1119). Happy to split it into loader/geometry, attention and MoE PRs if that is easier to review.
No site
Links install, modelos, releases.