#6406 [Engine] dflash2: unlock spec 8 window for desktop Blackwell cc 12.0 cards >= 15 GB (+89% decode on code review)
open · @cobra91 · 0 评论 · 去 GitHub 看
BenchmarksAMD / HIPNVIDIA / CUDAWindows
说明
## Problem and result In `engine/include/strata/core/draft_cap.hpp` and `terminal/strata_control/drafting.py`, `dflash2_auto_cap()` and `window_for()` currently cap consumer Blackwell (`sm_120`, cc 12.0) cards under 48 GB to a draft window of 6 tokens (`dflash2_auto_cap = 6`), noting: `// A consumer RTX 50 card with less memory was not measured and keeps the older values.` In the desktop Electron app and CLI, this cap cannot be adjusted by the user because `--draft-cap` is not exposed in the GUI, keeping desktop cards clamped to 6. **Before and after:** - **Before:** On desktop Blackwell cards (such as RTX 5080 16 GB and RTX 5090 32 GB), `--draft dflash2` is clamped to `--spec 6`. - **After:** Desktop Blackwell cards (`sm_120`, cc 12.0) with at least 15 GB of VRAM automatically unlock the wider window 8 (`--spec 8`). This boosts sustained decode speed by up to +89% on realistic code review (230.4 vs 122.3 tok/s), +10.5% on real 5K prompts (161.3-163.6 vs 148.0 tok/s), and greedy counting (238.7 vs 232.4 tok/s), for an incremental device buffer overhead of only 11.3 MiB (68.5 MiB vs 57.2 MiB total). ## Related discussion none ## Approach - In `engine/include/strata/core/draft_cap.hpp`, lower the threshold for desktop Blackwell (`sm_120`, cc 12.0) from 48 GB to 15 GB (`kDflash2BlackwellW8Bytes = 15'000'000'000ull`). Consumer cards with < 15 GB (such as RTX 5070 12 GB) and the DGX Spark (cc 12.1) keep the conservative cap of 6. - In `terminal/strata_control/drafting.py`, add `blackwell_w8()` and update `window_for()` to assign `WIDE_WINDOW` (8) for desktop `sm_120` cards with >= 15 GB and normal expert cache (>= 3,000 slots). - Keep `big_blackwell` intact for `big_blackwell_conf_for` (only the 72 GB card gets conf 0.55). - Update unit tests in `engine/src/core/draft_cap_test.cpp` and `terminal/tests/terminal/test_drafting_card_window.py`. ## Validation **Passed:** - `pytest terminal/tests/terminal/test_drafting_card_window.py`: 10 passed in 0.29s on Windows 11 Python 3.12 (commit 56b082e24). - `pytest terminal/tests/terminal/test_drafting.py`: 75 passed in 10.34s on Windows 11 Python 3.12. - Empirical benchmark on Windows 11 WDDM (RTX 5080 16 GB, cc 12.0, driver 621.14, model Swift-Qwen3.8-Flash-Next GSQ-RCO IQ2_XS with `dflash2-tuned-selq4.pack`): - Review 8K (temp 0.7): 230.4 tok/s (window 8) vs 122.3 tok/s (window 6 / MTP) (+89% speedup). - Review JS 5K (temp 0.7): 161.3 - 163.6 tok/s (window 8) vs 148.0 tok/s (window 6). - Counting 500 tokens (temp 0.0): 238.7 tok/s (window 8) vs 232.4 tok/s (window 6). - Draft acceptance rate: 81.6% (307 of 376 drafts accepted at window 8). - Greedy output identity verified 100%. **Failed:** none observed **Not run:** ROCm, Metal, and SYCL native builds (unaffected C++ header and Python-only logic). ## Limits Cards under 15 GB (such as RTX 5070 12 GB) and DGX Spark (`sm_121`) intentionally remain capped at 6 to avoid tight VRAM overhead on lower-tier cards.
本站相关内容
相关页面的快捷入口。