#6406 [Engine] dflash2: unlock spec 8 window for desktop Blackwell cc 12.0 cards >= 15 GB (+89% decode on code review)

open · @cobra91 · 0 commentaires · Sur GitHub

BenchmarksAMD / HIPNVIDIA / CUDAWindows

Description

## Problem and result
In `engine/include/strata/core/draft_cap.hpp` and `terminal/strata_control/drafting.py`, `dflash2_auto_cap()` and `window_for()` currently cap consumer Blackwell (`sm_120`, cc 12.0) cards under 48 GB to a draft window of 6 tokens (`dflash2_auto_cap = 6`), noting:
`// A consumer RTX 50 card with less memory was not measured and keeps the older values.`

In the desktop Electron app and CLI, this cap cannot be adjusted by the user because `--draft-cap` is not exposed in the GUI, keeping desktop cards clamped to 6.

**Before and after:**
- **Before:** On desktop Blackwell cards (such as RTX 5080 16 GB and RTX 5090 32 GB), `--draft dflash2` is clamped to `--spec 6`.
- **After:** Desktop Blackwell cards (`sm_120`, cc 12.0) with at least 15 GB of VRAM automatically unlock the wider window 8 (`--spec 8`). This boosts sustained decode speed by up to +89% on realistic code review (230.4 vs 122.3 tok/s), +10.5% on real 5K prompts (161.3-163.6 vs 148.0 tok/s), and greedy counting (238.7 vs 232.4 tok/s), for an incremental device buffer overhead of only 11.3 MiB (68.5 MiB vs 57.2 MiB total).

## Related discussion
none

## Approach
- In `engine/include/strata/core/draft_cap.hpp`, lower the threshold for desktop Blackwell (`sm_120`, cc 12.0) from 48 GB to 15 GB (`kDflash2BlackwellW8Bytes = 15'000'000'000ull`). Consumer cards with < 15 GB (such as RTX 5070 12 GB) and the DGX Spark (cc 12.1) keep the conservative cap of 6.
- In `terminal/strata_control/drafting.py`, add `blackwell_w8()` and update `window_for()` to assign `WIDE_WINDOW` (8) for desktop `sm_120` cards with >= 15 GB and normal expert cache (>= 3,000 slots).
- Keep `big_blackwell` intact for `big_blackwell_conf_for` (only the 72 GB card gets conf 0.55).
- Update unit tests in `engine/src/core/draft_cap_test.cpp` and `terminal/tests/terminal/test_drafting_card_window.py`.

## Validation

**Passed:**
- `pytest terminal/tests/terminal/test_drafting_card_window.py`: 10 passed in 0.29s on Windows 11 Python 3.12 (commit 56b082e24).
- `pytest terminal/tests/terminal/test_drafting.py`: 75 passed in 10.34s on Windows 11 Python 3.12.
- Empirical benchmark on Windows 11 WDDM (RTX 5080 16 GB, cc 12.0, driver 621.14, model Swift-Qwen3.8-Flash-Next GSQ-RCO IQ2_XS with `dflash2-tuned-selq4.pack`):
  - Review 8K (temp 0.7): 230.4 tok/s (window 8) vs 122.3 tok/s (window 6 / MTP) (+89% speedup).
  - Review JS 5K (temp 0.7): 161.3 - 163.6 tok/s (window 8) vs 148.0 tok/s (window 6).
  - Counting 500 tokens (temp 0.0): 238.7 tok/s (window 8) vs 232.4 tok/s (window 6).
  - Draft acceptance rate: 81.6% (307 of 376 drafts accepted at window 8).
  - Greedy output identity verified 100%.

**Failed:**
none observed

**Not run:**
ROCm, Metal, and SYCL native builds (unaffected C++ header and Python-only logic).

## Limits
Cards under 15 GB (such as RTX 5070 12 GB) and DGX Spark (`sm_121`) intentionally remain capped at 6 to avoid tight VRAM overhead on lower-tier cards.

Sur le site

Liens install, modèles, releases.