#3322 [Engine] support the official Flash-Next DFlash2 checkpoint

Beschreibung

## Problem and result

The official `incoai/Qwen3.8-Flash-Next-DFlash2` checkpoint requires five attention layers with 128-wide heads, raw HC-mean target features, dynamic convolution and a predecessor-conditioned selector. This change implements that checkpoint contract and supports original BF16 weights plus Q8_0, Q5_0 and Q4_0 backbone quantizations, using the full shared target output head.

Setup offers MTP (default), experimental DFlash2, or no model drafter. DFlash2 selection enables the local CUDA build, pinned download/conversion and artifact-specific calibration. Weights, private KV pools and attention scratch are accounted for before auto-sizing the expert cache, including the configured context length and elastic growth. MTP and DFlash2 cannot be selected together.

## Related discussion

Explicit maintainer confirmation of the expanded official-checkpoint/setup scope remains pending. This remains a draft while that scope and missing backend validation are resolved.

## Approach

DFlash2 is opt-in behind `STRATA_ENABLE_DFLASH2=ON`. The converter validates the checkpoint geometry and preserves all 45 norm, convolution and selector tensors in BF16; the 36 backbone matrices can be quantized. Setup offers BF16, Q8_0 and Q4_0; Q5_0 is also supported by the converter/runtime and included in the measurements below.

Prompt and conversation snapshots retain the draft context. Cache controls enforce matching actual expert placement before checking output. The shared runtime includes the infrastructure required for proposal verification, a fix for state publication on the opt-in QFUSE path and a fix for the CPU-only loader-test build dependency. The existing default drafter stays MTP.

## Performance: MTP versus DFlash2

Fresh same-source runs on RTX 4070 Ti SUPER 16 GiB, Ryzen 9 5900X, 125.7 GiB RAM, Linux, CUDA 13.4 / SM89. Target: local Qwen3.8-Flash-Next NVFP4 pack, BF16 embeddings, FP8 PLE, shared Q5_K output head, INT8 KV. Context 8192, prefill 256, auto expert cache, allocation reserve 512 MiB. All arms fill available VRAM; smaller draft allocations permit more target experts.

Greedy seed 1234, 96 new tokens, four fixed prompts (27/54/33/2425 input tokens), one warmup per prompt and three measured repeats per speculative arm. MTP at floor 0 is run before and after all four DFlash2 variants. An additional MTP run at setup's floor 0.5 follows those series and is the table reference; this sensitivity check was added after the initial series to avoid comparing only against an untuned zero-floor baseline. Both MTP settings remain in the raw evidence.

DFlash2 uses three proposals (verifier capacity 8, fixed block 3); MTP uses up to three proposals (verifier capacity 4), probability floor 0 in the fixed-budget series, plus the separate MTP setup-floor 0.5 control. The DFlash2 block length is fixed; the study does not independently tune each drafter.

CPU prompt sharing, PCIe sharing, adaptive swaps, suffix/lookup drafting and prompt/conversation snapshots are off. Engines run sequentially, with no build or other benchmark during measurements. Values are medians of native decode throughput; prefill/startup are excluded. Geomean weights all four task medians equally.

| Drafter | Code tok/s | Math tok/s | Prose tok/s | Long tok/s | Geomean tok/s (vs MTP) |
| --- | ---: | ---: | ---: | ---: | ---: |
| MTP (setup floor 0.5) | 38.05 | 44.84 | 39.94 | 55.19 | 44.04 (reference) |
| MTP (floor 0, pooled) | 36.81 | 43.68 | 36.26 | 48.61 | 41.03 (-6.8%) |
| DFlash2 BF16 | **43.25 (+13.7%)** | 43.16 (-3.7%) | 36.40 (-8.9%) | 41.77 (-24.3%) | 41.05 (-6.8%) |
| DFlash2 Q8_0 | **45.08 (+18.5%)** | 44.45 (-0.9%) | 37.05 (-7.2%) | 44.58 (-19.2%) | 42.65 (-3.1%) |
| DFlash2 Q5_0 | **45.19 (+18.8%)** | 44.87 (+0.1%) | 33.71 (-15.6%) | 43.89 (-20.5%) | 41.62 (-5.5%) |
| DFlash2 Q4_0 | **47.11 (+23.8%)** | **45.36 (+1.2%)** | **40.66 (+1.8%)** | 43.52 (-21.1%) | 44.10 (+0.1%) — tie |

Q4_0 has the best observed DFlash2 aggregate, 44.10 versus MTP's 44.04 tok/s (+0.1%). Their per-repeat geometric-mean ranges overlap: treat this as an overall tie. Its clear code advantage is +23.8%; math/prose improve by 1.2%/1.8% in this sample, while the long prompt regresses by 21.1%. Q8_0, Q5_0 and BF16 are slower in aggregate.

Per-repeat geometric means span 43.00–44.10 tok/s for MTP floor 0.5 and 44.07–44.20 tok/s for DFlash2 Q4_0. Bold task cells mean every measured DFlash2 sample exceeded every MTP floor-0.5 sample for that task; percentages are relative to MTP for that column. The bracketing zero-floor MTP's end/start geomean changed by -0.49%. Three samples per DFlash2/default-floor MTP task and six pooled zero-floor MTP samples are too few to establish a general speedup. Non-overlapping observed throughput ranges: bf16/code, Q8_0/code, Q5_0/code, Q4_0/code, Q4_0/math, Q4_0/prose. These are descriptive ranges, not confidence intervals.

| Drafter | Initial runtime MiB | Extra draft-head MiB | GPU expert slots | Minimum GPU free MiB |
| --- | ---: | ---: | ---: | ---: |
| MTP floor 0, start | 801.0 | 275.7 | 3140 | 182 |
| DFlash2 BF16 | 1354.6 | 0.0 | 3038 | 130 |
| DFlash2 Q8_0 | 945.0 | 0.0 | 3193 | 132 |
| DFlash2 Q5_0 | 780.9 | 0.0 | 3256 | 133 |
| DFlash2 Q4_0 | 726.2 | 0.0 | 3277 | 129 |
| MTP floor 0, end | 801.0 | 275.7 | 3141 | 181 |
| MTP floor 0.5 | 801.0 | 275.7 | 3143 | 166 |

The initial runtime column is the engine's load counter, not total process VRAM. MTP subsequently allocates another 275.7 MiB of selected output-head rows, plus index/sampling buffers. DFlash2's counter includes its weights, private KV pools and scratch, and it shares the full target output head without another head-weight copy. Expert slots and GPU-free snapshots show the practical final placement.

[Raw runs, ranges, memory and output controls](https://github.com/j-luwierski/Strata/blob/86af7286254d1efb148cf0aa8c9a96868d4e9f09/bench/results/2026-10-19-dflash2-pr-performance/README.md).

## Long-prompt diagnosis and fix (2026-10-20)

The earlier long-case loss used DFlash2 at floor 0 against MTP at setup's floor 0.5. A controlled Q4_0 follow-up isolates the gate in one process at the same actual expert placement. Fixed-block manual DFlash2 launches now also default to `--spec-min-p 0.5`; an explicit floor, including zero, still wins. Setup already selected 0.5, so existing setup configurations do not change.

Fresh same-hardware runs at `c7bebe215a6feb754e08dfbc02a3ac12d38fa55f`: three measured repeats per cell after warmup, 96 generated tokens, the same 27/2425-token code/long fixtures, context 8192, prefill 256, auto expert cache and 512 MiB reserve. Q4_0 fits 3279 target experts; MTP fits 3134. These are new runs, not ratios against the previous day's data.

| Drafter | Code tok/s (vs MTP) | Long tok/s (vs MTP) |
| --- | ---: | ---: |
| MTP, floor 0.5 | 38.33 (reference) | 51.57 (reference) |
| DFlash2 Q4_0, floor 0 | **46.18 (+20.5%)** | 42.57 (-17.5%) |
| DFlash2 Q4_0, floor 0.5 | **48.04 (+25.3%)** | 48.29 (-6.4%) |

Gating improves Q4_0 by 4.0% on code and 13.4% on the long case. The long verifier processes 113 rows instead of 133 while still accepting 62 proposals. Median target verification falls from 62.09 to 54.30 ms/window; draft context/proposal time stays near 4.2 ms. The saving is predominantly avoided verification of rejected suffixes, not a changed attention kernel.

MTP still leads this long fixture: 33 verification windows versus 34, and 2.42 versus 4.15 ms/window for drafting. The remaining loss is preserved in the table. The two fixtures have different task content as well as length; this does not establish a generic context-length slowdown. BF16/Q8_0/Q5_0 and the other two tasks were not rerun in this follow-up, and no new four-task aggregate is claimed.

[Follow-up raw runs, profiles, exact output controls and reproduction](https://github.com/j-luwierski/Strata/blob/c7bebe215a6feb754e08dfbc02a3ac12d38fa55f/bench/results/2026-10-20-dflash2-long-prompt/README.md).

## Draft attention fix (2026-10-20)

The remaining long-case draft overhead included duplicated softmax exponentials and paged value addresses in all 128 value lanes of each attention block. The kernel now computes these common values once per cell in shared memory, retaining the original dot products, maximum, ordered sums, mask and FP16 KV format. It adds 768 bytes of shared memory per block and no global allocation. No weight, threshold, block or target-verification change is involved.

Fresh runs at `2ad8e64662a6847db61aa12783fc82d862fbf87c`, against parent `c7bebe215a6feb754e08dfbc02a3ac12d38fa55f`, on the same RTX 4070 Ti SUPER / Ryzen 9 5900X: Q4_0 block 3, floor 0.5, context 8192, prefill 256, 96 generated tokens, warmup plus three repeats per task, normal CUDA Graphs enabled. Both DFlash2 processes fill VRAM with exactly 3274 target experts; fresh MTP independently fits 3139. Medians for the 2425-token fixture:

| Drafter | Draft ms/window | Decode tok/s |
| --- | ---: | ---: |
| MTP fresh control | 2.46 | 51.24 |
| DFlash2 Q4_0 before | 4.18 | 48.45 |
| DFlash2 Q4_0 patched | **3.19** | **49.39** |

Drafting falls **23.7%**, whole decode improves **1.9%**, and about 58% of the draft-stage gap to fresh MTP is removed. MTP still leads this long fixture by throughput: patched DFlash2 is 3.6% slower and still needs 34 verification windows versus 33. On the 27-token code fixture, **patched DFlash2 leads fresh MTP by 28.9%** (48.29 versus 37.47 tok/s); its own before/after change (+0.3%) is within measured variation. These two-task results do not replace the earlier four-task/all-quantization table.

Passed: CUDA ON build and three parity/policy/loader CTests; unchanged OFF build and two CTests; 192/192 fixture token IDs match target-only controls across every warmed/measured arm, with no near-tie exemption. The expanded attention test covers official 32Q/8KV geometry, local masks, padded graphs and nonfinite inputs. Graph-replayed full-window four-row attention is 2.95× faster; all 114688 FP32 outputs are byte-identical in three paired binary-dump comparisons.

The parent's runtime CI suite failed the portable child-watcher assertion (1 failure in 882 tests, 14 skipped); that exact case passes locally. Its cause is not established by this kernel investigation. No new-head CI success is claimed. HIP/SYCL/Metal/Windows and other quantizations were not rerun; this stays a draft.

[Raw measurements, ranges, profiles, commands, output controls and limits](https://github.com/j-luwierski/Strata/blob/2ad8e64662a6847db61aa12783fc82d862fbf87c/bench/results/2026-10-20-dflash2-attention/README.md).

## Validation

Earlier checkpoint validation used engine source at `af85f5ce3b4411833b5529361ae18ffdadbdce3f`, based on main `e88ee39d1d6be6e3364183af3b39b62a7ed6f211`. Performance evidence commit: `86af7286254d1efb148cf0aa8c9a96868d4e9f09`; it adds scripts/results without changing engine code. Linux, RTX 4070 Ti SUPER 16 GiB, Ryzen 9 5900X, 125.7 GiB RAM, CUDA 13.4, GCC 14.2, SM89.

**Passed:**

- Follow-up fix at `c7bebe215a6feb754e08dfbc02a3ac12d38fa55f`: CUDA extension ON/OFF builds; three ON and two OFF CTests; native fixed-block default/explicit-zero checks; 192/192 target-only fixture IDs match all threshold/block repeats at matching 3279 expert slots. Same-day MTP fixture IDs also match.
- CUDA extension ON/OFF builds; 12 selected CUDA CTests, plus extension-OFF loader rejection.
- 203 setup/conversion/cache/tool tests and 342 server tests.
- CPU-only full build after fixing the loader test's CUDA-header dependency.
- Main/OFF/ON regression comparison: 768 candidate IDs match main across AR/MTP, greedy/sampled requests and three prompts. Final rebuild executable hashes are unchanged.
- Official Q8 DFlash2 cache test: eight cases, 256/256 IDs equal both fresh DFlash2 and target-only controls at matching 2000 expert slots. Checks l