#5807 [Engine] DFlash2 drafting confidence is mis-tuned for a 16 GB Ada card: `CONF_STEPS` chooses 0.8, the measured optimum is 0.5, worth +16.7 % decode

open · @njcray2 · 1 コメント · GitHub で見る

BenchmarksNVIDIA / CUDAModels & quantsSecurityDocumentationWindows

本文

# [Engine] DFlash2 drafting confidence is mis-tuned for a 16 GB Ada card: `CONF_STEPS` chooses 0.8, the measured optimum is 0.5, worth +16.7 % decode

**Model:** qwen3.8-flash-next, IQ3_XXS pack (47,039,860,096 B on disk, 70.6 GiB resident), 512 experts, 10 routed per token, 393,216 context
**Card:** RTX 4070 Ti SUPER, 16,376 MiB, **sm_89 (Ada)**, driver 617.42 - 63.4 GiB system RAM, Ryzen 9 7900X
**Software:** Strata 0.2.13.1, engine 0.1.59-windows-x86_64-cuda-cuda13 (`12a21cddb83e`); engine 0.1.52 (`dbd0f9b692e9`) used for cross-checks
**Start flags held identical across every measurement below:** `--max-context 393216 --rope-scaling yarn --rope-scale 1.5 --vision --vram-reserve-mib 1600 --kv-resident 32768 --pcie-frac 0.20 --adapt-every 1 --adapt-swaps 160 --adapt-decay 0.97 --kv k8v4`
**Drafter pack used in every DFlash2 run:** `qwen3.8-flash-next-dflash2.q4_0-selq4.pack` (339,694,592 B), logging `drafter 591 MiB of VRAM (Q4_0 weights)` plus a 178.4 MiB draft head

## Author Note

When updating to 0.2.11/0.2.13 the drafter was changed from MTP to DFlash2. This resulted in a drop in decode tokens/s. The report below was generated by Qwen which investigated the cause (as well as a check on the benefit of STRATA_ROUTE_TAIL_SKIP including a check that TAIL_SKIP does not damage result quality.) The headline finding is that this card can benefit from DFlash2 but needs tuning as detailed below.

## Summary

1. `CONF_STEPS` gives this card `--draft-conf 0.8`. On this card that value costs **16.7 % of decode speed** compared with 0.5 (92.0 against 107.4 tok/s). At the engine default rank it is the largest single lever measured - ahead of the route tail skip (+14.7 % at conf 0.5) and ahead of the drafter swap (MTP is +11.8 % over DFlash2 at conf 0.8, but only +2.2 % over DFlash2 at conf 0.5, and 0.6 % *behind* DFlash2 at 0.35). At rank 0, with the skip off, the confidence alone is still worth +11.3 %.
2. The curve is a **slope with a broad top at 0.5**, not a cliff. 0.35-0.5 are all within 1.6 % of each other; 0.7 is already 10 % below the top.
3. `sm_89` is in neither `_TESTED` nor `_SLOWER`, so Auto classifies this card "untested" and starts DFlash2 anyway - which turns out to be the **right drafter**, provided the confidence is not 0.8. Eager DFlash2 beats MTP here *and* holds 156 more resident experts.

## Method

Twenty arms, each a copy of the same 384K launch config with one thing changed. Every arm ran `strata tune auto` three times on the same 76-token prompt and 200 generated tokens, with the pinned arena registered (`cudaHostRegister` portable, 42,916,118,528 B - no mmap fallback in any arm) and `outcome default_confirmed`. Per-arm repeat spread 0.5-3.4 %; every percentage below is quoted only when it is larger than the spread of the arms concerned. Speed, draft volume and cost per pass come from the engine's own `generated in N ms` and `drafts accepted x of y` counters per request. The cache columns sum `hits` and `lookups` from every `decode expert cache hit rate` line the arm produced; a single one of those lines can come from a very short request (one arm logged 94.1 % over 4,076 lookups), so a last-line figure is not a measurement.

## 1. The confidence curve (rank 7 = engine default, DFlash2, `--spec 6`, everything else identical)

| `--draft-conf` | draft tokens proposed | accepted | accepted per proposed | ms per verified pass | verified passes per 5,288 tokens | decode tok/s | cache hit, all requests | expert lookups missed |
|---|---|---|---|---|---|---|---|---|
| **0.8 - what Auto gave this card** | 1,763 | 1,646 | 93 % | 15.8 | 3,642 | **92.0** | 87.6 % | 543,431 |
| 0.7 | 2,214 | 1,904 | 86 % | 16.3 | 3,384 | 96.7 | 87.2 % | 575,559 |
| **0.5** | 3,381 | 2,454 | 73 % | 17.5 | 2,834 | **107.4** [106.5-108.3] | 87.0 % | 648,632 |
| 0.35 | 4,488 | 2,734 | 61 % | 19.4 | 2,554 | 105.7 | 87.6 % | 703,622 |
| 0.2 | 6,326 | 2,905 | 46 % | 21.2 | 2,383 | 105.2 | 88.1 % | 837,616 |

The turn is arithmetic, and it is exact. Decode time is `verified passes x ms per pass`, and passes are `generated - accepted`: 3,642 x 15.8 ms = 57.5 s for 5,288 tokens at conf 0.8, 2,834 x 17.5 = 49.6 s at 0.5, 2,554 x 19.4 = 49.5 s at 0.35, 2,383 x 21.2 = 50.5 s at 0.2. **Opening the dial from 0.5 to 0.35 cuts 9.9 % of the passes and raises the cost per pass by 10.9 % - the total comes out the same to 0.1 %.** Below 0.5 each further reduction in passes is smaller than the cost it buys.

What eagerness does cost is memory traffic, and it is monotonic: the hit *rate* stays flat (87.0-88.1 %) because the engine counts more lookups as the windows widen, but the number of lookups that miss the resident set climbs from 543,431 at conf 0.8 to 837,616 at conf 0.2, **+54 %**. On a card that streams experts over PCIe that is headroom spent, and it is the reason to prefer 0.5 over 0.35 or 0.2 even where the bench times are equal.

**How the value was forced:** `--draft-conf 0.5` was placed in the run config's `user_args`, which is where `drafting.apply` looks for it (`drafting.py:1140`, kept as `kept_conf` at `:1157`, written into the engine command at `:1162`). Written into `args` instead it is removed by `strip_draft_flags` and replaced by the card's own value - which reads as if the setting had no effect at all.

## 2. Why 0.8 is the problem: the drafter stops proposing

`STRATA_SPEC_DEPTH=1`, acceptance at each draft depth, at rank 7:

- DFlash2 at conf 0.8: `33/33 10/10 7/7 5/6 4/4` - almost never wrong, and it **stops drafting**: a second draft token in about 30 % of windows, a fifth in about 12 %. Over a window group it tested 60 candidates and accepted 59.
- MTP under the same conditions: `48/57 24/34 17/21 0/0 0/0` - 57 candidates tested at depth 1 against DFlash2's 33, accepting 84 / 71 / 81 %, stopping at depth 3. It tested 112 and accepted 89.

The difference between the two drafters on this card is **draft volume, not accuracy**, and conf 0.8 suppresses DFlash2's volume while leaving its accuracy untouched. Opening the dial recovers all of it.

## 3. The ladder across skip settings, and the drafter comparison

`STRATA_ROUTE_TAIL_SKIP`, decode tok/s:

| skip rank | Auto, conf 0.8 | MTP | DFlash2 conf 0.35 | DFlash2 conf 0.5 | DFlash2 conf 0.7 |
|---|---|---|---|---|---|
| 0 - skip off | 84.1 | 95.7 | 96.3 | 93.6 | - |
| 8 - only the 9th and 10th of ten | 89.6 | 102.0 | 103.5 | - | - |
| 7 - engine default | 92.0 | 102.9 / 105.7 | 105.7 | **107.4** | 96.7 |

- Confidence alone, skip off: **+11.3 %** (84.1 → 93.6). Confidence alone, rank 7: **+16.7 %**. Skip at conf 0.5: **+14.7 %** (93.6 → 107.4). Skip at conf 0.8: +9.4 %.
- At rank 7 with both sides drafting eagerly, eager DFlash2 (107.4) is 1.6-4.4 % ahead of MTP (102.9 and 105.7 in two runs of the same configuration) **while holding 4,445 resident experts against MTP's 4,289** and about 550 MiB less drafter VRAM (591 + 178.4 MiB against 836-837 + 178.4 MiB). MTP reaches similar speed a different way: lower draft volume (0.46 accepted per generated token, against 0.46 at conf 0.5 and 0.52 at 0.35) with a similar verified pass - 17.8-17.9 ms against DFlash2's 17.5 ms at conf 0.5 and 19.4 ms at 0.35. At rank 0 the picture is flatter: MTP 95.7, DFlash2 at conf 0.35 96.3, DFlash2 at conf 0.5 93.6.
- The tail skip's own counters go the same way: at rank 7 it skipped 37,688 experts at conf 0.8 and 44,658 at conf 0.5, so eager drafting did not starve it. Its effect on memory traffic is the bigger part of its value: with the skip off, the same drafting missed 895,865 lookups at conf 0.8 and 962,222 at conf 0.5, on hit rates of 81.2 % and 81.9 %, against 543,431 and 648,632 missed at rank 7 (87.6 % and 87.0 %).

## 4. Drafting pays for its VRAM, decisively

With `"drafting": "off"` the start loaded no draft head at all, held **4,918 resident experts** - more than either drafter - hit 90.6 % of the expert cache, and was still the **slowest** configuration measured: 78.9 tok/s, with a verified pass of 13.8 ms, the cheapest of any arm. Rank-matched against that floor at rank 0: MTP **+21.3 %**, DFlash2 at conf 0.8 only **+6.6 %**, DFlash2 at conf 0.5 **+18.6 %**. There is no no-drafter measurement at rank 7; against the same rank-0 floor the rank-7 arms read MTP +30.4 % (102.9) and DFlash2 at conf 0.5 **+36.1 %** (107.4). So the drafter earns far more than the VRAM it costs - but almost all of DFlash2's value was being held back by the confidence, not by the memory it uses.

## 5. Secondary results

- **`SPEC_WINDOW = 6` is right for this card.** Widening `--spec` to 8 at conf 0.5 measured 105.7 [103.5-109.2] against 107.4 - no gain, and the widest repeat spread of any arm. Confidence is the axis; window size is not.
- **Engine version is not a factor once the drafter matches.** 0.1.59 gives 102.9-105.7 with MTP at rank 7; 0.1.52 gives 105.0-105.7 with MTP at rank 7 (four measurements, two of them re-runs with a logging flag).
- **Engine 0.1.52 will not take a named DFlash2.** With `"drafting": "dflash2"` its command line came out `--spec 4 --spec-min-p 0.5 --mtp M:\Strata-data\mtp\rt` with no draft flags, and the log showed the 836 MiB MTP draft layer loading. Worth stating in the docs next to `--draft-fallback`, which 0.1.52 does not have.
- **The step that decides this card's confidence is chosen from a slot estimate calibrated on a different quantisation.** `slots_of()` divides an estimated free-VRAM figure by `SLOT_MIB = 1.95`, documented in the source as "one expert slot of Qwen3.8-Flash-Next **IQ3_S**" ([drafting.py:445](M:\Strata\strata_control\drafting.py#L445), [drafting.py:813](M:\Strata\strata_control\drafting.py#L813)), and that number picks the `CONF_STEPS` step. Run against this user's own two configs it returns **2,451 slots** at 393,216 context and **3,289** at 240,000, while the engine on those same configs logged `-> 3331 slots` and `-> 3450 slots`. The 384K estimate is 26 % below the engine's own figure and it straddles the boundary: 2,451 falls under 3,000 so `conf_for` returns **0.8**, while 3,289 clears it and the same model on the same card gets **0.7**. The ratio between the two counts (3,331 / 2,451 = 1.36) is close to the IQ3_S-to-IQ3_XXS weight ratio (4.55 / 3.44 bpw = 1.32), which is what a single per-slot size measured on IQ3_S and then applied to an IQ3_XXS pack would produce. `slots_of`'s own comment says it is "an estimate good to about 10%" and that "the steps of CONF_STEPS are wider"; here the error exceeds that and crosses a step. What it costs is in the table above: 0.8 was applied where 0.5 measures 16.7 % better, and the 0.7 that the 240K configuration did get measures 11.1 % worse than 0.5.
- 
## Suggested changes

1. Re-measure the `CONF_STEPS` steps for the 12-16 GB streaming class. The direction of the ladder - fewer resident experts, more conservative drafting - is confirmed by this data; the magnitude is not. For this card 0.8 is 16.7 % of decode, and the step it reserves for cards holding every expert (0.35) is *worse* than 0.5 here.
2. `sm_89` deserves an entry in `_TESTED` or `_SLOWER`. Today it reports "untested" and Auto starts DFlash2, which is the correct choice on this card - but by luck rather than measurement.
3. Give `SLOT_MIB` a value per quantisation, or take the slot size from the engine's own line for the pack in use. As it stands, `CONF_STEPS` chooses this card's step from a count that is 26 % off for an IQ3_XXS pack, and the error is large enough to cross a step boundary - the difference between 0.8 and 0.7, and in this case 16.7 % of decode.
4. If only one card class is re-measured, make it this one: `--draft-conf 0.5` is worth more than every other setting tested on a 16 GB Ada card with this model.

## Caveats

Every figure here is a `tune auto` measurement: a 76-token prompt and 200 generated tokens, three rounds. The user's own practical experience agrees with the direction, but a long chat session at conf 0.5 has not been measured. What has been measured is that more eager 

関連リンク

インストール・モデル・リリースへのサイト内リンク。