#5432 [Engine] Do not force WindowInvariant CPU experts on DFlash2 windows when MTP fallback is loaded
open · @bsorensen110 · 0 评论 · 去 GitHub 看
Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAWindowsLinux
说明
## Problem and result When DFlash2 runs with the MTP layer loaded beside it as the fallback (`--draft auto` / `use_mtp == true` with `req_df2 == true` in `serve` or `spec_uses_df2 == true` in `generate`) and `STRATA_SPEC_PROB=1` is enabled (including on `gfx12` where `arch_defaults.cpp` sets `STRATA_SPEC_PROB=1`), `mtp_q_applies` (`include/strata/spec/window_rule.hpp:35`) correctly omits the MTP proposal distributions `q` on DFlash2 verify windows (`!df2_active`). However, the immediately following `decode_pool_fn` selection (`src/program/generate.cpp:18858` in the `serve` loop and `src/program/generate.cpp:20650` in the CLI loop) and `gated_proposal` (`src/program/generate.cpp:17550`) evaluated `use_mtp && mtp.prob()` without checking `!req_df2` / `!spec_uses_df2`. As a result, every DFlash2 verify window with an MTP fallback loaded forced CPU expert misses onto `invariant_win_pool_fn` / `drive_pool_multi_invariant` (`NativeArithmetic::WindowInvariant`, which sets `singleton_iq = true` and runs 1-token reference kernels per row instead of the multi-token grouped AVX2/AVX-512 IQ kernels) even though DFlash2 uses deterministic point-mass candidates (`mtp_q_applies == false`). This change adds `strata::spec::mtp_window_invariant_applies(bool use_mtp, bool df2_active, bool mtp_prob)` in `include/strata/spec/window_rule.hpp` and gates `gated_proposal` and both `decode_pool_fn` sites on it, keeping multi-token grouped CPU expert kernels active whenever DFlash2 is the active drafter. ## Related discussion None. ## Approach - **`include/strata/spec/window_rule.hpp:39-42`:** Add `mtp_window_invariant_applies(bool use_mtp, bool df2_active, bool mtp_prob)` returning `use_mtp && !df2_active && mtp_prob`, alongside `mtp_q_applies`. - **`src/program/generate.cpp:17550, 18858, 20650`:** Gate `gated_proposal` and `decode_pool_fn` (both `serve` and `generate` loops) through `strata::spec::mtp_window_invariant_applies(...)` using the request/session DFlash2 flag (`req_df2` in `serve`, `spec_uses_df2` in `generate`). - **`src/spec/window_rule_test.cpp:105-108`:** Add unit test assertions covering all four `(use_mtp, df2_active, mtp_prob)` states of `mtp_window_invariant_applies`. - **No behavior change for:** MTP-only probabilistic decoding (`use_mtp && !df2_active && mtp.prob()`), greedy decoding, or DFlash2-only runs without MTP loaded (`--draft-fallback none`). ## Validation Tested on revision `e6609ae52` (based on `main` `54828c7ef`, Strata 0.2.10 / engine 0.1.57) on Linux (`7.3.1-1-cachyos-server`, AMD Radeon AI PRO R9700 `gfx1201` 32 GB VRAM, 128 GB RAM). **Passed:** - `g++ -std=c++20 -O2 -Iinclude src/spec/window_rule_test.cpp -o window_rule_test && ./window_rule_test` on `e6609ae52`: `window_rule_test: ok` (all 22 checks pass, including the 4 new `mtp_window_invariant_applies` cases). - Base vs. fix probe on `54828c7ef`: for `(use_mtp=true, df2_active=true, mtp_prob=true)`, the prior inline condition `use_mtp && mtp_prob` returned `true` (selecting `invariant_win_pool_fn` while `mtp_q_applies(true, true, true, false, false, 4)` returned `false`); `mtp_window_invariant_applies(true, true, true)` returns `false`. - `cmake --build build-hip --target window_rule_test spec_prob_test spec_admission_test strata -j16` and CTest `window_rule_test`, `spec_prob_test`, `spec_admission_test`: 3/3 pass (`0` failures). **Failed:** - none observed **Not run:** - CUDA and Metal builds (no NVIDIA or Apple Silicon hardware on this Linux HIP host; `include/strata/spec/window_rule.hpp` and `src/program/generate.cpp` are shared across CUDA and HIP). ## Limits - Does not change DFlash2 candidate verification itself (which remains exact-match / sampled residual verification without MTP `q` lists, as intended by `mtp_q_applies`). - When MTP itself is the active drafter (`df2_active == false`) and `STRATA_SPEC_PROB=1` is active on a sampled request (`temperature >= 0.05`), `mtp_window_invariant_applies` still returns `true` so prefix-invariant CPU expert arithmetic is preserved for MTP rejection sampling. ## Reproduction ```sh g++ -std=c++20 -O2 -Iinclude src/spec/window_rule_test.cpp -o window_rule_test ./window_rule_test ```
本站相关内容
相关页面的快捷入口。