Pull requests / #1818
#1818 fix: make CPU IQ dispatch independent of draft occupancy
open · @CC-David-CC · 0 commentaires · Sur GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentation
Description
Uses the same supported CPU IQ kernel for a token alone and for a group of draft tokens. The previous default switched arithmetic at two tokens, allowing group occupancy to change rounding even when the accepted prefix and rollback were correct. The default now matches `STRATA_IQ_MT_MIN=1`, including IQ4_NL down rows. `STRATA_IQ_MT_MIN=2` retains the legacy behavior for comparison; `STRATA_IQ3S_MT1` is redundant. Refreshed onto current main (`61b3fb5d`). Direct native CPU checks compare each token's gate/up and down rows bit for bit across widths 1 through 8. Default, forced AVX2 and ggml controls pass; the explicit legacy threshold fails the negative control as expected. CUDA/HIP/SYCL builds and CPU checks passed. CUDA/HIP GPU synthetic format comparisons use numerical tolerances and do not prove end-to-end speculative token identity. This is a correctness tradeoff, not an unconditional speed improvement. Fresh Ryzen 7600X3D one-token CPU expert microbenchmarks measured +68.61% time for IQ2_XS, +17.65% IQ2_XXS, +13.22% IQ3_XXS and +10.03% IQ3_S; IQ2_S and IQ4_XS were approximately unchanged. Three processes per setting alternate default/legacy order; these are per-expert costs, not whole-model decode regressions. The fresh 14-request mixed CPU/GPU no-draft/single-token/drafted comparison passes exactly when approximate tail skipping is disabled (`STRATA_ROUTE_TAIL_SKIP=0`). Earlier default-policy divergences are retained and explained: rank-7 skipping uses minimum expert rank across the token window, so adding draft rows can change target contributions. This PR preserves ordinary tail-skip defaults. The controlled main pipeline matrix also passes; suffix-on-pipeline coverage remains incomplete. These finite controls do not claim arbitrary model, CPU/GPU or pipeline parity. [Fresh results, full token records, diagnostic controls and all cost logs](https://github.com/CC-David-CC/Strata-a5500/blob/18d3a37f/bench/results/2026-10-11-draft-independent-iq/README.md). [Historical validation on its original source](https://github.com/CC-David-CC/Strata-a5500/blob/18d3a37f/bench/results/2026-10-10-draft-independent-iq/README.md). No deployed server configuration was changed.
Sur le site
Liens install, modèles, releases.