Pull requests / #1338
#1338 cuda: reuse secondary Q8 copies in native async refills
open · @CC-David-CC · 0 Kommentare · Auf GitHub
Beschreibung
Depends on perf/q8-compact-fill-main. Default-off native async D2D reuse. RTX PRO 6000 Blackwell, Q8/FP16 KV, 32K/128K input and 1K output: 51.4-64.4% of primary upload payload replaced; initial decode changes +0.94-3.84%. All four token streams match; 32K MTP work differs. One observation per arm. Lifetime/sanitizer and STOP checks passed. 
Mehr auf der Site
Links zu Install, Modellen, Releases.