Pull requests / #2742
#2742 Qwen3.6: opt-in int8 K/V cache and single staging buffer
open · @ggeorgovassilis · 0 commentaires · Sur GitHub
NVIDIA / CUDAModels & quantsDocumentation
Description
## Title Qwen3.6: opt-in int8 K/V cache and single staging buffer ## Summary Two opt-in improvements to the Qwen3.6-35B-A3B own-engine path, both off by default so the default path stays byte-identical to the last release: 1. **int8 (q8_0) K/V cache** behind `--kv int8` — roughly 4x smaller K/V cache (~44 KiB/token fp32 -> ~11 KiB/token int8), so an 8 GB card doubles its context (16K -> 32K). 2. **Single staging buffer** behind `STRATA_Q36_SINGLE_STAGE=1` — halves the prompt-streaming staging VRAM (one buffer instead of two double-buffered). ## What changed - `CMakeLists.txt` — build wiring for the new q8_0 K/V kernels. - `include/strata/kernels/q36_kv_q8.hpp` — new int8 K/V kernel interface. - `src/kernels/cuda/q36_kv_q8.cu` — CUDA q8_0 K/V cache kernel. - `src/kernels/q36_kv_q8_parity.cpp` — CPU reference kernel for parity testing. - `src/qwen35/q36_engine.cu` — the `--kv int8` engine path; `STRATA_Q36_SINGLE_STAGE=1` (single staging buffer), numeric `STRATA_Q36_NO_MMQ`, `STRATA_Q36_PLACE_DEBUG`, and an attention-scores-buffer shrink (256 -> 64 MB) so staging placement still succeeds on tight VRAM. - `docs/Q36_INT8_KV.md` — design note with the measured numbers. - `docs/DETAILS.md` — links the new section into the table of contents. ## Extra Notes - `--kv int8` defaults to `--kv fp32`, which is byte-identical to the previous release. - Benchmarks were measured on an RTX 4060 (8 GB); see `docs/Q36_INT8_KV.md`.
Sur le site
Liens install, modèles, releases.