Pull requests / #2742

#2742 Qwen3.6: opt-in int8 K/V cache and single staging buffer

open · @ggeorgovassilis · 0 comentários · No GitHub

NVIDIA / CUDAModels & quantsDocumentation

Descrição

## Title
Qwen3.6: opt-in int8 K/V cache and single staging buffer

## Summary
Two opt-in improvements to the Qwen3.6-35B-A3B own-engine path, both off by default so the default path stays byte-identical to the last release:

1. **int8 (q8_0) K/V cache** behind `--kv int8` — roughly 4x smaller K/V cache (~44 KiB/token fp32 -> ~11 KiB/token int8), so an 8 GB card doubles its context (16K -> 32K).
2. **Single staging buffer** behind `STRATA_Q36_SINGLE_STAGE=1` — halves the prompt-streaming staging VRAM (one buffer instead of two double-buffered).

## What changed
- `CMakeLists.txt` — build wiring for the new q8_0 K/V kernels.
- `include/strata/kernels/q36_kv_q8.hpp` — new int8 K/V kernel interface.
- `src/kernels/cuda/q36_kv_q8.cu` — CUDA q8_0 K/V cache kernel.
- `src/kernels/q36_kv_q8_parity.cpp` — CPU reference kernel for parity testing.
- `src/qwen35/q36_engine.cu` — the `--kv int8` engine path; `STRATA_Q36_SINGLE_STAGE=1` (single staging buffer), numeric `STRATA_Q36_NO_MMQ`, `STRATA_Q36_PLACE_DEBUG`, and an attention-scores-buffer shrink (256 -> 64 MB) so staging placement still succeeds on tight VRAM.
- `docs/Q36_INT8_KV.md` — design note with the measured numbers.
- `docs/DETAILS.md` — links the new section into the table of contents.

## Extra Notes
- `--kv int8` defaults to `--kv fp32`, which is byte-identical to the previous release.
- Benchmarks were measured on an RTX 4060 (8 GB); see `docs/Q36_INT8_KV.md`.

No site

Links install, modelos, releases.