#705 Feat/k8v4 kv streaming

closed · @merbanan · 0 comments · View on GitHub

Description

`--kv k8v4` (8-bit K, rotated 4-bit V) could not stream: the engine refused `--kv k8v4 --kv-resident N` and setup turned streaming off for it. This adds streaming for the hybrid format, so K8V4's smaller KV cache and the freed VRAM of streaming add up.

Related on strata.com

Install Strata
Model sizes & RAM
Release notes

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.