Pull requests / #2775
#2775 Per-client expert memory
open · @apatassini · 0 comentários · No GitHub
BenchmarksNVIDIA / CUDAModels & quants
Descrição
When two programs share the server (an agent and a mail assistant), each request from the other one refills the VRAM cache with its experts, and the agent's next step finds the "wrong" card (measured 64.6% instead of 84% of picks on VRAM). `STRATA_CLIENT_MEMORY=N`: the engine recognises the client from the first 64 prompt tokens, keeps slow per-client usage counts (0.97 per step, up to 16 clients, LRU), and when a client comes back restores its counts and runs one adaptation of up to N swaps (also with `--adapt-async 1`). RTX 5070 Ti, Q2_0, 61-75K context: agent step right after another program +21% tok/s (+24% by ms), cache already adapted +11%, short code unchanged. Without the variable nothing changes. Full notes and measurements: https://github.com/apatassini/Strata/blob/qwen38-16gb/QWEN38.md 🤖 Generated with [Claude Code](https://claude.com/claude-code)
No site
Links install, modelos, releases.