Pull requests / #2775

#2775 Per-client expert memory

open · @apatassini · 0 comentarios · En GitHub

BenchmarksNVIDIA / CUDAModels & quants

Descripción

When two programs share the server (an agent and a mail assistant), each request from the other one refills the VRAM cache with
its experts, and the agent's next step finds the "wrong" card (measured 64.6% instead of 84% of picks on VRAM).
`STRATA_CLIENT_MEMORY=N`: the engine recognises the client from the first 64 prompt tokens, keeps slow per-client usage counts
(0.97 per step, up to 16 clients, LRU), and when a client comes back restores its counts and runs one adaptation of up to N swaps
(also with `--adapt-async 1`). RTX 5070 Ti, Q2_0, 61-75K context: agent step right after another program +21% tok/s (+24% by
ms), cache already adapted +11%, short code unchanged. Without the variable nothing changes.

Full notes and measurements: https://github.com/apatassini/Strata/blob/qwen38-16gb/QWEN38.md

🤖 Generated with [Claude Code](https://claude.com/claude-code)

En el sitio

Enlaces a install, modelos, releases.