Issues / #1923
#1923 Engine-side expert-cache eviction policy (LRU / LFU-decay), opt-in; measured headroom on gfx906
open · @phoenixclyde · 2 comments · View on GitHub
Description
Context ------- `ExpertCache::admit` never evicts. `include/strata/core/expert_cache.hpp:143` says: "it never evicts, because eviction policy is a measured question (R4.1's LFU-decay vs LRU sweep)". `tools/bench_eviction.py` is the harness for that measurement, and the policies it defines (fill-only, lru, lfu-decay, oracle) are exactly the candidates. Measurement ----------- On a gfx906 / Radeon Instinct MI50 32GB, UD-IQ4_XS pack, engine 0.1.42, we replayed a `--dump-routing` trace (3,838 positions x 48 layers x k=10 = 1,842,240 (layer,expert) lookups) through `tools/bench_eviction.py`, prefilled with the production `expert-profile.bin`: slots fill-only (today) lru lfu-decay oracle 9,761 (production) 0.7086 0.9751 0.9767 0.9890 64 0.0105 0.3931 0.0481 0.4221 At the production slot count the current policy leaves about 24-28 points of hit rate on the table, and two policies that already exist in your own tool recover about 95% of the oracle bound. At small slot counts the gap is far larger (slots=64: 0.0105 vs 0.4221), i.e. today's behaviour is mostly a function of "are the slots big enough", not of any policy. Caveat: these are hit-rate figures from offline replay, not wall-clock; the speed benefit depends on the miss cost (PCIe / CPU pool share), which we have not measured live yet. The trace is one text-continuation workload, not the full production distribution. Oracle reads the future and is not implementable; it is only the upper bound. Note on answers: an eviction policy changes WHICH experts are computed on the GPU (vs read from RAM/files and computed on the CPU). The engine already documents that the GPU and the CPU round differently and that a reply "can differ slightly" (bench/results/2026-09-27-cache-parity). So this is NOT answer-neutral by construction; adoption would need a quality gate, the same way the other answers-can-differ switches are handled. Ask --- Would an opt-in eviction policy for the VRAM expert cache be in scope? Concretely, a `STRATA_EXPERT_EVICT=lru|lfu-decay` environment switch (default off = today's fill-only), with the policy implemented to match `tools/bench_eviction.py` exactly so the offline prediction and the live engine stay comparable. We are happy to prepare the patch if that helps.
Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.