Issues / #1923
#1923 Engine-side expert-cache eviction policy (LRU / LFU-decay), opt-in; measured headroom on gfx906
open · @phoenixclyde · 2 comentários · No GitHub
Descrição
Context ------- `ExpertCache::admit` never evicts. `include/strata/core/expert_cache.hpp:143` says: "it never evicts, because eviction policy is a measured question (R4.1's LFU-decay vs LRU sweep)". `tools/bench_eviction.py` is the harness for that measurement, and the policies it defines (fill-only, lru, lfu-decay, oracle) are exactly the candidates. Measurement ----------- On a gfx906 / Radeon Instinct MI50 32GB, UD-IQ4_XS pack, engine 0.1.42, we replayed a `--dump-routing` trace (3,838 positions x 48 layers x k=10 = 1,842,240 (layer,expert) lookups) through `tools/bench_eviction.py`, prefilled with the production `expert-profile.bin`: slots fill-only (today) lru lfu-decay oracle 9,761 (production) 0.7086 0.9751 0.9767 0.9890 64 0.0105 0.3931 0.0481 0.4221 At the production slot count the current policy leaves about 24-28 points of hit rate on the table, and two policies that already exist in your own tool recover about 95% of the oracle bound. At small slot counts the gap is far larger (slots=64: 0.0105 vs 0.4221), i.e. today's behaviour is mostly a function of "are the slots big enough", not of any policy. Caveat: these are hit-rate figures from offline replay, not wall-clock; the speed benefit depends on the miss cost (PCIe / CPU pool share), which we have not measured live yet. The trace is one text-continuation workload, not the full production distribution. Oracle reads the future and is not implementable; it is only the upper bound. Note on answers: an eviction policy changes WHICH experts are computed on the GPU (vs read from RAM/files and computed on the CPU). The engine already documents that the GPU and the CPU round differently and that a reply "can differ slightly" (bench/results/2026-09-27-cache-parity). So this is NOT answer-neutral by construction; adoption would need a quality gate, the same way the other answers-can-differ switches are handled. Ask --- Would an opt-in eviction policy for the VRAM expert cache be in scope? Concretely, a `STRATA_EXPERT_EVICT=lru|lfu-decay` environment switch (default off = today's fill-only), with the policy implemented to match `tools/bench_eviction.py` exactly so the offline prediction and the live engine stay comparable. We are happy to prepare the patch if that helps.
No site
Links install, modelos, releases.