Issues / #1923

#1923 Engine-side expert-cache eviction policy (LRU / LFU-decay), opt-in; measured headroom on gfx906

open · @phoenixclyde · 2 comments · View on GitHub

BenchmarksModels & quants

Description

Context
-------
`ExpertCache::admit` never evicts.  `include/strata/core/expert_cache.hpp:143` says: "it never evicts,
because eviction policy is a measured question (R4.1's LFU-decay vs LRU sweep)".  `tools/bench_eviction.py`
is the harness for that measurement, and the policies it defines (fill-only, lru, lfu-decay, oracle) are
exactly the candidates.

Measurement
-----------
On a gfx906 / Radeon Instinct MI50 32GB, UD-IQ4_XS pack, engine 0.1.42, we replayed a `--dump-routing`
trace (3,838 positions x 48 layers x k=10 = 1,842,240 (layer,expert) lookups) through
`tools/bench_eviction.py`, prefilled with the production `expert-profile.bin`:

  slots                     fill-only (today)   lru      lfu-decay   oracle
  9,761  (production)       0.7086              0.9751   0.9767      0.9890
  64                        0.0105              0.3931   0.0481      0.4221

At the production slot count the current policy leaves about 24-28 points of hit rate on the table, and
two policies that already exist in your own tool recover about 95% of the oracle bound.  At small slot
counts the gap is far larger (slots=64: 0.0105 vs 0.4221), i.e. today's behaviour is mostly a function of
"are the slots big enough", not of any policy.

Caveat: these are hit-rate figures from offline replay, not wall-clock; the speed benefit depends on the
miss cost (PCIe / CPU pool share), which we have not measured live yet.  The trace is one text-continuation
workload, not the full production distribution.  Oracle reads the future and is not implementable; it is
only the upper bound.

Note on answers: an eviction policy changes WHICH experts are computed on the GPU (vs read from
RAM/files and computed on the CPU).  The engine already documents that the GPU and the CPU round
differently and that a reply "can differ slightly" (bench/results/2026-09-27-cache-parity).  So this
is NOT answer-neutral by construction; adoption would need a quality gate, the same way the other
answers-can-differ switches are handled.

Ask
---
Would an opt-in eviction policy for the VRAM expert cache be in scope?  Concretely, a
`STRATA_EXPERT_EVICT=lru|lfu-decay` environment switch (default off = today's fill-only), with the policy
implemented to match `tools/bench_eviction.py` exactly so the offline prediction and the live engine stay
comparable.  We are happy to prepare the patch if that helps.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.