反馈 / #1923

#1923 Engine-side expert-cache eviction policy (LRU / LFU-decay), opt-in; measured headroom on gfx906

open · @phoenixclyde · 2 评论 · 去 GitHub 看

BenchmarksModels & quants

说明

Context
-------
`ExpertCache::admit` never evicts.  `include/strata/core/expert_cache.hpp:143` says: "it never evicts,
because eviction policy is a measured question (R4.1's LFU-decay vs LRU sweep)".  `tools/bench_eviction.py`
is the harness for that measurement, and the policies it defines (fill-only, lru, lfu-decay, oracle) are
exactly the candidates.

Measurement
-----------
On a gfx906 / Radeon Instinct MI50 32GB, UD-IQ4_XS pack, engine 0.1.42, we replayed a `--dump-routing`
trace (3,838 positions x 48 layers x k=10 = 1,842,240 (layer,expert) lookups) through
`tools/bench_eviction.py`, prefilled with the production `expert-profile.bin`:

  slots                     fill-only (today)   lru      lfu-decay   oracle
  9,761  (production)       0.7086              0.9751   0.9767      0.9890
  64                        0.0105              0.3931   0.0481      0.4221

At the production slot count the current policy leaves about 24-28 points of hit rate on the table, and
two policies that already exist in your own tool recover about 95% of the oracle bound.  At small slot
counts the gap is far larger (slots=64: 0.0105 vs 0.4221), i.e. today's behaviour is mostly a function of
"are the slots big enough", not of any policy.

Caveat: these are hit-rate figures from offline replay, not wall-clock; the speed benefit depends on the
miss cost (PCIe / CPU pool share), which we have not measured live yet.  The trace is one text-continuation
workload, not the full production distribution.  Oracle reads the future and is not implementable; it is
only the upper bound.

Note on answers: an eviction policy changes WHICH experts are computed on the GPU (vs read from
RAM/files and computed on the CPU).  The engine already documents that the GPU and the CPU round
differently and that a reply "can differ slightly" (bench/results/2026-09-27-cache-parity).  So this
is NOT answer-neutral by construction; adoption would need a quality gate, the same way the other
answers-can-differ switches are handled.

Ask
---
Would an opt-in eviction policy for the VRAM expert cache be in scope?  Concretely, a
`STRATA_EXPERT_EVICT=lru|lfu-decay` environment switch (default off = today's fill-only), with the policy
implemented to match `tools/bench_eviction.py` exactly so the offline prediction and the live engine stay
comparable.  We are happy to prepare the patch if that helps.

本站相关内容

相关页面的快捷入口。