#6660 [Engine] Per-layer VRAM expert-cache admission is measured 2.97% vs 21-70% but left off by default
open · @xvcezrqpbxtpzblrt · 0 Kommentare · Auf GitHub
Setup & installNVIDIA / CUDAModels & quants
Beschreibung
Thanks for the report. Please read the contributing guide first and search open and closed issues before filing.
**Write only what you saw.** Where a claim below is from reading code rather than a run, it is marked as static analysis in Reproduction status.
> This report is a static source review of the official tree. It was not reproduced on a running build; Reproduction status below says so explicitly.
### Area
[Engine] (speed, kernels, models, the server and API)
### Desktop app, Terminal or source?
Source (the engine's VRAM expert cache)
### What happened
The engine already contains a per-layer expert-cache slot admission policy (`--expert-cache-per-layer` / `ExpertCache::set_per_layer_admission`). It is implemented, unit-tested and measured by the project itself, but it is **off by default**. With the default policy every layer takes slots from one shared, arrival-ordered counter: the first `n_slots` distinct `(layer, expert)` pairs are roughly 26 layers of router position 0, so the whole cache is filled inside a few layers and never changes for the rest of the run.
The project's own numbers in the source say this is the difference between a cache that works and one that mostly does not:
- `engine/include/strata/core/expert_cache.hpp:177-179`: at `--expert-cache 256`, the default policy measured **1781/60000 = 2.97%** hits, while 8 slots per layer is **21.4%** and 64 per layer is **70.4%**. "The cache was never short of capacity — it was handing all of it to two layers."
- `engine/src/program/generate.cpp:1398-1401` (built-in `--help`): at 4096 slots **with** per-layer on, **54.4% hits**, CPU-pool drain **19.1 -> 10.3 ms/token**, **-2.7 ms/token end to end**.
- `engine/src/program/generate.cpp:886-893`: turning the cache on moves work off the CPU pool (drain 19.076 -> 10.312 ms/token), but the earlier round "found nothing" *because* the admission policy gave every slot to the first position.
A user who runs the engine without the opt-in flag gets the 2.97% policy even though the 21.4–70.4% policy is already in the binary. No project code path enables it automatically.
### Smallest steps or command that reproduce it
1. Start the engine with a VRAM expert tier, e.g. `--expert-cache 256` (or `--expert-cache auto`), with or without `--expert-profile`.
2. Do **not** pass `--expert-cache-per-layer`.
3. Look at the startup log. It prints `strata generate: expert cache N slots, X GiB of VRAM; policy is` followed by `compulsory-miss (fills with whatever the run routes first).` or `PROFILE, ranked by routing frequency, no eviction.` — but **never** `R4.2g PER-LAYER: each layer owns N slots (...)`.
4. Restart with `--expert-cache-per-layer` added. The line `R4.2g PER-LAYER: each layer owns N slots (lo..hi).` now appears; the same model and routing use the measured higher-hit-rate policy.
5. Static confirmation of the default: `engine/include/strata/core/expert_cache.hpp:275` has `bool per_layer_ = false;` and `engine/src/program/generate.cpp:905` has `bool expert_cache_per_layer = false;`. The only writer is the flag parser at `engine/src/program/generate.cpp:4120`.
### Reproduction status
Unconfirmed static analysis (I read code or ran a scanner; I did not see it fail)
### Strata version
0.2.19 · engine 0.1.65 (source tree under review, static analysis; not a running-build query)
### Graphics card, memory and system
Not measured locally. No local hardware claim is made; the quoted measurements are the ones the source attributes to its own rounds.
### Model, size and settings
Not measured locally. Relevant switches: `--expert-cache N|auto`, `--expert-cache-per-layer`, and optional `--expert-profile`. The policy applies to both uniform and sized (native-pack) caches.
### Logs
Static evidence — verbatim source excerpts (repository-relative paths):
`engine/include/strata/core/expert_cache.hpp:169-186`
```cpp
/// **R4.2g: GIVE EACH LAYER ITS OWN SLOTS. ROUND 328 MEASURED WHY THE GLOBAL FORM CANNOT WORK.**
///
/// `admit`'s original policy hands out slots in ARRIVAL ORDER from one counter shared by every layer
/// (`if (next_free_ >= slots_) return kNotResident;`). A position looks at 10 experts in each of 48
/// layers, so the first `n_slots` distinct (layer, expert) pairs are **about 26 layers of POSITION 0** -
/// the cache fills entirely inside the first position it ever sees and never changes again, so hits are
/// confined to those few layers for the rest of the run.
///
/// Measured, `--expert-cache 256`: **1781 of 60000 = 2.97%**. `Memory/cache_allocation.py` scores the
/// same run's routing and says **8 slots per layer is worth 21.4%** and 64 is worth **70.4%**. The cache
/// was never short of capacity - it was handing all of it to two layers.
///
/// With this on, layer `l` may only use slots `[l*q, (l+1)*q)` where `q = slots()/n_layers`, the last
/// layer taking the remainder. **EVICTION IS STILL ABSENT** - the property the comment above calls
/// deliberate is preserved exactly; only WHICH slot a layer may take changes. Default off, so every
/// caller that predates this flag is bit-for-bit unaffected.
void set_per_layer_admission(bool on) { per_layer_ = on; }
bool per_layer_admission() const { return per_layer_; }
```
`engine/include/strata/core/expert_cache.hpp:273-275`
```cpp
/// R4.2g. `per_layer_` off (the default) leaves `next_free_` as the only admission counter, so the
/// pre-existing path is untouched.
bool per_layer_ = false;
```
`engine/src/program/generate.cpp:886-905`
```cpp
/// R4: slots of VRAM-resident experts. **0 = off, and off is the default.**
/// **THE COMMENT THAT USED TO BE HERE WAS FALSE AND ROUND 328 MEASURED IT.** It said "the cache has no
/// consumer yet - `moe_hit_grouped_s2` does not exist - so switching it on costs the fill traffic and
/// saves nothing". The kernel exists (`src/kernels/cuda/s2_expert_grouped.cu`), it is wired at line ~660
/// via `expert_hit_run`, and switching the cache on **does** move work off the CPU pool: the drain fell
/// **19.076 -> 10.312 ms/token** at 4096 per-layer slots, for **-2.7 ms/token** end to end. What was
/// true is that the ADMISSION POLICY gave every slot to the first position, which is why the earlier
/// measurement found nothing - see `expert_cache_per_layer`.
int expert_cache = 0;
```
```cpp
/// **R4.2g. ROUND 328 MEASURED THAT THE GLOBAL ADMISSION POLICY CANNOT WORK, AND THIS IS THE FIX.**
/// The default policy hands out slots in arrival order from one counter shared by all 48 layers, so the
/// first `n_slots` distinct pairs - about 26 LAYERS OF POSITION 0 - take every slot and hits are confined
/// to them. Measured at 256 slots: **1781 of 60000 = 2.97%**, against **21.4%** for 8 slots per layer and
/// **70.4%** for 64, from `Memory/cache_allocation.py` on the same run's routing. Off by default.
bool expert_cache_per_layer = false;
```
`engine/src/program/generate.cpp:1398-1401` (built-in help text)
```cpp
" --expert-cache N R4: keep N expert blobs resident in VRAM and compute their rows on the\n"
" GPU via `moe_hit_grouped_s2`. DEFAULT 0. Measured at 4096 slots\n"
" with --expert-cache-per-layer: 54.4%% hits, CPU pool drain 19.1 -> 10.3\n"
" ms/token, -2.7 ms/token end to end.\n"
```
`engine/src/program/generate.cpp:1416-1421` (built-in help text)
```cpp
" --expert-cache-per-layer R4.2g: give each layer its OWN slots instead of letting the first\n"
" position take all of them. The default policy fills in arrival order\n"
" from one shared counter, so 256 slots went to ~26 layers of position 0\n"
" and measured **2.97%%**. Per-layer, the same routing gives 21.4%% at 8\n"
" slots/layer and 70.4%% at 64. On a native pack, N is still the budget of N\n"
" largest blobs; each layer's slots are that layer's own blob.\n"
```
`engine/src/program/generate.cpp:4120` (the only place the flag is set)
```cpp
else if (a == "--expert-cache-per-layer") o.expert_cache_per_layer = true;
```
`engine/src/program/generate.cpp:9359`, `9370-9379` (the policy is passed straight through, and the log line that tells users which one they got)
```cpp
xcache.set_per_layer_admission(o.expert_cache_per_layer);
```
```cpp
if (o.expert_cache_per_layer) {
int64_t lo = 0, hi = 0;
xcache.layer_slot_range(0, lo, hi);
std::fprintf(stderr, " R4.2g PER-LAYER: each layer owns %lld slots (%lld..%lld).\n",
(long long) (hi - lo), (long long) lo, (long long) (hi - 1));
} else if (profile.empty()) {
std::fprintf(stderr, " compulsory-miss (fills with whatever the run routes first).\n");
} else {
std::fprintf(stderr, " PROFILE, ranked by routing frequency, no eviction.\n");
}
```
`engine/src/program/generate.cpp:9399-9401` (the profile-fill path already honours it, so enabling it by default does not break profile builds)
```cpp
const bool per_layer = xcache.per_layer_admission();
const int64_t want = per_layer ? (int64_t) profile.size()
: std::min<int64_t>((int64_t) profile.size(), xcache.slots());
```
`engine/tests/core/expert_cache_per_layer_test.cpp:1-3` (the policy is covered for both uniform and sized caches)
```cpp
// #369: per-layer admission on a sized cache (open_sized, every native pack) - each layer admits into its own slot
// range, as on the uniform cache, so no two layers write the same slot. Needs a CUDA device (exits 77 without one).
```
### Why this is worth fixing for everyone
This is not a workload-, card- or quant-specific tweak. It changes *which layer a VRAM slot belongs to*, a property of the cache allocator, and the project's own round-328 measurement holds for the same routing across the board: 2.97% versus 21.4–70.4% at 256 slots, and 54.4% hits with a 2.7 ms/token end-to-end gain at 4096 slots.
Anyone who actually gets a VRAM expert tier is currently put on the losing arm unless they know to pass the opt-in flag. That includes the setup/desktop configurations that pass `--expert-cache auto`, and any run with `--expert-profile` (where `engine/src/program/generate.cpp:6580` turns `--expert-cache 0` into auto: `if (o.expert_cache == 0) o.expert_cache = -1;`). A user with no profile and no explicit cache size gets no cache at all, which is a separate choice; the point here is that **every user who does have the cache pays the 2.97% policy by default**.
The CPU expert pool is the engine's largest single cost (`engine/include/strata/core/expert_cache.hpp:3-7`: 663.6 MB/token at ~40 GB/s = 16.2 ms on a ~53 ms token). Raising the hit rate directly removes CPU expert reads, so this is a general speed-up for generation, not a niche one. The change is safe to make default: it adds no eviction, does not touch expert bytes, routing or numerics, and the existing parity note (`engine/src/program/generate.cpp:9360-9369`) already documents that GPU-versus-CPU rounding is equal quality.
### Root cause
`ExpertCache::admit` uses a single `next_free_` counter for all layers unless `per_layer_` is set (`engine/include/strata/core/expert_cache.hpp:271-275, 273-275`), so the first position's experts occupy the whole arena. The fix, `per_layer_`, partitions the arena into per-layer ranges `[l*q, (l+1)*q)` (`expert_cache.hpp:181-185`) and is fully wired through both the engine and the profile-fill path (`generate.cpp:9359, 9399-9401`). The oMehr auf der Site
Links zu Install, Modellen, Releases.