#6660 [Engine] Per-layer VRAM expert-cache admission is measured 2.97% vs 21-70% but left off by default

open · @xvcezrqpbxtpzblrt · 0 コメント · GitHub で見る

Setup & installNVIDIA / CUDAModels & quants

本文

Thanks for the report. Please read the contributing guide first and search open and closed issues before filing.

**Write only what you saw.** Where a claim below is from reading code rather than a run, it is marked as static analysis in Reproduction status.

> This report is a static source review of the official tree. It was not reproduced on a running build; Reproduction status below says so explicitly.

### Area


[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?


Source (the engine's VRAM expert cache)

### What happened
The engine already contains a per-layer expert-cache slot admission policy (`--expert-cache-per-layer` / `ExpertCache::set_per_layer_admission`). It is implemented, unit-tested and measured by the project itself, but it is **off by default**. With the default policy every layer takes slots from one shared, arrival-ordered counter: the first `n_slots` distinct `(layer, expert)` pairs are roughly 26 layers of router position 0, so the whole cache is filled inside a few layers and never changes for the rest of the run.

The project's own numbers in the source say this is the difference between a cache that works and one that mostly does not:

- `engine/include/strata/core/expert_cache.hpp:177-179`: at `--expert-cache 256`, the default policy measured **1781/60000 = 2.97%** hits, while 8 slots per layer is **21.4%** and 64 per layer is **70.4%**. "The cache was never short of capacity — it was handing all of it to two layers."
- `engine/src/program/generate.cpp:1398-1401` (built-in `--help`): at 4096 slots **with** per-layer on, **54.4% hits**, CPU-pool drain **19.1 -> 10.3 ms/token**, **-2.7 ms/token end to end**.
- `engine/src/program/generate.cpp:886-893`: turning the cache on moves work off the CPU pool (drain 19.076 -> 10.312 ms/token), but the earlier round "found nothing" *because* the admission policy gave every slot to the first position.

A user who runs the engine without the opt-in flag gets the 2.97% policy even though the 21.4–70.4% policy is already in the binary. No project code path enables it automatically.

### Smallest steps or command that reproduce it
1. Start the engine with a VRAM expert tier, e.g. `--expert-cache 256` (or `--expert-cache auto`), with or without `--expert-profile`.
2. Do **not** pass `--expert-cache-per-layer`.
3. Look at the startup log. It prints `strata generate: expert cache N slots, X GiB of VRAM; policy is` followed by `compulsory-miss (fills with whatever the run routes first).` or `PROFILE, ranked by routing frequency, no eviction.` — but **never** `R4.2g PER-LAYER: each layer owns N slots (...)`.
4. Restart with `--expert-cache-per-layer` added. The line `R4.2g PER-LAYER: each layer owns N slots (lo..hi).` now appears; the same model and routing use the measured higher-hit-rate policy.
5. Static confirmation of the default: `engine/include/strata/core/expert_cache.hpp:275` has `bool per_layer_ = false;` and `engine/src/program/generate.cpp:905` has `bool expert_cache_per_layer = false;`. The only writer is the flag parser at `engine/src/program/generate.cpp:4120`.

### Reproduction status
Unconfirmed static analysis (I read code or ran a scanner; I did not see it fail)
### Strata version

0.2.19 · engine 0.1.65 (source tree under review, static analysis; not a running-build query)

### Graphics card, memory and system
Not measured locally. No local hardware claim is made; the quoted measurements are the ones the source attributes to its own rounds.

### Model, size and settings
Not measured locally. Relevant switches: `--expert-cache N|auto`, `--expert-cache-per-layer`, and optional `--expert-profile`. The policy applies to both uniform and sized (native-pack) caches.

### Logs
Static evidence — verbatim source excerpts (repository-relative paths):

`engine/include/strata/core/expert_cache.hpp:169-186`

```cpp
    /// **R4.2g: GIVE EACH LAYER ITS OWN SLOTS.  ROUND 328 MEASURED WHY THE GLOBAL FORM CANNOT WORK.**
    ///
    /// `admit`'s original policy hands out slots in ARRIVAL ORDER from one counter shared by every layer
    /// (`if (next_free_ >= slots_) return kNotResident;`).  A position looks at 10 experts in each of 48
    /// layers, so the first `n_slots` distinct (layer, expert) pairs are **about 26 layers of POSITION 0** -
    /// the cache fills entirely inside the first position it ever sees and never changes again, so hits are
    /// confined to those few layers for the rest of the run.
    ///
    /// Measured, `--expert-cache 256`: **1781 of 60000 = 2.97%**.  `Memory/cache_allocation.py` scores the
    /// same run's routing and says **8 slots per layer is worth 21.4%** and 64 is worth **70.4%**.  The cache
    /// was never short of capacity - it was handing all of it to two layers.
    ///
    /// With this on, layer `l` may only use slots `[l*q, (l+1)*q)` where `q = slots()/n_layers`, the last
    /// layer taking the remainder.  **EVICTION IS STILL ABSENT** - the property the comment above calls
    /// deliberate is preserved exactly; only WHICH slot a layer may take changes.  Default off, so every
    /// caller that predates this flag is bit-for-bit unaffected.
    void set_per_layer_admission(bool on) { per_layer_ = on; }
    bool per_layer_admission() const { return per_layer_; }
```

`engine/include/strata/core/expert_cache.hpp:273-275`

```cpp
    /// R4.2g.  `per_layer_` off (the default) leaves `next_free_` as the only admission counter, so the
    /// pre-existing path is untouched.
    bool per_layer_ = false;
```

`engine/src/program/generate.cpp:886-905`

```cpp
    /// R4: slots of VRAM-resident experts.  **0 = off, and off is the default.**
    /// **THE COMMENT THAT USED TO BE HERE WAS FALSE AND ROUND 328 MEASURED IT.**  It said "the cache has no
    /// consumer yet - `moe_hit_grouped_s2` does not exist - so switching it on costs the fill traffic and
    /// saves nothing".  The kernel exists (`src/kernels/cuda/s2_expert_grouped.cu`), it is wired at line ~660
    /// via `expert_hit_run`, and switching the cache on **does** move work off the CPU pool: the drain fell
    /// **19.076 -> 10.312 ms/token** at 4096 per-layer slots, for **-2.7 ms/token** end to end.  What was
    /// true is that the ADMISSION POLICY gave every slot to the first position, which is why the earlier
    /// measurement found nothing - see `expert_cache_per_layer`.
    int expert_cache = 0;
```

```cpp
    /// **R4.2g.  ROUND 328 MEASURED THAT THE GLOBAL ADMISSION POLICY CANNOT WORK, AND THIS IS THE FIX.**
    /// The default policy hands out slots in arrival order from one counter shared by all 48 layers, so the
    /// first `n_slots` distinct pairs - about 26 LAYERS OF POSITION 0 - take every slot and hits are confined
    /// to them.  Measured at 256 slots: **1781 of 60000 = 2.97%**, against **21.4%** for 8 slots per layer and
    /// **70.4%** for 64, from `Memory/cache_allocation.py` on the same run's routing.  Off by default.
    bool expert_cache_per_layer = false;
```

`engine/src/program/generate.cpp:1398-1401` (built-in help text)

```cpp
                  "  --expert-cache N     R4: keep N expert blobs resident in VRAM and compute their rows on the\n"
                  "                       GPU via `moe_hit_grouped_s2`.  DEFAULT 0.  Measured at 4096 slots\n"
                  "                       with --expert-cache-per-layer: 54.4%% hits, CPU pool drain 19.1 -> 10.3\n"
                  "                       ms/token, -2.7 ms/token end to end.\n"
```

`engine/src/program/generate.cpp:1416-1421` (built-in help text)

```cpp
                  "  --expert-cache-per-layer  R4.2g: give each layer its OWN slots instead of letting the first\n"
                  "                       position take all of them.  The default policy fills in arrival order\n"
                  "                       from one shared counter, so 256 slots went to ~26 layers of position 0\n"
                  "                       and measured **2.97%%**.  Per-layer, the same routing gives 21.4%% at 8\n"
                  "                       slots/layer and 70.4%% at 64.  On a native pack, N is still the budget of N\n"
                  "                       largest blobs; each layer's slots are that layer's own blob.\n"
```

`engine/src/program/generate.cpp:4120` (the only place the flag is set)

```cpp
        else if (a == "--expert-cache-per-layer") o.expert_cache_per_layer = true;
```

`engine/src/program/generate.cpp:9359`, `9370-9379` (the policy is passed straight through, and the log line that tells users which one they got)

```cpp
        xcache.set_per_layer_admission(o.expert_cache_per_layer);
```

```cpp
        if (o.expert_cache_per_layer) {
            int64_t lo = 0, hi = 0;
            xcache.layer_slot_range(0, lo, hi);
            std::fprintf(stderr, "                 R4.2g PER-LAYER: each layer owns %lld slots (%lld..%lld).\n",
                         (long long) (hi - lo), (long long) lo, (long long) (hi - 1));
        } else if (profile.empty()) {
            std::fprintf(stderr, "                 compulsory-miss (fills with whatever the run routes first).\n");
        } else {
            std::fprintf(stderr, "                 PROFILE, ranked by routing frequency, no eviction.\n");
        }
```

`engine/src/program/generate.cpp:9399-9401` (the profile-fill path already honours it, so enabling it by default does not break profile builds)

```cpp
        const bool per_layer = xcache.per_layer_admission();
        const int64_t want = per_layer ? (int64_t) profile.size()
                                       : std::min<int64_t>((int64_t) profile.size(), xcache.slots());
```

`engine/tests/core/expert_cache_per_layer_test.cpp:1-3` (the policy is covered for both uniform and sized caches)

```cpp
// #369: per-layer admission on a sized cache (open_sized, every native pack) - each layer admits into its own slot
// range, as on the uniform cache, so no two layers write the same slot.  Needs a CUDA device (exits 77 without one).
```

### Why this is worth fixing for everyone
This is not a workload-, card- or quant-specific tweak. It changes *which layer a VRAM slot belongs to*, a property of the cache allocator, and the project's own round-328 measurement holds for the same routing across the board: 2.97% versus 21.4–70.4% at 256 slots, and 54.4% hits with a 2.7 ms/token end-to-end gain at 4096 slots.

Anyone who actually gets a VRAM expert tier is currently put on the losing arm unless they know to pass the opt-in flag. That includes the setup/desktop configurations that pass `--expert-cache auto`, and any run with `--expert-profile` (where `engine/src/program/generate.cpp:6580` turns `--expert-cache 0` into auto: `if (o.expert_cache == 0) o.expert_cache = -1;`). A user with no profile and no explicit cache size gets no cache at all, which is a separate choice; the point here is that **every user who does have the cache pays the 2.97% policy by default**.

The CPU expert pool is the engine's largest single cost (`engine/include/strata/core/expert_cache.hpp:3-7`: 663.6 MB/token at ~40 GB/s = 16.2 ms on a ~53 ms token). Raising the hit rate directly removes CPU expert reads, so this is a general speed-up for generation, not a niche one. The change is safe to make default: it adds no eviction, does not touch expert bytes, routing or numerics, and the existing parity note (`engine/src/program/generate.cpp:9360-9369`) already documents that GPU-versus-CPU rounding is equal quality.

### Root cause
`ExpertCache::admit` uses a single `next_free_` counter for all layers unless `per_layer_` is set (`engine/include/strata/core/expert_cache.hpp:271-275, 273-275`), so the first position's experts occupy the whole arena. The fix, `per_layer_`, partitions the arena into per-layer ranges `[l*q, (l+1)*q)` (`expert_cache.hpp:181-185`) and is fully wired through both the engine and the profile-fill path (`generate.cpp:9359, 9399-9401`). The o

関連リンク

インストール・モデル・リリースへのサイト内リンク。