#6443 [Engine] Avoid redundant top-K scans in Metal expert read-ahead
open · @Maryarty · 0 comentarios · En GitHub
Descripción
## Problem and result Metal expert read-ahead scans the selected K entries for the weakest one before considering each remaining expert. If the candidate cannot replace an entry, the next iteration repeats the scan. Caching the minimum reduced host time outside I/O wait by 0.71 and 0.67 ms per window at read splits 1 and 2 on M1 Max. All runs produced the same token IDs. ## Related discussion None. This is a focused, opt-in change to the existing selector. ## Approach `STRATA_METAL_PREFETCH_CACHE_MIN=1` retains the weakest index and recomputes it after each replacement. Both paths use the same iteration order and strict comparisons, including ties. The switch defaults to off, and both paths use the existing read-ahead queue. ## Validation **Passed** - Release arm64 Metal build at `03d39285260d5fe30111bc2c83dca47af917cb08`, based on `3142b66`. `ctest --test-dir engine/build-metal -R '^runtime_hello$' --output-on-failure` passed on macOS 26.6.2 (25G83). - A harness compiled the two selectors with `clang++ -std=c++20 -O2` and compared 6,400 ordered selections. It covered 256 and 512 experts, every K from 1 to 32, and input containing ties, infinities and NaNs. - Four full-model CLI runs on Apple M1 Max with 32 GPU cores and 64 GiB. Qwen3.8-Flash-Next Q2_0 GSQ-RCO with PLE, streaming with a 12 GB arena, prefetch K=10, 2,048 fixed input IDs, 128 greedy output tokens, MTP width 3, context 4096 and chunk 512. The switch was off then on for split 1, and on then off for split 2. `STRATA_METAL_PROFILE=1` captured the counters. | Read split | Host time minus I/O wait, off → on | I/O wait, off → on | Decode, off → on | | --- | ---: | ---: | ---: | | 1 | 1.85 → 1.14 ms/window | 10.83 → 11.00 ms/window | 21.41 → 22.12 tok/s | | 2 | 1.79 → 1.12 ms/window | 14.45 → 14.45 ms/window | 20.12 → 20.55 tok/s | All four produced the same 128 token IDs and MTP acceptance (68/120), also matching the stock `3142b66` output at both splits. Zero new Swapouts. **Failed** None observed in the standalone patch checks. **Not run** Additional Apple generations and SSDs. The changed files belong to the Metal expert reader. ## Limits The profiler's host time minus wait includes other read management work. The end-to-end figures are single runs per setting and include SSD timing variation.
En el sitio
Enlaces a install, modelos, releases.