#5217 [Other] Control: recommend_context counts the resident plan's expert arbitration
open · @Flo5k5 · 0 comments · View on GitHub
BenchmarksSetup & installModels & quants
Description
## Problem and result `recommend_context()` returns 32768 below 96 GB of RAM. On a 64 GB Mac that reads as a memory wall, but it is not one: the resident plan arbitrates expert-cache slots against the context's prompt buffers on its own, so a bigger window is a speed trade, not a failure. The setup was recommending the pure-speed pick (every expert resident) without saying so, and people on the 64 GB class who need a longer window for agent sessions had no signal that 64K or 128K were one flag away. After this change the recommendation is 65536 below 96 GB and 131072 from 96 GB up. ## Related discussion All the measurements are in #5176 (two comments today: the 32K/64K comparison, then the 128K step). Maintainer's agreement on the new defaults: not yet - the numbers are here for the review; if the 32K-every-expert pick should stay the default below 96 GB, the docstring alone would still be worth taking. ## Approach `recommend_context` changes its two steps, with the measurements in its docstring. Nothing else moves: the plan, the engine and the setup flow are untouched; a person who wants every expert still passes `--context 32768` explicitly. The 96 GB step keeps its meaning: from there the plan holds every expert beside 65536, so the recommendation can take the next window. 262144 boots on a 64 GB Mac (79% of the experts, the engine itself prints a LOW warning) but is not recommended below 96 GB. ## Validation **Passed:** - `python -m unittest tools.test_setup_mac` - revision c1ba3795, Python 3.15 on macOS: 32 tests OK (the two that pin the recommendation updated with it). - The measurements behind the change (M5 Pro 64 GB, Gyro-M TQ2_0, `--expert-cache 24576 --mmap-experts --draft dflash2`, standard bench, 3 runs per case, medians): 32K 24576/24576 experts, 44.0 tok/s; 64K 21593/24576 (88%), 34.7 tok/s; 128K 20859/24576 (85%), 33.0 tok/s; sanity 12/12 at every step; a real 129,224-token request read at 497 tok/s and answered correctly at 131072. **Failed:** - none observed. **Not run:** - The recommendation change on other RAM sizes (48 GB, 96 GB, 128 GB): only the 64 GB Mac was available; the function is a step function, so the risk is the step placement, not the code. ## Limits The trade (-21% / -25% of median decode against the all-expert 32K) is measured on one model (Gyro-M TQ2_0); a denser or sparser pack shifts where the steps sit. A 262144 request is being measured now (the KV at peak is the open question) - results will follow in #5176.
Related on this site
Quick links to related pages.