Pull requests / #1117
#1117 serve: --elastic, an automatic elastic expert cache with a fixed core (opt-in; resubmission of #1030 on the new main)
open · @imanu86 · 0 Kommentare · Auf GitHub
Server & APINVIDIA / CUDAModels & quantsDocumentationWindows
Beschreibung
Resubmission of #1030 (closed by the history cleanup), cherry-picked onto the new `main` (82f46a8) as you suggested, and brought up to 0.1.40. **What it is.** `--elastic` (opt-in): an automatic elastic expert cache for a card that is shared with desktop programs whose VRAM use comes and goes while the server runs. The cache is a VMM arena: a fixed core filled from the profile, plus a tail that grows when VRAM has stayed free and shrinks (in steps, between decode rounds) when it is needed back, keeping a reserve free. It never opens together with `--vram-elastic` (#533), which stays the manual control. Design notes: `docs/ELASTIC_CACHE.md`. **Since #1030** - rebased on 0.1.40: the cache's flag is `elastic_vmm_` (0.1.40 added `std::unique_ptr<VmmRange> vmm_`); - works with `--batch`: the elastic step also runs between two batch windows (refused only with `--batch-groups > 1`); - at startup an arena that does not fit (the last chunks' `cuMemCreate` out of memory) retries up to 4 times, 512 MiB smaller each, before falling back to the fixed cache; - `--kv-grow` stays off with `--elastic` (the K/V would be sized before the arena that it grows into). **Verified** (RTX 2080 Ti 22 GB, sm_75, Windows, CUDA 12.6, Qwen3.8-Flash-Next IQ3_XXS): - without `--elastic`, fixed-placement teacher forcing (two prompts, 169 and 595 positions) is byte-identical to upstream 0.1.40; - the cache CTests pass (`expert_cache_elastic_test`, `_per_layer_test`, `_segmented_test`, `expert_profile_save_test`); - on our fork the same code has run as the daily default since September 30; under a 3 GB allocation by another process the tail shrinks between rounds (8.3 -> 5.7 GB) while two answers keep going, then grows back. Happy to split it into smaller pieces if you prefer (for example the arena alone). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Mehr auf der Site
Links zu Install, Modellen, Releases.