Pull requests / #3392
#3392 [Engine] add opt-in advisory VRAM plan
open · draft · @j-luwierski · 0 commentaires · Sur GitHub
Multi-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityWindowsLinux
Description
## Problem and result
Before the expert cache takes the remaining VRAM, users have no diagnostic estimate of the prompt buffers or cache size the current sizing rules will choose. This adds a warnings-only advisory plan, enabled with `STRATA_ADVISORY_PLAN=1`. The unset default retains the existing startup output and runtime path. The existing sizing, loan scan, and error paths remain authoritative; the planner does not allocate, resize, or refuse startup.
## Related discussion
Continuation of the closed [#1133](https://github.com/Niko1221/Strata/pull/1133), closed during the 0.2.0-beta tracker reset rather than declined. It implements part C of [#796](https://github.com/Niko1221/Strata/issues/796), with scope previously agreed in that discussion: no startup refusal, runtime contract, or second allocator. Related foundational discussion: [#765](https://github.com/Niko1221/Strata/pull/765).
## Approach
The planner estimates owned-prefill buffers and the auto-loan/cache plan from the same pricing and sizing inputs. It then re-derives the estimate after cache open and reports runtime differences as warnings. It is opt-in to preserve the default path.
## Validation
Tested submitted tree `39f98ea1` based on current `main` `90767252`, on Linux, CUDA 13.4.92, CUDA arch 89, NVIDIA RTX 4070 Ti SUPER (16 GiB), IQ3_XXS.
**Passed:**
- `cmake -S . -B build-advisory -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DSTRATA_NATIVE_EXPERTS=ON -DSTRATA_BUILD_TESTS=ON -DCMAKE_CUDA_ARCHITECTURES=89` — configured successfully.
- `cmake --build build-advisory --target strata vram_plan_test -j 12` — engine and planner test built successfully.
- `ctest --test-dir build-advisory -R '^vram_plan_test$' --output-on-failure` — 1/1 passed.
- `git diff --check` — passed.
- Runtime, same model/config and a 12,288-ID prompt file, once with `STRATA_ADVISORY_PLAN` unset and once with `STRATA_ADVISORY_PLAN=1`: both exited 0, processed 12,287 prompt tokens in two chunks, chose an 8,192-token chunk and 2,489 loan slots, opened a 5,537-slot (8.98 GiB) cache, and generated token 198. The enabled run predicted those chunk, loan, and cache values exactly and emitted no mismatch warning. The unset run emitted no advisory diagnostics. One run per mode is not a performance comparison.
Runtime command (with `DATA` pointing to the local model data directory and `/tmp/strata-pr1133-prompt.txt` containing `198 ` repeated 12,288 times):
```sh
./build-advisory/strata --pack "${DATA}/packs/iq3_xxs" \
--native "${DATA}/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf" \
--ple-gguf "${DATA}/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf" \
--expert-profile data/expert-profile.bin --expert-cache auto --prefill auto \
--spec 4 --mtp "${DATA}/mtp/rt" --max-context 32768 --max-new 1 --kv int8 \
--tokens-file /tmp/strata-pr1133-prompt.txt
```
**Failed:** none observed.
**Not run:** HIP and SYCL builds (their compilers are unavailable in this environment); multi-GPU and Windows/WDDM runtime checks; full CTest suite.
## Limits
Hardware execution covers one single-GPU CUDA configuration. The advisory is diagnostic and does not guarantee that a configuration will avoid allocation failure. No performance claim is made.Sur le site
Liens install, modèles, releases.