Pull requests / #2957
#2957 engine: a WARNING when --max-context is past the model's trained context and no rope scaling is active
open · @sergqwer · 0 comentários · No GitHub
Setup & installServer & APINVIDIA / CUDAModels & quantsWindows
Descrição
A config written by hand with `--max-context 524288` and no `--rope-scaling` starts without a word. Positions past 262144 then rotate by angles the model never saw. Setup adds yarn above the trained context, but a config written by hand does not go through setup. **The change.** Where the engine resolves the rope config to `none`, it now prints one WARNING to stderr when `--max-context` is past the trained context (the model's, or `--yarn-orig-ctx`). The WARNING names the fix. Nothing else changes, and a context within the trained one prints nothing new. Measured on the builds of 477ac13c (upstream/main, v0.1.45) and this branch: | start | 477ac13c | this PR | | --- | --- | --- | | `--max-context 524288`, no `--rope-scaling` | starts, no line | starts, one line (below) | | `--max-context 524288 --rope-scaling yarn --rope-scale 2` | - | no WARNING (the existing "rope scaling yarn, factor 2 ..." line) | | `--max-context 262144` | no line | no line | ``` strata generate: WARNING: --max-context 524288 is past the model's trained context of 262144 and no rope scaling is active: positions past it rotate by angles the model never saw, and it can lose track there. To extend the context: --rope-scaling yarn --rope-scale 2 (what setup writes above 262144) ``` **Identity at the default 262144.** - The first-token logits and the greedy tokens (32 max; both prompts stop at the end of turn) are byte-identical to 477ac13c's for a 2K and a 32K prompt. - 2K: `1e9c612c2e6c7a32...`, 4 tokens. 32K: `f5b2872c7ac30b17...`, 20 tokens. **Setup.** - Machine: RTX 5090, Ryzen 9 9950X3D, 128 GB, Windows 11. - Builds: MSVC 2022, CUDA 13.3, portable, `75-real;86-real;89-real;120-real`. - Model: the NVFP4 GPTQ pack. **How to reproduce.** - Install with upstream's `python setup.py --family nvfp4 --model NVFP4` (Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata). - For the WARNING: start `strata --serve` with the model arguments setup writes, `--max-context 524288` and no `--rope-scaling`. The line comes before the experts load. - For the identity: - flags: `--expert-cache 3000 --prefill auto --spec 6 --spec-min-p 0.7 --max-context 262144 --kv int8 --pcie-frac 0 --adapt-every 1000000`; - env: `STRATA_PREFILL_CPU_SHARE=0 STRATA_PCIE_BALANCE=0 STRATA_IQ_MT_MIN=1 STRATA_DUMP_FIRST_LOGITS=<file>`; - send a 2K and a 32K prompt, greedy, 32 tokens. - All of it was measured on that pack with upstream/main builds. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
No site
Links install, modelos, releases.