#2961 rope: --rope-scaling dynamic (opt-in): yarn by the dialog's length past the trained context; cached keys rotated when the factor changes

Descrição

Static yarn, which setup writes for a context past 262,144, changes every dialog, the short ones too: at factor 2 the first-token distribution of a dialog inside the trained range moves by KL 0.056, and the top-1 token differs in 9.2% of positions (measured on our fork, below). `--rope-scaling dynamic` keeps such a dialog on the trained angles and turns yarn on only for the dialog that passes 262,144.

### What it does

- **The factor.** None while a dialog fits the trained 262,144 positions. Past them, yarn with factor = positions / 262,144, rounded up to `--rope-dynamic-step` (0.125) and capped at `--rope-scale` (default `--max-context` / 262,144, so 2 at 524,288). It is chosen per request from the prompt's length, and again in decode when the reply grows past the table's positions.
- **The cached keys follow a change.** K is stored post-RoPE, so a key written at factor A becomes the key of factor B after one more rotation of each pair by p * (w_B - w_A) and the magnitude ratio mscale_B / mscale_A. The new kernel (`src/kernels/cuda/rope_dynamic.cu`) applies this to everything the session holds: K (int8 or fp16, with or without `STRATA_KV_ROT`'s Hadamard: dequantize, H, turn the 32 rotary pairs, H, requantize), the indexer's pooled keys, its spare row and `dead`, the checkpoints' `dead`, and the drafter's K. With `--kv-resident` the pinned host copy is turned and the VRAM slots are fetched again. `STRATA_ROPE_DYNAMIC_REREAD=1` reads the prompt again instead.
- **The table.** The cos/sin table is rebuilt in place (on 16 host threads, bit for bit `build_rope_table`), so the captured CUDA graphs keep their pointer and read the new angles. That needs #280's table on the analytic paths too (`--native-rope`, the native indexer, the prompt path): their constants are baked into the graphs at capture. So dynamic turns the table on, as `STRATA_ROPE_TABLE=1` does (`rope_table_force_on()`).
- **Files.** Parked conversations and session files record the factor their keys hold. It rides in the upper bits of the existing cvec word, so a file written without dynamic reads as factor 1. A file is still byte-identical to before when the factor is 1 (`conversation_file_test`'s golden file passes unchanged). A restored image is turned to the next request's factor. The session config fingerprint takes the yarn template and a `rope_dynamic` switch.
- **Not yet.** `--batch` slots, a layer split and `--kv q4_0` / `k8v4`. The engine refuses them at start and names static yarn as the alternative. One-shot (non-serve) runs pick one factor for prompt + `--max-new`.

### Default behaviour

Without the flag nothing changes. With it, a dialog below 262,144 runs the bits of `STRATA_ROPE_TABLE=1` with no scaling. These are not the bits of the default run: there the angles come from fast-math `cosf`/`sinf` (#280), and on long_32k the two differ by KL 0.047 with the same greedy tokens.

### Measured on this branch

RTX 5090 (Windows 11, CUDA 13.3, driver 617.14), 9950X3D, 128 GB. upstream/main 477ac13c and this branch, the same release configure. The NVFP4 GPTQ pack (`Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata`, upstream setup's `--family nvfp4 --model NVFP4`).

**Identity.** `--exact --cache 3000`, the p2k and long_32k prompts, first-token logits (sha256) and 32 greedy tokens:

| run | p2k logits | long_32k logits | greedy tokens |
| --- | --- | --- | --- |
| upstream/main | `1e9c612c...` | `f5b2872c...` | 4 / 20 |
| this branch, no flag | `1e9c612c...` identical | `f5b2872c...` identical | identical |
| upstream/main, `STRATA_ROPE_TABLE=1` | `67da66a7...` | `a9cf6bf8...` | identical to the default |
| this branch, `--rope-scaling dynamic --max-context 524288` | `67da66a7...` identical | `a9cf6bf8...` identical | identical |

**A switch in decode**, at small scale. The run uses `--rope-scaling dynamic --yarn-orig-ctx 8100` and the 8,035-token long_8k prompt, so the reply passes the "trained" 8,100 positions:
- The prompt reads at factor 1: the first-token logits are byte-identical to upstream/main with `STRATA_ROPE_TABLE=1`.
- At token 63 of the reply the engine logs `rope dynamic: factor 1 -> 1.125 (decode): 8098 cells re-rotated in 0.4 ms, table 18.4 ms`. The table covers the 262,144 cells of `--max-context`.
- The 94-token greedy reply equals the unscaled run's. A factor of 1.125 moves the angles past 8,100 only a little. So this run shows that the switch works in upstream's decode loop; it does not measure the quality effect.

**The rotation against writing at the new factor** (`rope_dynamic_parity`, new ctest, GPU, synthetic). Random keys at positions 0 to 519,999 (48 pages at the real page size, both sides of 262,144 among them) are rotated on the host by A and by B as the table path does. The engine's own append kernels write them: `kv_append_step` (fp16), `kv_append_q8_step` (int8), and int8 through `fwht256` first. A's pool is then turned to B and compared with the float64 key at B. The error of the key written directly at B is the yardstick:

| A -> B | fp16: RMS turned / written, worst | int8: RMS ratio, worst | int8 + Hadamard: RMS ratio, worst |
| --- | --- | --- | --- |
| 1 -> 2 | 1.48x, 0.99 ulp | 1.33x, 1.12 steps | 1.41x, 1.61 steps |
| 1.5 -> 2 | 1.44x, 0.99 ulp | 1.36x, 1.13 steps | 1.40x, 1.55 steps |
| 2 -> 1 | 1.46x, 1.27 ulp | 1.33x, 1.15 steps | 1.39x, 1.95 steps |
| 1 -> 1.125 | 1.45x, 1.14 ulp | 1.33x, 1.17 steps | 1.42x, 1.55 steps |

- "ulp" is an fp16 ulp of the row's largest value; "steps" are the quantization step of B's group (with the Hadamard, of the row's largest group).
- A turn re-codes once more, so its error is the written key's plus A's carried over: sqrt(2) = 1.41x for equal steps. The test's bounds are 1.6x RMS, and a worst element of 3 ulps / 2 steps / 3 steps.
- Without the Hadamard, dims 64-255 (codes and scales) stay bitwise what was written.
- The indexer's pooled keys are within 3.8 float ulps of the row's largest value. The spare row and `dead` are exactly mscale_B / mscale_A times their old values.
- `build_rope_table_mt` equals `build_rope_table` bitwise (factors 1, 1.125 and 2; 1, 7 and 16 threads). So a dialog that goes back below 262,144 gets the startup table.

### Measured earlier on our fork

The same change on our fork's 0.1.44 code (sergqwer/strata-nvfp4), RTX 5090, the same pack, `--kv int8`. I did not repeat these on upstream's code: each band run reads 13 bands up to 516K, about 72 s per run.

Teacher-forced NLL on a corpus of C++ sources read once in order. At 13 band starts from 8K to 516K the next 1,024 tokens are scored. Paired per-token differences vs `none`, pooled over the 7 bands past 262,144 (7,168 tokens), with the SE in brackets:

| arm | far bands vs none | in range (6 bands) |
| --- | --- | --- |
| static yarn 2 | -0.0110 (0.0048) nats | KL 0.056, top-1 agreement 90.8% |
| dynamic | -0.0106 (0.0044) nats | byte-identical to none (logpos rows) |
| dynamic with `--kv-resident 32768` | -0.0117 (0.0045) nats | byte-identical to none |

- **Rotating vs reading the prompt again.** -0.0002 (0.0026) nats over the 7 far bands. That is after 7 switches (1 -> 1.125 -> 1.25 -> 1.5 -> 1.625 -> 1.75 -> 1.875 -> 2) of a 472K-cell int8+H cache. The re-read arm is the static yarn run at the factor dynamic picks for each band. KL(re-read || rotated) was 0.008-0.023 per band, vs 0.03-0.07 between neighbouring factors.
- **Switch cost** (engine log, 12 QSA layers). K/V in VRAM: 8-15 ms rotation plus a 30-48 ms table rebuild at 259K-472K cells. `--kv-resident`: 354-677 ms (the host copy over PCIe). There are at most 8 switches between 262K and 512K, each paid once per dialog. Prompt time was unchanged within noise: 72.4-72.6 s for the 13 band reads, vs 72.1-77.5 s for none and static yarn.
- **Needle in a haystack.** A growing dialog (250K at factor 1, then 380K turned to 1.5, then the 505K haystack turned to 2) found 13/13 needles at 505K. Plain `none` also finds 13/13 there, so this test does not separate the arms.

The gain is small: about 1% perplexity past 262,144, 2.4 SE on one corpus. The case for dynamic is yarn's gain past the trained context without yarn's cost on every short dialog.

### How to reproduce

- Install the model with upstream setup: `--family nvfp4 --model NVFP4`.
- Identity: our driver starts `strata --serve` with setup's model files and `--expert-cache 3000 --prefill auto --spec 6 --spec-min-p 0.7 --max-context 262144 --kv int8 --gpu 0 --pcie-frac 0 --adapt-every 1000000`, env `STRATA_PREFILL_CPU_SHARE=0 STRATA_PCIE_BALANCE=0 STRATA_IQ_MT_MIN=1 STRATA_DUMP_FIRST_LOGITS=<dir>/first`. It sends two requests (a 2,100-token and a 32,037-token prompt, 32 greedy tokens each) and compares the logits dumps and the tokens.
  - Plus `--max-context 524288 --rope-scaling dynamic` for this branch.
  - Plus `STRATA_ROPE_TABLE=1` for upstream/main.
- Serving: setup's config with `--max-context 524288 --rope-scaling dynamic` in place of `--rope-scaling yarn --rope-scale 2`.
- `ctest -R rope_dynamic_parity` on a CUDA build with tests.

### Not measured

- HIP and SYCL: CUDA only here; the SYCL copy of generate.cpp is unchanged.
- A real dialog past 262,144 on upstream's code: the long band reads above are from the fork.
- A layer split, `--batch`: refused.

Setup and the Model settings page do not offer `dynamic` yet. If this is wanted, I would send that as a follow-up: setup would write dynamic instead of static yarn past 262,144 where it runs (one GPU, no `--batch`, `--kv int8` or `fp16`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio