Pull requests / #2042

#2042 STRATA_EXCHANGE_ROTATE: rotation for packs whose layers differ in expert block size (spares kept per size)

open · @Xiaomi2008 · 0 comentários · No GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Descrição

Follow-up to #864 (the opt-in exchange buffer rotation). I searched the open threads: #1340 and #1838 touch the resident RAM exchanges but not this limit.

## What it changes

`STRATA_EXCHANGE_ROTATE=1` now works on packs whose layers differ in expert block size. Until now rotation required one block size for all layers and fell back to the copy path otherwise ("requires equal-size expert blocks"). Swift 1.5 IQ3_S has seven sizes (1,510,400 to 2,662,400 bytes) and IQ3_XXS nine, so neither could rotate.

- `ExchangeStorage` keeps its spare buffers per block size (`initialize_sized`). A swap exchanges two experts of one layer, so an exchange takes a spare of its layer's size and hands back one of the same size; every size keeps its number of spares. `commit()` refuses a spare of another size and an expert of another layer.
- `exchange_buffer()` takes the layer; `reserve_exchanges()` takes the number of trailing buffers that only stage copies (`--adapt-async`'s second half), kept in the largest size alone.
- With `--adapt-async 1` a rotated round flips in the tick that sees the copies in (nothing for the job to move), one verify window sooner.

## What it leaves alone

Rotation stays opt-in. Unset, nothing new runs: the copy path, the uniform `initialize()` geometry and its checks are as before, and no default or preset changes. Switched on, the exchange allocation is one spare of every size per swap of a round: 1.06 GiB of page-locked RAM on IQ3_S with `--adapt-swaps 80` (1.26 GiB with `--adapt-async 1`) against 0.20 and 0.40 GiB.

## Measured

Core i7-13700K, RTX 5090 32 GB, 64 GB DDR5, Windows 11, driver 617.42. Swift 1.5 IQ3_S, 128K, `--resident-experts` (about 11,000 of 24,576 experts in VRAM), `--adapt-every 1 --adapt-swaps 80 --adapt-decay 0.97`, `--spec 4`, `--pool-affinity p-cores`.

**This branch (main at 7695859a + this commit, engine 0.1.43), same day, one binary, only the variable differs.** One 8K prompt, the same 1,500 forced tokens (`--spec-follow`), blocking tier, runs interleaved:

| | decode tok/s |
|---|---|
| unset | 171.2, 170.3, 173.2 |
| `STRATA_EXCHANGE_ROTATE=1` | 195.9, 194.9, 197.2 |

An unset run and two rotated runs that sized the cache alike (11,099 slots) wrote the same `--window-hashes` file for all 434 windows. (The unset run that sized 11,017 slots differs from the others, as any other cache size does.)

**The same commit on 0.1.42** (my fork's main, which also carries #1911 and #1912):

| Run | Rotation off | Rotation on |
|---|---|---|
| The same forced-token run | 161.6-165.1 (five runs) | 184.9-188.7 (four runs) |
| `--serve`, 23 coding-agent turns (`tools/agent_turns_bench.py`'s turns, greedy, up to 400 new tokens), three interleaved rounds, blocking tier (off = the build before) | 147.8 / 159.2 / 158.4 | 175.2 / 184.9 / 172.8 |
| The same turns with `--adapt-async 1`, one binary, two rounds | 166.0 / 170.1 | 185.6 / 182.5 |

The tier swaps about 30 experts per verify window all through a run on this box, so the copy path moved about 50 MiB of RAM per window on the decode thread.

## How it was checked, and what was not

- `tests/core/exchange_storage_test.cpp` gains a three-size case (6,144 exchanges: bytes, aliases, the refusals, overlapping resident blocks) and passes on this branch, built alone with MSVC 14.50. I did not run it under ASan/UBSan, and I did not run `file_expert_source_test --rotation-gpu`.
- Built for CUDA only on this branch (sm_120, CUDA 13.0.88, MSVC 14.50 with nvcc's `--allow-unsupported-compiler`). **HIP and SYCL were not compiled**; `expert_source.cpp` and `generate.cpp` are built by both.
- Not measured: Linux, AMD cards, a layer split, a pack with one block size on this machine, the serve runs on 0.1.43.

Details and the numbers: `docs/EXCHANGE_ROTATION.md`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

No site

Links install, modelos, releases.