Pull requests / #2901
#2901 STRATA_LOOP_BREAK=1 (opt-in): a --serve reply that repeats one span three times is pushed out of the copy
open · @zsariboga · 0 Kommentare · Auf GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows
Beschreibung
## The problem
`repeat_stop_tokens` (#606) ends a reply stuck on one token. It never sees a model that writes the same sentence or
paragraph again and again, and that is the loop we meet in agent use: across 4.06M tokens of one user's opencode
sessions (Swift 1.5 IQ3_XXS on a 12 GB laptop), 11% of the sessions longer than 3K tokens had a hard loop and 5.7% of
all tokens were repetition. One regex task ran a 29K-token loop of the same paragraph. Once a span has been written
three times the model's own probabilities keep it in the copy; sampling rarely leaves it.
## What it changes
With `STRATA_LOOP_BREAK=1`, the serve decode path watches the reply (not the prompt):
- For every period p = 1..512 it keeps `run[p]`, the number of tokens in a row equal to the one p before them (one
compare per period per token). When `run[p] >= 2p` (three copies) and `run[p] >= STRATA_LOOP_BREAK_MIN` (default
256 tokens), the reply enters break mode for `STRATA_LOOP_BREAK_SPAN` tokens (default 1024). A `<think>`,
`</think>`, `<tool_call>`, `</tool_call>` or ``` token ends break mode early, so it stays inside the span it began in.
- In break mode, before each window, it looks up where the reply's last 8 tokens occurred before (a hash of every
8-gram -> where it last ended) and takes the token that followed them there: the token that would continue the copy.
That one id gets `STRATA_LOOP_BREAK_BIAS` (default -100) in the window's FIRST row only. Rows past the first follow
drafts the target has not taken yet.
- The sampler applies it in `sample_tokens` and in `sample_tokens_spec` (after its early returns, so a fallback never
adds it twice), on the target's full-vocabulary rows only. The id and bias live in mapped host memory and are read
by a one-thread kernel, so captured windows pick up each new value.
- One `strata serve: loop break: N loops broken, M windows with a banned copy token` line per request that needed it.
## What it leaves alone
- Unset (the default): `set_loop_ban` is never called and no kernel is launched, so the default path is unchanged.
- Set, but a reply never repeats a span three times: the kernel is launched with bias 0 and returns at once; the
picks are the same as unset.
- `--pipeline-windows` decode (its next window is built before this one's tokens are known), batch slots and the
multi-card drive are not covered.
- The edge token ids are Qwen3.8's (the tokenizer every supported Flash-Next build uses).
## Measured / checked
- Offline replay of the trigger over the 4.06M tokens above: it fired 96 times, each on a real loop; no false alarm.
- In daily use on our fork since 2026-10-14 (Swift 1.5 IQ3_XXS, RTX 5070 Ti Laptop, opencode).
- This branch: CUDA build (sm_120, CUDA 13, MSVC) clean. HIP and SYCL not built (the change is in
`src/kernels/cuda/sampler.cu`, a plain kernel launch, and `generate.cpp`).
- SMOKE (2026-10-17, RTX 5070 Ti Laptop, Swift 1.5 IQ3_XXS, this branch's engine `--serve` rebased on the current
`main`, the branch's own server; the model ran in the RAM-budget file tier, so the run's tok/s are not comparable and
are not a speed claim): with `STRATA_LOOP_BREAK=1` the repetition prompt ("write this sentence exactly 60 times,
nothing else") ended after 9 lines / 2 unique (305 tokens) and the log carried
`strata serve: loop break: 1 loops broken, 2 windows with a banned copy token`; with the variable unset the same
prompt produced 50 identical lines / 1 unique (1750 tokens, 5.7x). A normal prompt (a median function) answered
correctly in both arms; the two answers are equivalent but not byte-identical (temperature 1.0, and GPU/CPU expert
rounding differs run to run in this configuration). The engine line above is the one the change promises.
Mehr auf der Site
Links zu Install, Modellen, Releases.