Pull requests / #2901

#2901 STRATA_LOOP_BREAK=1 (opt-in): a --serve reply that repeats one span three times is pushed out of the copy

open · @zsariboga · 0 commentaires · Sur GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows

Description


## The problem

`repeat_stop_tokens` (#606) ends a reply stuck on one token. It never sees a model that writes the same sentence or
paragraph again and again, and that is the loop we meet in agent use: across 4.06M tokens of one user's opencode
sessions (Swift 1.5 IQ3_XXS on a 12 GB laptop), 11% of the sessions longer than 3K tokens had a hard loop and 5.7% of
all tokens were repetition. One regex task ran a 29K-token loop of the same paragraph. Once a span has been written
three times the model's own probabilities keep it in the copy; sampling rarely leaves it.

## What it changes

With `STRATA_LOOP_BREAK=1`, the serve decode path watches the reply (not the prompt):

- For every period p = 1..512 it keeps `run[p]`, the number of tokens in a row equal to the one p before them (one
  compare per period per token). When `run[p] >= 2p` (three copies) and `run[p] >= STRATA_LOOP_BREAK_MIN` (default
  256 tokens), the reply enters break mode for `STRATA_LOOP_BREAK_SPAN` tokens (default 1024). A `<think>`,
  `</think>`, `<tool_call>`, `</tool_call>` or ``` token ends break mode early, so it stays inside the span it began in.
- In break mode, before each window, it looks up where the reply's last 8 tokens occurred before (a hash of every
  8-gram -> where it last ended) and takes the token that followed them there: the token that would continue the copy.
  That one id gets `STRATA_LOOP_BREAK_BIAS` (default -100) in the window's FIRST row only. Rows past the first follow
  drafts the target has not taken yet.
- The sampler applies it in `sample_tokens` and in `sample_tokens_spec` (after its early returns, so a fallback never
  adds it twice), on the target's full-vocabulary rows only. The id and bias live in mapped host memory and are read
  by a one-thread kernel, so captured windows pick up each new value.
- One `strata serve: loop break: N loops broken, M windows with a banned copy token` line per request that needed it.

## What it leaves alone

- Unset (the default): `set_loop_ban` is never called and no kernel is launched, so the default path is unchanged.
- Set, but a reply never repeats a span three times: the kernel is launched with bias 0 and returns at once; the
  picks are the same as unset.
- `--pipeline-windows` decode (its next window is built before this one's tokens are known), batch slots and the
  multi-card drive are not covered.
- The edge token ids are Qwen3.8's (the tokenizer every supported Flash-Next build uses).

## Measured / checked

- Offline replay of the trigger over the 4.06M tokens above: it fired 96 times, each on a real loop; no false alarm.
- In daily use on our fork since 2026-10-14 (Swift 1.5 IQ3_XXS, RTX 5070 Ti Laptop, opencode).
- This branch: CUDA build (sm_120, CUDA 13, MSVC) clean. HIP and SYCL not built (the change is in
  `src/kernels/cuda/sampler.cu`, a plain kernel launch, and `generate.cpp`).
- SMOKE (2026-10-17, RTX 5070 Ti Laptop, Swift 1.5 IQ3_XXS, this branch's engine `--serve` rebased on the current
  `main`, the branch's own server; the model ran in the RAM-budget file tier, so the run's tok/s are not comparable and
  are not a speed claim): with `STRATA_LOOP_BREAK=1` the repetition prompt ("write this sentence exactly 60 times,
  nothing else") ended after 9 lines / 2 unique (305 tokens) and the log carried
  `strata serve: loop break: 1 loops broken, 2 windows with a banned copy token`; with the variable unset the same
  prompt produced 50 identical lines / 1 unique (1750 tokens, 5.7x). A normal prompt (a median function) answered
  correctly in both arms; the two answers are equivalent but not byte-identical (temperature 1.0, and GPU/CPU expert
  rounding differs run to run in this configuration). The engine line above is the one the change promises.

Sur le site

Liens install, modèles, releases.