Pull requests / #2646

#2646 batch: repetition / frequency / presence penalties apply in the batch windows (a slot counts its own conversation's tokens)

open · @qiuzan2001 · 0 comentarios · En GitHub

AMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Descripción

docs/BATCHING.md lists, under "Limits (for now)": *Repetition / frequency / presence penalties are not applied in batch windows.* This applies them.

## What it changes

- `Verifier::sample_rows` (src/core/verify.cpp) cleared `penalty_last_n` for every slot row and handed the sampler no history, so a request with `repetition_penalty` / `frequency_penalty` / `presence_penalty` decoded in a slot as if it had none. It now samples a row whose slot has penalties on the host side with that row's history, greedy or not (the graph's argmax pick is replaced, as the solo path does for a greedy request with penalties).
- `batch_step` (src/program/generate.cpp) stages one history row per window row from the slot's own tokens, `ids` followed by the token the row feeds, and with `--batch-mtp` the proposal row follows the fed token too. The rows are built by the same `strata::kernels::penalty_rows` the solo path uses, so a slot counts exactly what plain decode of that conversation would count. A new `Verifier::set_slot_history` hands them over (it reaches the last stage of a layer split, where the rows are sampled; the buffer is on that device, next to the solo path's).
- `BSlot::penalty_last_n`, set at admission from the request's sampling.
- docs/BATCHING.md: the limit is reworded; the `tools/batch_test.py` row says how to check it. docs/DETAILS.md: one sentence in the penalties paragraph.

## What it leaves alone

- A window in which no active slot has penalties hands the verifier no rows and takes the old code path; a slot without penalties in a window where another has them leaves its rows at -1 and is sampled as before (its `penalty_last_n` is 0).
- The pipelined slot groups (`--batch-groups` above 1) never stage rows and are unchanged. The docs say so.
- The SYCL tree has its own `verify.cpp` / `generate.cpp` and is not changed; its batch windows still skip penalties (the docs say so).
- The solo path, the drafts (the draft layer still guesses without penalties) and every request without penalties.

## How it was checked

RTX 5070 Ti 16 GB, Ryzen 7 5800X3D, Linux, CUDA 13.1 (arch 120); Swift 1.5 IQ3_S; `tools/batch_test.py --batch 2 --n 2 --max-new 64 --extra "--max-context 32768 --pcie-frac 0 --adapt-every 1000000"` (the script sets `STRATA_IQ_MT_MIN=1`), on the binary before this change and after it, the engine restarted for every arm:

| arm | binary | `--keys` | slots vs their solo tokens |
| --- | --- | --- | --- |
| plain | before | none | identical (both slots) |
| plain | after | none | identical, and the same solo and batch tokens as "before" |
| penalties | before | `penalty_repeat=1.5 penalty_last_n=64` | **differ** at token 3 (slot 0) and 22 (slot 1); the batch tokens equal the plain arm's, i.e. the penalties were ignored |
| penalties | after | the same | identical (both slots) |
| penalties + `--batch-mtp` | after | the same | identical (both slots) |
| penalties, sampled | after | `... temperature=0.7 top_k=20 seed=11` | identical (both slots) |

The solo tokens with penalties are the same on both binaries. An independent review pass read the diff (row/slot mapping across stages, the history contract against the solo path's `consumed` + `window`, buffer bounds, the no-penalty path) and found nothing to change beyond the docs wording above.

Not built: HIP (no AMD card here; `src/core/verify.cpp` and `src/program/generate.cpp` are the shared sources), Metal (no Mac). SYCL not touched.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

En el sitio

Enlaces a install, modelos, releases.