Pull requests / #2646
#2646 batch: repetition / frequency / presence penalties apply in the batch windows (a slot counts its own conversation's tokens)
open · @qiuzan2001 · 0 commentaires · Sur GitHub
AMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Description
docs/BATCHING.md lists, under "Limits (for now)": *Repetition / frequency / presence penalties are not applied in batch windows.* This applies them. ## What it changes - `Verifier::sample_rows` (src/core/verify.cpp) cleared `penalty_last_n` for every slot row and handed the sampler no history, so a request with `repetition_penalty` / `frequency_penalty` / `presence_penalty` decoded in a slot as if it had none. It now samples a row whose slot has penalties on the host side with that row's history, greedy or not (the graph's argmax pick is replaced, as the solo path does for a greedy request with penalties). - `batch_step` (src/program/generate.cpp) stages one history row per window row from the slot's own tokens, `ids` followed by the token the row feeds, and with `--batch-mtp` the proposal row follows the fed token too. The rows are built by the same `strata::kernels::penalty_rows` the solo path uses, so a slot counts exactly what plain decode of that conversation would count. A new `Verifier::set_slot_history` hands them over (it reaches the last stage of a layer split, where the rows are sampled; the buffer is on that device, next to the solo path's). - `BSlot::penalty_last_n`, set at admission from the request's sampling. - docs/BATCHING.md: the limit is reworded; the `tools/batch_test.py` row says how to check it. docs/DETAILS.md: one sentence in the penalties paragraph. ## What it leaves alone - A window in which no active slot has penalties hands the verifier no rows and takes the old code path; a slot without penalties in a window where another has them leaves its rows at -1 and is sampled as before (its `penalty_last_n` is 0). - The pipelined slot groups (`--batch-groups` above 1) never stage rows and are unchanged. The docs say so. - The SYCL tree has its own `verify.cpp` / `generate.cpp` and is not changed; its batch windows still skip penalties (the docs say so). - The solo path, the drafts (the draft layer still guesses without penalties) and every request without penalties. ## How it was checked RTX 5070 Ti 16 GB, Ryzen 7 5800X3D, Linux, CUDA 13.1 (arch 120); Swift 1.5 IQ3_S; `tools/batch_test.py --batch 2 --n 2 --max-new 64 --extra "--max-context 32768 --pcie-frac 0 --adapt-every 1000000"` (the script sets `STRATA_IQ_MT_MIN=1`), on the binary before this change and after it, the engine restarted for every arm: | arm | binary | `--keys` | slots vs their solo tokens | | --- | --- | --- | --- | | plain | before | none | identical (both slots) | | plain | after | none | identical, and the same solo and batch tokens as "before" | | penalties | before | `penalty_repeat=1.5 penalty_last_n=64` | **differ** at token 3 (slot 0) and 22 (slot 1); the batch tokens equal the plain arm's, i.e. the penalties were ignored | | penalties | after | the same | identical (both slots) | | penalties + `--batch-mtp` | after | the same | identical (both slots) | | penalties, sampled | after | `... temperature=0.7 top_k=20 seed=11` | identical (both slots) | The solo tokens with penalties are the same on both binaries. An independent review pass read the diff (row/slot mapping across stages, the history contract against the solo path's `consumed` + `window`, buffer bounds, the no-penalty path) and found nothing to change beyond the docs wording above. Not built: HIP (no AMD card here; `src/core/verify.cpp` and `src/program/generate.cpp` are the shared sources), Metal (no Mac). SYCL not touched. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sur le site
Liens install, modèles, releases.