Pull requests / #2977

#2977 logit_bias: accept it with --batch (one bias table per slot; 150 ban pairs: 10 -> 0 answers in Chinese script, -1..-3 % batch throughput)

open · @qni-live · 0 Kommentare · Auf GitHub

BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows

Beschreibung

Issue: none (no open issue; `--batch` answers `logit_bias` with HTTP 400 today)

## Summary
`logit_bias` is refused whenever the engine runs with `--batch`. That makes the Han/Kana/Hangul ban (docs/LOGIT_BIAS.md) unusable on a 4-slot server, which is the setup for several parallel agents. This PR lets each batch slot carry its own bias table, so a bias reaches only its own request. `--batch-mtp` still refuses it.

## What changed
- `include/strata/core/verify.hpp`, `src/core/verify.cpp`: `Verifier::set_slot_logit_bias(slot, bias, err)` and `slot_has_bias(slot)`. Only the final stage allocates a table (one float per vocabulary entry, ~1 MB for 248,320 tokens) and copies it, and only when it changed. `set_slot_sampling` keeps the slot's table pointer instead of the caller's. `init_slots` frees and clears the tables.
- `sample_rows`: a greedy row whose slot has a table is now drawn on the last stage like a sampled row (the graph's argmax does not know the bias). Rows without a table take the same path as before.
- `src/program/generate.cpp`: the bias is set at admission, before `set_slot_sampling`, and an empty one clears the slot. `--batch-mtp` still answers `ERR logit_bias: --batch-mtp does not take it yet`. The engine advertises `logit_bias_batch=1` in `INFO`.
- `serve/logit_bias.py`: the server only refuses `--batch` when the engine does not advertise `logit_bias_batch=1` (older engines keep the clear 400).
- `tools/batch_test.py --bias-han` (every second prompt carries the ban and asks for Chinese) and the new `tools/batch_bias_reuse_test.py` (3 waves, banned and free prompts swap slots every wave): a slot's tokens must equal its solo tokens.
- `docs/LOGIT_BIAS.md`, `serve/test_logit_bias.py` updated (12 tests ok).

## Test results
Hardware/build: v0.1.45 + this change, 2x RTX PRO 4500 Blackwell (32 GB each), Swift IQ3_XXS, `--batch 4 --trim-stage-weights --pipeline-windows 2`, layer split. Ban list: 65,842 tokens (all Han/Kana/Hangul).

Exactness (`STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 1000000`, slot tokens against the same request run alone):

| Test | Result |
|---|---|
| no bias anywhere, `--batch-groups 1` (old path unchanged) | 4 of 4 slots identical |
| ban on 2 of 4 slots, greedy, groups 1 / auto | identical, 0 Han characters in the banned slots |
| same, sampled (temperature 0.7, top_k 20, seed 7), groups 1 / auto | identical |
| slot reuse, 3 waves, banned and free swap every wave: greedy g1, greedy groups auto, sampled g1 | 36 of 36 slot rows identical |
| `--batch-mtp` with a bias | refused as before |
| HTTP API on the server in batch (4 slots): zh+ban / zh free / cn+ban / en free | 0 / 6 / 0 / 0 Chinese characters, all 200 |

Effect (4 requests at once, the same 150 prompt pairs and seeds as my solo run on 2026-10-10 (5 of 150 without the ban, 0 of 150 with it); 900 max tokens):

| | answers with Chinese script | Chinese characters | tokens |
|---|---|---|---|
| without ban | 10 of 150 | 28 | 134,801 |
| with ban | **0 of 150** | 0 | 134,827 |

Cost (same server run, median of 3 rounds, 256 tokens per request, 65,929-entry ban):

| Arm | tok/s | vs. no bias |
|---|---|---|
| greedy, 4 at once, all with ban | 187.1 | -2.8 % (192.4 without) |
| greedy, 4 at once, 2 with / 2 without | 190.2 | -1.2 % |
| sampled (t=1), 4 at once, all with ban | 186.7 | -1.1 % (188.7 without) |
| greedy, 1 request | 141.8 | +1.8 % (noise, 139.3 without) |
| fixed cost per request (max_tokens=1) | +21 ms | 35 ms -> 57 ms |

## Extra notes
- Not covered: `--batch-mtp` (stays refused), the Qwen3.6 engine path, layer splits over 3 or more GPUs (only 2 cards here), HIP/SYCL.
- Overlap with #1891 (`serve/logit_bias.py`, `serve/test_logit_bias.py`): `git apply --3way` of #1891 on top of this branch is clean.
- The 150 pairs ran as three consecutive runs of the same build (pairs 0-31, 32-113, 114-149).
- Written with Claude Code (AI-assisted); I ran and checked every test above on my hardware.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.