Pull requests / #2977
#2977 logit_bias: accept it with --batch (one bias table per slot; 150 ban pairs: 10 -> 0 answers in Chinese script, -1..-3 % batch throughput)
open · @qni-live · 0 commentaires · Sur GitHub
BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
Description
Issue: none (no open issue; `--batch` answers `logit_bias` with HTTP 400 today) ## Summary `logit_bias` is refused whenever the engine runs with `--batch`. That makes the Han/Kana/Hangul ban (docs/LOGIT_BIAS.md) unusable on a 4-slot server, which is the setup for several parallel agents. This PR lets each batch slot carry its own bias table, so a bias reaches only its own request. `--batch-mtp` still refuses it. ## What changed - `include/strata/core/verify.hpp`, `src/core/verify.cpp`: `Verifier::set_slot_logit_bias(slot, bias, err)` and `slot_has_bias(slot)`. Only the final stage allocates a table (one float per vocabulary entry, ~1 MB for 248,320 tokens) and copies it, and only when it changed. `set_slot_sampling` keeps the slot's table pointer instead of the caller's. `init_slots` frees and clears the tables. - `sample_rows`: a greedy row whose slot has a table is now drawn on the last stage like a sampled row (the graph's argmax does not know the bias). Rows without a table take the same path as before. - `src/program/generate.cpp`: the bias is set at admission, before `set_slot_sampling`, and an empty one clears the slot. `--batch-mtp` still answers `ERR logit_bias: --batch-mtp does not take it yet`. The engine advertises `logit_bias_batch=1` in `INFO`. - `serve/logit_bias.py`: the server only refuses `--batch` when the engine does not advertise `logit_bias_batch=1` (older engines keep the clear 400). - `tools/batch_test.py --bias-han` (every second prompt carries the ban and asks for Chinese) and the new `tools/batch_bias_reuse_test.py` (3 waves, banned and free prompts swap slots every wave): a slot's tokens must equal its solo tokens. - `docs/LOGIT_BIAS.md`, `serve/test_logit_bias.py` updated (12 tests ok). ## Test results Hardware/build: v0.1.45 + this change, 2x RTX PRO 4500 Blackwell (32 GB each), Swift IQ3_XXS, `--batch 4 --trim-stage-weights --pipeline-windows 2`, layer split. Ban list: 65,842 tokens (all Han/Kana/Hangul). Exactness (`STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 1000000`, slot tokens against the same request run alone): | Test | Result | |---|---| | no bias anywhere, `--batch-groups 1` (old path unchanged) | 4 of 4 slots identical | | ban on 2 of 4 slots, greedy, groups 1 / auto | identical, 0 Han characters in the banned slots | | same, sampled (temperature 0.7, top_k 20, seed 7), groups 1 / auto | identical | | slot reuse, 3 waves, banned and free swap every wave: greedy g1, greedy groups auto, sampled g1 | 36 of 36 slot rows identical | | `--batch-mtp` with a bias | refused as before | | HTTP API on the server in batch (4 slots): zh+ban / zh free / cn+ban / en free | 0 / 6 / 0 / 0 Chinese characters, all 200 | Effect (4 requests at once, the same 150 prompt pairs and seeds as my solo run on 2026-10-10 (5 of 150 without the ban, 0 of 150 with it); 900 max tokens): | | answers with Chinese script | Chinese characters | tokens | |---|---|---|---| | without ban | 10 of 150 | 28 | 134,801 | | with ban | **0 of 150** | 0 | 134,827 | Cost (same server run, median of 3 rounds, 256 tokens per request, 65,929-entry ban): | Arm | tok/s | vs. no bias | |---|---|---| | greedy, 4 at once, all with ban | 187.1 | -2.8 % (192.4 without) | | greedy, 4 at once, 2 with / 2 without | 190.2 | -1.2 % | | sampled (t=1), 4 at once, all with ban | 186.7 | -1.1 % (188.7 without) | | greedy, 1 request | 141.8 | +1.8 % (noise, 139.3 without) | | fixed cost per request (max_tokens=1) | +21 ms | 35 ms -> 57 ms | ## Extra notes - Not covered: `--batch-mtp` (stays refused), the Qwen3.6 engine path, layer splits over 3 or more GPUs (only 2 cards here), HIP/SYCL. - Overlap with #1891 (`serve/logit_bias.py`, `serve/test_logit_bias.py`): `git apply --3way` of #1891 on top of this branch is clean. - The 150 pairs ran as three consecutive runs of the same build (pairs 0-31, 32-113, 114-149). - Written with Claude Code (AI-assisted); I ran and checked every test above on my hardware. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sur le site
Liens install, modèles, releases.