#3800 [Engine] Token mask for JSON response formats: MTP drafts survive constrained decoding
open · @ncphri · 0 Kommentare · Auf GitHub
BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsSecurityWindowsLinux
Beschreibung
## Problem and result Structured JSON replies (`response_format: json_schema`) were served by prompt instruction + server-side validation only: the model could sample a token that makes the required schema keys unreachable, the reply finished, validation failed, and the client got `502 structured_output_failed` and retried the whole request. In a production multi-turn agentic pipeline on a quantized model (one deployment run, ~75 report requests), 50 schema-failure events were logged across 10 samples (~13% of samples affected, up to 5 client reconnects each); ~92% were one missing required top-level key, the rest additional-property violations. An agentic CLI client that re-validates the schema client-side over the Responses route failed 3/20 real report prompts (15%) end-over-end on the unmasked build. With this change the engine cuts grammar-illegal tokens before sampling, so constrained replies are valid by construction. Same 20 real prompts, same machine, same day, on the masked build: **20/20 first-pass valid, zero client retries**. Per-run decode on the masked arm was steady at 65-81 tok/s (median 73) across all 20 runs; the unmasked arm's effective throughput was far more variable (median 21, range 11-65 tok/s) because failed generations burn whole windows before the client retries. Failures are not fully deterministic - the 15% rate reproduces intermittently, matching the original field logs. ## Related discussion PR #1328 (closed by the tracker reset, not declined; the maintainer asked for a rebase onto main and a fresh PR, which this is). ## Approach The server (`serve/constrain.py`) compiles the schema to a token mask with llguidance and answers a per-window mask exchange: MQ (window candidates + drafts) -> MF (free) or MK/MKN (masked rows). Masking happens between logits and sampling, so MTP drafts ride along inside masked multi-row windows instead of dying at the first rejected draft. Opt-in: active only when the request declares a JSON schema AND the engine advertises `token_mask=1/2` in its startup INFO; the default path is byte-identical. `/v1/status` reports `constrained_decoding: true` only when the gate passes. `/v1/responses` supports both (its `prepare_format` runs with `with_tools`: the tool turn stays free and only the answer turn is validated). The chat route allows the pair too (main's own #4756 covers that half; the merged branch carries main's wording, so this PR's diff no longer carries it); a tool-call turn is not an answer, so it passes through as written and only an answer turn is validated. `/v1/messages` does the same (a1e2242f): `output_config.format` is the primary carrier and `response_format` the secondary one, the answer's text deltas are held while the schema is in force and validated at the turn's end, and the mask rides via `constrain.format_of`. `token_mask=2` also lets a whole MTP window be masked; row-count shrink trims drafts server-side. ## Validation **Passed:** - Merged with `origin/main` twice on 2026-10-28 (0.2.7.x -> 266ad181) and 2026-11-04 (0.2.10 -> merge head **fc2c1ebb**). The first round: keep-both in `serve/server.py` (main's `prefill_only` head rewrite beside the mask block; main's LP-line capture beside `c.feed(t)`) and main's window-T refactor in `generate.cpp` at the mask-comment hunks. The second round: `serve/server.py` takes main's `rid`/`pen_from` machinery beside the mask gate (the `mask=1` append rides after main's rid head), `verify.cpp` takes main's `STRATA_VERIFY_HGL` guard beside the token_mask include, `generate.cpp` takes main's DFlash2/window-rule restructure with the MQ exchange beside main's `keep_sync` (window filled -> mask cut -> MQ/MK/MKN rows -> adoption; the HIP stop-bound block follows the mask cut), and SYCL keeps its bare-MQ mask block beside the window rule — its result is one row under the mask (`if (gm_on) T = 1`, the deferred lookup gated off). At that merged head the filed diff against `main` is the mask machinery and its serve half only (14 files, +1342/−16, engine files 7 files, +399/−11); the head has since grown to 16 files (serve tests + the SYCL apply-kernel harness), engine files still 7. - Re-validated at the merged heads: the same suite -> **465 tests OK** at 266ad181 and **467 tests OK** (~150 s, miniforge python 3.12, llguidance 1.9.1, WSL Ubuntu, no GPU) at fc2c1ebb, zero skips. - Engine identity with the stacked serve PR's merged head (#4222, ed7f0d1e): `git diff fc2c1ebb ed7f0d1e -- src include sycl` = **0**, so the CUDA build and hardware verification run there on 2026-11-04 (170/170 steps, 0 errors, engine 0.1.56, `STRATA_TOOL_MASK=1`, 15/15 first-pass probes, `/v1/status` `constrained_decoding: true`) covers this PR's merged engine files byte-identically; the serve-side delta is that PR's lark chain, covered by its own mock suite. - Serve suite: `python -m unittest serve.test_constrain serve.test_structured serve.test_responses serve.test_server serve.test_mcp` -> 450 tests OK @ 8979f87f (WSL, CPython 3.12). - CUDA build: full engine build 165/165 steps, zero errors @ 8979f87f (RTX 4090 Laptop, Windows, CUDA 13.0), and the resulting binary serves the hardware round-trip above (production model, IQ3_S quant; the A/B arms ran on the 3766738d build, the same C++ mask code through this merge commit - the 0.2.3 merge is comment-only on the C++ side). - Metal build: `cmake -S metal -B build-metal -G Ninja -DCMAKE_BUILD_TYPE=Release -DMETAL_CPP_DIR=<metal-cpp> -DSTRATA_GGML_DIR=<llama.cpp @3cf03257>` + `cmake --build` -> 139/139 steps, zero errors @ 8979f87f (macOS 27 arm64, Apple clang 21); the binary carries the mask code paths. Compilation only - the Metal runtime was not exercised. - SYCL build: `sycl/tools/build.sh` in the `strata-sycl-dev` image (oneAPI 2026.1, icpx) -> 232/232 steps, zero errors @ 8979f87f; the binary carries the mask code paths. Compilation only - see Not run. - Hardware mask round-trip, production workload: 20 real multi-turn report prompts (sanitized copies attached on request) through an agentic CLI client with client-side schema validation. Unmasked build (0.1.44.1, `constrained_decoding: false`): 17/20 valid, 3 terminal failures (all one missing required key after 5 client reconnects, ~11 min wasted each with zero usable output). Masked build (this branch): 20/20 valid, 0 retries. Same host, same model (IQ3_S quant), same prompts, same day. - Single-shot `json_schema` requests: 3/3 first-pass valid, up to 15k-token structured outputs, `finish_reason=stop`. - Mask decode-cost isolation, same revision: the same build (8979f87f) serving the same prompt plain vs `json_schema` (masked), interleaved single-shots, max_tokens 6000, 3 pairs: masked 74.6 / 79.7 / 79.2 tok/s, plain 76.5 / 97.5 / 74.3 (wall-time rates, prefill included; the plain 97.5 is a warm-cache run). Medians 79.2 vs 76.5 - no measurable mask cost; the delta is inside the run-to-run spread. - `/v1/status` gate: `constrained_decoding: true` with the masked engine, `false` with the main-branch engine, as designed. - Three more merges of `origin/main`: 2026-11-11 (5f0ae5b5 -> **08e544e8**; keep-both — the mask lines run first in the window's acceptance and the cut caps a before main's `STRATA_FOLLOW_LOGPOS` block picks its rows) and 2026-11-12 (5f0ae5b5..697b6d67 -> **60667774**; clean — main's three MMQ row-id margin commits touch none of the mask PR's files, no resolution decisions). - The llguidance pin moved to opt-in (**80901277**, 2026-11-12): llguidance 1.9.1 ships a cp39-abi3 wheel only, so every runtime-*-cp312 lock refuses the pin (`test_packaging_pins` fails on the committed locks) and every deployment paid for a package the availability gate may never use — the pin is now `pip install llguidance` into the venv (opt-in) and setup.py's fallback drops it. Tests: py-tests 3664 passed / py-serve 1589 passed. - Batched masked windows, SYCL exchange device-side (**32235744**, 2026-11-13): `supports_mask` no longer requires a single slot (a batched window masks only its solo rows; the stream-ownership regression fixed by passing `mask_answer` only on a masked turn); the SYCL mask exchange moves after the window's candidates and drafts (the CUDA side's shape), drafts ride along, the T=1 row drop is gone and the INFO line advertises `token_mask=2`; the verify writes `kMaskedLogit` on-device from a per-window bitmap and returns only per-row allowed counts on the sampler's own wait. CUDA engine files untouched. Tests: py-serve 1594 passed (base 1589 + FakeEngineBatch 5) / py-tests 3663 passed with 2 environment-existing failures reproduced on the base worktree. - MKN reply rows capped at `kVerifyMaxT` (**c2f18f16**, 2026-11-14): the parser rejects a reply claiming rows above the build's cap and ends the request (measured on the live serve: 125 aborts — prompt-lookup windows run to `kVerifyLongT`=12 rows on an 8-row CUDA build and a repetitive JSON answer filled them); the engine shrinks its window to the reply's row count, so a smaller reply is safe. Tests: `test_constrain` 27 OK; live E2E: the previously fatal repetitive-JSON request finishes stop with valid JSON and no new MKN-rows line, verify_f2a 8/8, schema_probe 7/7. - SYCL apply-kernel row-offset fix (**c8921a3d**, 2026-11-14): `tmask_apply_launch` indexed `words[w]` instead of `words[t * n_words + w]`, so every window row past the first was masked with row 0's bitmap (measured on the SYCL rig at c2f18f16: masked turns corrupt 6/6 at temp 0 and 5/5 at 0.7 while row 0 stays right; the fallback fires only on rows 1-3). The fix is the one index; CUDA files untouched, so multi-row masked windows keep full depth. Host harness `sycl/tools/test_tmask_apply.sh`: RED on the unfixed kernel (10 assertion failures), GREEN (3 cases). Suite **489 OK** on the fixed tree and on the clean base (no test lost); CUDA E2E regression 10/10 PASS, corruption scan 0 hits, 4-row MKN windows observed running. - SYCL build + real-device runtime, the maintainer's Arc rig (**rriggs**, 2026-11-14, at c8921a3d; AOT bmg-g31, engine 0.1.59.1, Qwen3.8-Flash-Next IQ3_S, `--spec 4 --mtp`): corruption 0/6 + 0/5 masked (was 6/6 and 5/5 on the unfixed head), TC-64/65/66/67/69 **PASS 2/2**, zero repeat-stops, zero HTTP 500s and zero token-mask fallback lines in the entire run; full draft depth retained with no correctness cost. - Reviewer patch ingested (**f1a8e86a**, 2026-11-14, `git am` of rriggs's `0001-serve-tools-with-response-format.patch`, `From:` authorship preserved): chat tools + `response_format` together — the tool turn stays free, only the answer turn is validated. Tests: `serve.test_structured` 15 OK (the 2 new cases), `test_responses` 43 OK, `test_server` 372 OK. - `/v1/messages` validates a schema'd output (**a1e2242f**, 2026-11-15, current head): tests `serve.test_structured` 21 OK (6 new Anthropic cases), `test_constrain` 27 OK, `test_server` 376 OK; suite `tools/check --only py-serve,py-tests`: py-serve 1602 passed 23 skipped / py-tests 3663 passed 67 skipped with one failure in a pre-existing TUI storage-path test, proven by a same-tree A/B run (`TMPDIR=/tmp` passes, a long scratch path fails) to be a harness path-length artifact; E2E on the CUDA rig, patchless: `tool-eval-bench --format anthropic` 75/100 at both temperatures 0 and 0.7, 31/31 `finish_reason=stop`, zero 500/502, structured_output_failed, error events or tracebacks. - Merged with `origin/main` again on 2026-11-20 (0.2.16, merge head **9d0a077b8** + the resolution fix **80c92284c**; 2,039 commits since 697b6d67 - the 0.2.16 layout move lands the mask files at their new paths: `serve/` -> `terminal/serve/`, `src/` -> `engine/src/`, `sycl/` -> `engine/sycl/`, `include/` -> `engine/include/`). Keep-both at the merged head: `server.py` takes main's `leg_keys`/hold/`slot_read` machinery beside the mask wiring (a masked request never takes the slot-read shortcut and nev
Mehr auf der Site
Links zu Install, Modellen, Releases.