#2639 Jev System One decision endpoint (/v1/systemone): calibrated typed decisions via an opt-in BROW logits wire

本文

# Jev systemone — System One decision endpoint for Strata

## Introduction

**What a System One decision model is.** Most uses of a language model are
"System Two": the model writes text and the caller reads, parses and trusts it. A
**System One** decision model does the opposite. It receives a **state** (text, JSON, or an
image) and a set of **typed questions**, and returns **typed answers with calibrated
probabilities**: a yes/no (`noul`), a pick-one (`choice`), or an ordinal rating (`score`).
It never generates prose to decide. Nothing has to be parsed, and the confidence is a number
the caller can threshold, route on, or log, rather than a sentence the model chose to write.
That makes it suited to fast, repeatable, auditable decisions inside a loop: routing,
guardrails, triage, scoring, and game or agent control.

**Jev.** "Jev" is TypeSafe's commercial System One decision API: `POST /v1/systemone` with
`{state, questions}`, answers read from token probabilities rather than generated text. Its
request/response shape is the wire this work follows. Open implementations of the same idea
already exist (see References); the goals here are correctness, speed and honest
probabilities, not novelty.

**Laya.** Laya (`convaiinnovations/laya`, Apache-2.0) approaches the same contract with a
different mechanism: a small purpose-trained encoder (ModernBERT-large, 421M parameters;
non-autoregressive) that answers all questions in one forward pass (~33 ms reported),
trained with reinforcement learning against strictly proper scoring rules (RLCD) so that
honest probabilities are what maximises reward. Two things in its documentation shaped this PR: its probabilities ship
over-confident until a temperature is refit **per question type and option count**
(reported mean ECE 0.466 → 0.081, with the instruction to do this on your own data before
trusting them), and its comparison against Jev uses third-party published Jev figures
that it did not measure itself. We kept the first idea (per-tuple calibration, refit on
local data) and measured our own numbers on our own harness. The mechanism differs: a
general LLM read at its label logits, with no task-specific training.

**jevcpp.** The reference implementation this port follows: a llama.cpp-based server that
serves the Jev wire from a generative model by reading the label logits at the final
prompt position (Qwen3.8-27B, with a shipped calibration file and a metamorphic/golden test
set). Its plan and amendments define the request contract, the frozen prompt template, the
readout math and the test strategy that this PR ports.

**What this PR does.** It brings that System One surface to **Strata**. Strata's engine could
already dump logprobs for verification, but had no per-request way to read the logits of
arbitrary label tokens. This PR adds an opt-in `--jev-brow` engine wire (`BROW`) that returns
the final-row logits for exactly the label tokens asked, and a serve-layer
`/v1/systemone` endpoint that applies the Jev readout math and **per-tuple post-hoc
calibration** (Platt scaling for yes/no, temperature scaling for choice/score) on top. The
same model keeps chat, and can answer decisions inside a chat call, from an image directly,
or decide-then-narrate. It is opt-in per process (`STRATA_JEV_MODES`).

Donor reference: jevcpp on Qwen3.8-27B (q4_k_xl), shipped calibration tuple
`p-jevcpp-qwen38-27b|q4_k_xl|timber|v1|V|all` (T_choice 1.5861). This port runs on
QFN = Qwen3.8-Flash-Next IQ3_XXS MoE (RTX 3090, Strata engine v0.1.43-lineage, 32k ctx).

## 1. Features (the full jevcpp surface)

| Feature | Request | Status |
|---|---|---|
| Core decision endpoint | `POST /v1/systemone` `{state, questions}` — noul (yes/no), choice (letters), score (0-10 levels) | ✅ live |
| Frozen D3 prompt | byte-frozen question block (donor render, `enable_thinking=false`, verbatim string states) | ✅ golden-tested |
| BROW readout | `--jev-brow` engine flag; final-row logits, fail-closed | ✅ engine commit |
| Calibration (M3.2) | post-hoc, argmax-preserving: Platt (a,c) for noul, temperature for choice/score; per-tuple (L/V/D/S) files `calib-<mode>.json`; `POST /admin/calib` hot-swap; donor-schema log rows for `fit.py` | ✅ live |
| V-mode (M3.1) | header `X-Jev-Mode: V` — per-option yes/no branches, `p = softmax(s_yes − s_no)`, name-keyed | ✅ live |
| Image → decision (M3.8) | `image_b64` + `image_mode`: `summary` (vision → 160-tok summary → decide) or **`direct`** (raw image embeddings + question, ONE readout) | ✅ live |
| Decide-then-generate (M3.9) | `"then": {"generate": {...}}` — answers folded into the last user turn, one wire request | ✅ live |
| Chat surface (M3.3) | `POST /v1/chat/completions` with `"jev": {state|messages, questions}` | ✅ live |
| Concurrency | FIFO-serialized engine turns; 7 clients × 3 questions, zero cross-talk (tested) | ✅ tested |

Error contract: donor's `422` + `{"error": {"field": ...}}`. Streaming refused with the
jev extension, `state` XOR `messages`, `image_mode` validated, D+V refused with 422.

## 2. Usage

```bash
# text decision (one decode step)
curl -s $URL/v1/systemone -d '{
  "state": {"engine": "hot", "load": 0.9},
  "questions": {"q1": {"type": "noul", "instructions": "Throttle?"}}}'

# image → decision, direct mode (raw image tokens decide)
curl -s $URL/v1/systemone -d '{
  "image_b64": "'"$(<screen.b64)"'", "image_mode": "direct",
  "questions": {"threat": {"type": "noul", "instructions": "Enemy visible?"}}}'

# decide, then narrate
curl -s $URL/v1/systemone -d '{
  "state": {"hp": 12}, "questions": {...},
  "then": {"generate": {"messages": [{"role": "user", "content": "Narrate."}],
                        "max_tokens": 128}}}'

# MULTIPLE questions, mixed types, ONE state (each question = its own readout;
# cost scales ~linearly until prefix reuse lands)
curl -s $URL/v1/systemone -d '{
  "state": "Reactor at 84%. Coolant loop A nominal, loop B pressure rising slowly.",
  "questions": {
    "threat":  {"type": "noul",   "instructions": "Is loop B a current threat?"},
    "action":  {"type": "choice", "instructions": "Which action first?",
                "criteria": {"inspect": null, "throttle": null, "shutdown": null}},
    "urgency": {"type": "score",  "instructions": "How urgent is intervention?",
                "criteria": ["none", "low", "medium", "high", "critical"]}}}'
# -> {"answers": {"threat":  {"type": "noul", "noul": 0.5429, "calib": "platt_v1"},
#                 {"action":  {"type": "choice", "choice": 0,
#                              "probabilities": [0.830, 0.160, 0.010],
#                              "confidence": 0.7453, "calib": "T_choice_v1"},
#                 {"urgency": {"type": "score", "score": 1.5928,
#                              "probabilities": [0.103, 0.374, 0.376, 0.121, 0.026],
#                              "confidence": 0.2201, "calib": "T_score_v1"}},
#      "margin": top1−top2 present on every answer (cascade-routing number)}
#     "usage": {"prompt_tokens": 303, "readout_tokens": 3}}
# (live response, QFN; note each answer carries its OWN calibrator id)

# INLINE jev on the chat surface — decisions inside an OpenAI chat call; the
# conversation itself is the state (v1-chat template), no "state" field needed
curl -s $URL/v1/chat/completions -d '{
  "model": "jev",
  "messages": [{"role": "system", "content": "You are the plant supervisor."},
               {"role": "user", "content": "Loop B pressure is rising. Inspect now?"}],
  "jev": {"questions": {"inspect":
      {"type": "noul", "instructions": "Should loop B be inspected now?"}}}}'
# -> {"id": "jev-1792055290765", "object": "jev.response",
#     "model": "qwen3.8-flash-next-iq3_xxs-jevport",
#     "jev": {"answers": {"inspect": {"type": "noul", "noul": 0.9230,
#                                      "calib": "platt_v1"}},
#             "usage": {"prompt_tokens": 82, "readout_tokens": 1}}}
# (live response; jev answers ride alongside the normal chat fields, so any
#  OpenAI client gets decisions without a second endpoint)

# MODE GATING — decisions are OPT-IN (donor --modes; the default differs, see section 4d)
# STRATA_JEV_MODES unset            -> "chat" ONLY: a process that never mentions
#                                      jev behaves exactly like stock Strata
# STRATA_JEV_MODES="chat,decision"  -> both (what the jevport presets run)
# STRATA_JEV_MODES="decision"       -> decision-only: no generation path at all
# Unknown tokens fail closed at load (serve refuses to start).
# "decision" gates /v1/systemone + the chat jev extension;
# "chat"     gates plain generation + then.generate narration.
# Gated surfaces return 404 not_found_error (probing learns nothing).
```

Full surface + examples: `docs/JEV-SYSTEMONE.md`. Refit calibration:
`calib/README.md` (the suite IS the procedure). Mode gating:
`STRATA_JEV_MODES` env (`"decision"` = decision-only — no generation path
exists on the instance at all; outputs are probabilities over the caller's own
labels. Unset = `"chat"` only: decisions are opt-in, and the jevport presets set
`"chat,decision"` explicitly. Fail-closed on unknown tokens and on gated
routes).

Why logit readout instead of asking the model for confidence: models asked to state their
own confidence are mostly overconfident with high calibration error (Groot &
Valdenegro-Toro, TrustNLP 2024, see References). Reading label probabilities from the
logits and calibrating them post-hoc on held-out data (Platt / temperature scaling) is the
standard alternative, and it is what this PR does; its measured effect is in section 4a.
Laya reaches the same conclusion from the training side: its probabilities also need a
temperature refit on local data before they can be trusted.

## 3. Test strategy

**Unit (mock engine) — 73/73 green (jev modules; full serve/ discovery 858 run, 0 fail, 11 skipped):** readout math invariants, prompt rendering
(byte-pinned noul/choice blocks — the Yes/No-swap regression is now a golden test),
per-tuple calibration isolation, V-mode softmax, image two-call/direct shapes,
decide-then-generate, chat-surface validation arms, 7×3 concurrency + FIFO
serialization.

**Live suites** (`calib/bench_suite.py`): dev-split fit → held-out test eval,
standard top-label ECE15 + mean NLL, per-item JSONL logs. Guards: Platt slope
a≤0 refused (label/prompt inversion detector — it caught a real bug), temperature
at search boundaries flagged.

**Adversarial audit protocol** (Justin's rule, 2026-10-15): 1-in-10 of all test
items re-verified by hand — text golds re-read from source passages, image items
regenerated from committed seeds with sha256 match against the logged hash, sampled
error images independently re-counted with vision. Result across text+image+real
suites: 0 scoring hallucinations, 0 gold errors, 1 documented CV-Bench ambiguity.
Artifacts: `audit-*.json`, `audit-imgs/`, `*-item-log.jsonl` (920 rows).

**Real-benchmark converters** (`calib/convert_benchmarks.py`): POPE adversarial,
CV-Bench, CountBenchQA via HF datasets-server; banned-field self-test (answer-bearing
fields never enter requests), sha256 image manifest, dev/test image-disjoint, POPE
sampled by image at exactly 50/50.

**Vision strategy** (externally reviewed before implementation):
6-arm design (D/S/S-QC/S-V/blind/shuffled — the last two are contamination
detectors), circular rotation for choice position bias, phased screen→full→frozen
PR gate.

## 4. Benchmarking results (all live, QFN IQ3_XXS, RTX 3090)

### 4a. Pre/post calibration (held-out test, offline replay of identical items)

| tuple (mode\|qtype\|set) | n | acc pre→post | NLL pre→post | ECE15 pre→post |
|---|---|---|---|---|
| L\|noul\|BoolQ | 100 | .890 → **.910** | .298 → **.262** | .084 → **.033** |
| L\|choice\|ARC | 100 | 1.000 → 1.000 | .005 → .007 | .005 → .006 |
| L\|score\|SST-5 | 100 | .520 → .520 | 1.471 → **1.049** | .301 → .154 |
| D\|noul\|POPE-adv | 60 | .867 → **.883** | .204 → **.195** | .122 → **.081** |
| D\|choice\|CV-Bench | 20 | .650 → .650