Pull requests / #2466

#2466 engine+serve: the confidence monitor (STRATA_CONFIDENCE=1, measurement only)

open · @MrKop-git · 0 Kommentare · Auf GitHub

Server & APIAMD / HIPModels & quants

Beschreibung

## Summary

An online number for "is the model confident right now" - ActKV's confidence signal (arXiv:2609.31395, section 6), which its evaluation used to detect an insufficient KV budget from the model's own behaviour:

- **Per kept decode row**: ActKV's token confidence `T` = the negative mean log-probability of the top 20 predicted tokens (a higher `T` is a sharper distribution).
- **Per request**: the mean of `T` and ActKV's trend slope `alpha` (sliding min-pool w=64 s=8, bottom 10% kept, first-order fit; negative means confidence is falling). The engine emits `CONF <tokens> <mean> <slope>` just before the request's `DONE`; the server attaches it to the request row and `GET /metrics` carries a `"confidence"` summary (count, mean, last mean, last slope).

Why it belongs in the engine: it is the signal for comparing lossy settings the numbers cannot - `--kv q4_0` against `int8`, a tight `--kv-resident` - and later for budget feedback. The constants are ActKV's evaluation values; nothing here claims they are right for Strata, and no default changes.

## What it changes and what it leaves alone

`STRATA_CONFIDENCE=1` in the environment turns it on. **Without the flag nothing runs**: no copies, no line changes, every answer unchanged. The flag copies the head's logits from the GPU once per window (the `STRATA_LOGPOS` pattern), so it slows the decode - like `STRATA_DECODE_TIMING`, for comparing, not timing. No Metal path (it says so and turns itself off there).

## How it was checked

- `src/core/confidence_test.cpp` (CPU): the math on hand-worked rows (a flat row gives `T = log V`, a peaked row the exact value), the pooling, and the time ordering the trend fit needs - that last one caught a bug in my first cut (sorting the bottom set destroyed the trend).
- `serve/test_server.py` (`ConfidenceMonitor`): the `CONF`-before-`DONE` pairing, a bad `CONF` line refusing to poison `DONE`, the row fields and the summary through `GET /metrics`. **The full serve suite passes: 787 tests, 11 skipped.**
- **Live on this box's GTX 1080 Ti** (IQ3_S, `--serve`, two `GEN`s in one session, `STRATA_CONFIDENCE=1`): a 128-token repeating prompt reports `CONF 7 5.7530 0.000000`, a scattered-id prompt `CONF 1 4.7600 0.000000` - the metric moves the way ActKV's signal should. `slope` is 0 for requests shorter than one pool window (pinned by the test too).

## Not tested

HIP (no toolchain here) and SYCL (separate engine tree). Multi-window trend behaviour on a long request (the live run's requests are short - the slope path is exercised in the unit test, not on hardware). Any threshold for "confidence is too low" - this PR is the measurement, not the alarm.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.