Pull requests / #1892

#1892 Non-finite logits (#879): a guard with self-healing

closed · @lask3802 · 0 comentarios · En GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Descripción

Refs #879, #606.

## What it changes

Rarely a window's logits become NaN and the reply degenerates into token 0 (`!`) repeated. The poison sits in the
conversation's K/V and recurrent state, so with the prompt cache every later turn of that conversation stays poisoned
until it is read again from token 0. The q8_1 clamps of 0.1.39 and later removed the overflows found so far; reports
on 0.1.41 (#879) show that some cause is left. This PR does not look for that cause. It stops serving the poison:

- Every verify window of `--serve` checks its head logits on the GPU beside the greedy pick: one small kernel writes a
  flag per slice of each row into mapped memory, and the host reads the flags after the sync the window already does.
  No extra sync.
- A flagged window emits none of its tokens; the engine prints one `non-finite logits (#879)` line with the position
  and ends the request with `DONE ... nonfinite` (a batch slot: `BDONE <slot> <n> nonfinite`).
- The next request first drops the live session, every checkpoint, every parked conversation and every idle batch
  slot, and reads its prompt from token 0. A slot still running finishes its reply and is not kept after it.
- The server retries a request that had sent nothing yet once, transparently; a reply that had started ends as
  `"length"` (as `repeat_stop_tokens` does); a second fire on the retry is an error (503 before an answer, an error
  event in a stream). With replicas, the router forgets what that replica held.
- `STRATA_NAN_INJECT=<n>[,<n>...]` (debug) reports the n-th checked window as non-finite, to exercise the path.

On by default, as a fix; `STRATA_NAN_GUARD=0` turns it off (no kernel is recorded).

Covered: the serial decode loop (one GPU or a layer split), the `--pipeline-windows` decode loop of a layer split, the
prompt's short tail (`read_windows`, `read_windows_pl`), and both batch-slot loops.

## What it leaves alone

- Answers without a NaN (checked below, byte for byte).
- The one-shot command line (`strata` without `--serve`) and the SYCL port in `sycl/` (it has its own verify path)
  do not check.
- `repeat_stop_tokens` stays as the backstop.

## Measured

RTX 3080 (PCIe 4.0 x8), Linux, RVN IQ3_S with the usual one-GPU config (`--expert-cache auto --prefill auto --spec 4
--mtp ... --conversation-cache-mib 8192`), this branch's build.

- **Answers**, guard on vs `STRATA_NAN_GUARD=0`, at fixed expert sets (`--pcie-frac 0 --adapt-every 0 --adapt-swaps 0
  --mtp-max-t 1 --suffix-draft 0`, `STRATA_IQ_MT_MIN=1 STRATA_PREFILL_CPU_SHARE=0`), a chat replay of 2 x 4 turns on a
  16K-token system prompt: byte-identical in 3 pairs (one with `STRATA_ROUTE_TAIL_SKIP=0`, one with
  `STRATA_SH_STREAM=0`); two guard-on runs 8 of 8. One more guard-off run differed from every other run from its first
  answer on, and a second guard-off run matched the guard-on runs, so that was a difference between runs of this
  config, not the guard. (Also 8 of 8 on an older base, fb58e0d plus the remote-stage PRs.)
- **Cost**, Nsight Systems, 6 requests x 200 tokens: `logits_nonfinite_kernel` 424 launches, 4.77 us on average
  (7.78 us at most), 2.0 ms against 21.6 s of decode: **0.009%**. A decode rate A/B (40 prompts x 200 tokens,
  on/off/off/on: 40.78 / 40.94 / 40.86 / 40.45 tok/s, on the older base) cannot resolve a difference that small.
- **`STRATA_NAN_INJECT=1,60`**: a new chat's first window flagged -> the server sent the request again from token 0
  and the client saw one normal answer; the 60th window, mid-reply in the next turn -> that reply ended as `"length"`
  after 58 tokens; the turn after it reused 0 cached tokens and answered normally. The engine logged exactly the two
  injected windows.

A real fire was not seen in these runs. On a two-PC remote-stage build (#1748 + #1887) the guard stayed silent through
163 requests of the load that had poisoned about 4 in 10 before that build's own fix, so the guard is the backstop,
not the fix, there.

## Builds and tests

- CUDA 13.0 on Linux (GCC, sm_75 + sm_86); serve tests 697 OK, 11 skipped (`NonFiniteLogits`,
  `test_nonfinite_in_batch_mode`, `test_nonfinite_forgets_the_replica` are new); ctest without a GPU: 24 pass
  (`conversation_cache_test` with the new `clear()` among them), the GPU parity tests not run.
- MSVC 2022 + CUDA 13.0: the touched files compile; `conversation_cache_test` and `conv_cache_test` pass.
- HIP (ROCm 7.0, gfx1100): compiles. SYCL (oneAPI 2026.1.1): compiles.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

En el sitio

Enlaces a install, modelos, releases.