Pull requests / #462

#462 diagnostics: dump every verify window's logits, so greedy decode can be bisected

closed · @constantindjonkam · 0 comments · View on GitHub

BenchmarksNVIDIA / CUDAModels & quantsWindows

Description

## What

A diagnostic that makes greedy decode bisectable, plus the traces that localised a real bug.

Greedy output on a two-GPU split was **not reproducible**: repeated identical requests diverged from
about the 28th generated character. Expert routing was byte-identical across runs and the first
verify window's logits were byte-identical, so both were ruled out by measurement rather than by
argument. What remained needed a way to see **which verify window** stopped reproducing.

`--dump-logits` cannot do that. It is written from the per-token loop, which a native pack leaves at

    if (native_pack) { spec_pos = pos; break; }

before the dump site. (Fixing that is a separate, small PR.)

## What this adds

- **`STRATA_DUMP_WINDOW_LOGITS=<path>`** records row 0 of every verify window - the row that chose the
  next token - with the position, the window length `T` and the accepted count `a`.
  Records are streamed and **no count is declared up front**, so a run killed mid-write leaves a short
  file rather than one that lies about its contents.
- **Env-gated traces, off by default**: `STRATA_TRACE_ADAPT` (whether an adaptive round considered,
  skipped or swapped, and whether the previous round's copies had landed) and `STRATA_TRACE_RES` (a
  digest of the expert residency table per request).
- The four probe scripts used to run all of this, under `bench-probes/`.

The file format lives in its own header so it is testable **without a GPU**:
`window_logits_dump_test` round-trips synthetic windows, and checks that a foreign or missing file is
rejected rather than misread.

## Cost

Default off: one function-local `static` getenv, and nothing opened, copied or allocated on the
default path.

## Evidence it produced

With the per-window dump, on an IQ3_XXS / RTX 3090 run with `--adapt-every 4`:

    windows 0-15   bit-identical across 5 runs
    window 16 (pos=86)   runs fork: 3 of 5 agreed, 2 differed
    8 of 24 windows differing afterwards; max |delta logit| 0.266

which pointed at the adaptive tier and became the fix in
[#463](https://github.com/Niko1221/Strata/pull/463), **merged in 0.1.38**. That release also waits
for the swap by default (`STRATA_ADAPT_NOWAIT=1` opts out), so the property this probe depends on is
on for everyone - not only for people who read the note.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.