#3631 [Engine] STRATA_WINDOW_TRACE: a record per decode window (opt-in, record only)

说明

## Problem and result

`STRATA_DECODE_TIMING=1` gives one line per request: the average decode window. It cannot show what happens across windows: a cost paid at one window boundary (an adaptive swap, a `STRATA_RAM_ADAPT` move) and what it saves in the windows after it, or which windows are slow and what they read.

With `STRATA_WINDOW_TRACE=<path>` the serial serve loop appends one 216-byte record per decode window: when it started, its boundary work (the adaptive tier's wait, the `STRATA_RAM_ADAPT` moves, the `--adapt-async` step), the verify window, commit and draft, and the expert reads it made (VRAM hits, CPU expert entries, RAM and file reads and bytes, RAM moves, adaptive swaps). `tools/window_trace.py` reads it (summary, or `--csv` for every window). Unset, nothing is timed or written.

Measured on an RTX 3060 12 GB, IQ3_XXS, 120 ShareGPT requests (about 12,000 windows, 2.6 MB of records), trace off / on / off, fresh engine each. This was measured on v0.2.1 with #3060 applied (`e26c68d6`) plus this change, before v0.2.2; the change itself is the same:

| | off | on | off |
| --- | ---: | ---: | ---: |
| decode tok/s | 38.52 | 38.57 | 38.56 |
| prompt tok/s | 104.3 | 103.6 | 104.4 |

The records cover 99.99% of the request's decode time, and their count equals the windows in the `strata decode timing:` lines.

What it showed, as an example: #3060 against main's `STRATA_RAM_ADAPT=8` saves 4.40 ms per token on this workload. 92% of it is the verify window (expert reads from the file 40.9 → 17.9 per window, VRAM hits and CPU entries unchanged), 7.5% the boundary (a move 0.91 → 0.17 ms). The decode timing line alone could not separate the two.

## Related discussion

#610 (`STRATA_DECODE_TIMING`, where a decode window's time goes) and #1864 (its serial sum). This is the same loop, per window instead of per request.

## Approach

All in the serial serve loop of `src/program/generate.cpp`: a static `FILE*` opened once from the variable (`nullptr` when unset), clock reads at the points the decode timing already uses plus three around the boundary work, and counter snapshots of fields that already exist (`ver.ms_wait/ms_pool`, `drive.d.*`, `src.ram_reads()/file_reads()/file_read_bytes()/ram_adapts()`, the adaptive swap count). The clock reads, counter reads and writes are all behind `if (wt_f)`; unset, a window does five null checks and initialises a few locals (an empty snapshot, the flags word, four time points), and no existing variable is written. The file starts with a 12-byte magic and the record size; a `static_assert` pins the record at 216 bytes, which the reader checks. The file is flushed at the end of each request. The `--pipeline-windows` loop and the batch loop are not traced. `docs/DETAILS.md` has a paragraph next to the decode timing one.

## Validation

**Passed:**
- CUDA sm_86 Release (CUDA 13.4, GCC 15.3) and HIP gfx1100 Release (ROCm 7.10 SDK, clang 22) builds of v0.2.2 (`011c9e60`, today's main) + this commit, clean build folders: compile and link, no warnings in the changed lines.
- Build commands: `cmake --build build-cuda-sm86 --target strata -j 14` and `cmake --build build-hip-gfx1100 --target strata -j 14` (revision `fe37cbb3`, environments above).
- `python -m unittest tools.test_window_trace`: 4 tests (record size, order, a partial last record, a foreign file).
- Output comparison: stock v0.2.2 (`011c9e60`) against this build (`fe37cbb3`), `STRATA_WINDOW_TRACE` unset: the same standard-v1 `performance-short-r1` request once per build, after the same warm-up. IQ2_XS, 77 prompt tokens (0 reused), 413 generated tokens; returned content byte-identical (807 characters, SHA-256 `9e91210d…`). `--adapt-async 0 --pcie-frac 0`, `STRATA_ADAPT_NOWAIT=0 STRATA_PREFILL_CPU_SHARE=0 STRATA_IQ_MT_MIN=1`, RAM adaptation unset. One output pair only.
- Default path, checked statically: the commit only adds lines (no line of v0.2.2 is removed or changed). Taking the 64 added lines out of the patched `generate.cpp` gives a file byte-identical to v0.2.2's (sha256 `89cdeb94…`). With the variable unset `wt_f` is `nullptr`, so nothing is timed, read or written; the added locals are not read by any existing code.
- Trace cost and coverage: the table above (v0.2.1 + #3060 `e26c68d6` + this change, IQ3_XXS read in place, `--resident-budget-gib 20.5`, `STRATA_RAM_ADAPT=1`, `--spec 3 --mtp-max-t 3`, greedy, ShareGPT 120 requests, one run per column).
- Smoke: 3 requests, 560 records = 75 + 197 + 288 windows of the engine's lines; verify ms per window equal to the decode timing line's.

**Failed:** none observed.

**Not run:**
- SYCL and Metal: they compile their own copies of the serve loop and are not changed.
- The cost check on v0.2.2 itself (it was run on v0.2.1 + #3060, above).
- AMD GPU run (HIP was built only).
- Windows.

## Limits

- Serial loop only. Times are wall time on the engine thread; `wait`, `pool` and `jobs` are the verifier's and the pool's own sums (overlapping threads). GPU stages are not split (no events, so no synchronisation is added).
- The RAM-move counter splits moves without a file read only where the engine counts them (#3060); here it is written as -1.
- One machine (RTX 3060, 32 GB RAM, NVMe) for the cost check.