#2689 verify: STRATA_VERIFY_DEFER_CPU=1, a layer's CPU expert rows join at the next layer's FFN half (opt-in)

Descripción

## What it changes

On one GPU with experts in RAM, every decode layer waits for the CPU pool's expert rows before its combine, and the GPU
idles while the CPU computes. With `STRATA_VERIFY_DEFER_CPU=1`:

- post(l) combines without the CPU's rows (shared expert, VRAM hits and the PCIe share only) and does not wait. The CPU
  rows are zeroed by a small kernel (`fill_zero`, not a memset node, so the window graph stays kernel-to-kernel).
- pre(l+1) runs its mixer (attention or GDN), then waits for layer l's flag, copies the rows and adds them to the
  residual with layer l's own routing weights and pending-write coefficient, `2*sigmoid(inject/hc)` (new kernel
  `defer_add_rows`), then does its FFN-side read as usual.

So the CPU computes layer l while the GPU runs layer l+1's mixer. That mixer is the only part that runs without layer
l's CPU rows; its router, experts and everything after see them. **This changes the output** (see Quality).

Not deferred: the stage's last layer, and layer 0 when layer 1 runs the PLE block (pre(1) copies the PLE rows, which
the host writes just before flag 1, and relies on post(0)'s wait). Windows that run without deferral (one line at the
first capture says which and why): batch windows, `--spec-split` token groups, `--remote-expert-opt`, the
all-resident graph (no CPU rows), a stage with no layer to defer, a control vector on the stage.

`STRATA_VERIFY_DEFER_CPU=check` runs the same steps but adds the rows back right after layer l's own combine: no
deferral, the default's sums in another order.

The profile (`STRATA_VERIFY_PROFILE`) and the #649 trace get one more stamp (slot 15) at the end of the mixer's
out-projection: "out-proj" now ends there and the next interval is "late rows" (the late wait and add). Without
deferral "late rows" is the cost of one stamp; the stamp is a graph node only when the profile or the trace is on.

Docs: a DETAILS.md section next to the route tail skip and the PCIe share.

Where it came from: we run a two-machine setup on top of the remote stage (#1748), where the main machine's GPU waits
for its CPU pool in every layer and was the bottleneck. Deferring the CPU rows by one mixer helped there, and the same
wait exists on a single PC with one GPU, which is what this PR is about and what the numbers below are from.

## What it leaves alone

- Unset (or `0`): no allocation, no graph node, no kernel choice and no host step changes. The env is read once in
  `Verifier::init`; the extra device buffer (`T * k * n_embd` floats) is carved after every other buffer, only when on.
- The host loop (`run`, `service`) is unchanged. Nothing the late addition reads is overwritten before it: pre(l+1)'s
  attention-side read writes `inj_` (not layer l's `inj2_`), and its router (`w_`), the next plan (`plan_`, `skip_`)
  and its doorbell come after the addition. So the host's next expert step never rewrites the doorbell payload or
  `h_ymiss_` earlier than without deferral, and no buffer is doubled.
- SYCL (`sycl/`) and Metal (`metal/`) are not touched. SYCL builds its own, hand-merged `sycl/src/core/verify.cpp`;
  the shared `verify_kernels.hpp` only gains two declarations. Bringing the deferral to SYCL would mean merging
  `late_add` into that file and porting `defer_add_rows` and `fill_zero` to `verify_kernels.dp.cpp`. On Metal the
  two calls of the new entry points are compiled out (`#if !defined(STRATA_GPU_METAL)`), so no link stub is needed;
  the Metal window returns from `init` before the env is read.

## Measured

GSQ-RCO IQ3_S (ISTA-DASLab), RTX 3080 10 GB alone (PCIe 4.0 x8, 13.0 GB/s) + Ryzen 9 5900XT, DDR4-3000, Linux;
`--serve` with the setup's config as it is (expert cache auto: 1,404 slots in every arm, the PCIe share automatic), one
fresh engine per arm, order unset, `=1`, two arms of the follow-up PR, unset again. Low context: 12
prompts x 200 tokens, three rounds, mean decode tok/s per round, the median of the three. High context: 3 prompts of
~29-30K tokens read from 0, 200 tokens each, the median of the three. The arms ran on an earlier build of the
follow-up PR #2690 (this branch + its row split, which that build left off unless asked: its `=1` matches this branch's
within two runs of one binary, below).

| arm | low context decode, tok/s | vs unset | high context decode, tok/s | vs unset |
|---|---:|---:|---:|---:|
| unset (two runs: 58.88, 58.49) | 58.69 | - | 52.10 | - |
| `STRATA_VERIFY_DEFER_CPU=1` | 67.74 | +15.4% | 61.00 | +17.1% |

The prompt read does not change (deferral is decode only): 956-962 tok/s at ~30K in both arms. The high-context
numbers are noisier: the first of the three prompts was the slowest in every arm (44.8-56.3), so take +17% as +15-20%.

## Quality

The output changes. Teacher-forced log-probabilities (`STRATA_LOGPOS`, top 20) on the same machine and model, 4 slices
of plain text, 5,257 positions, read through the verify windows (`--short-read 100000`) so every position takes the
decode path, `--pcie-frac 0`; mean top-20 KL(unset || run) and top-1 agreement with unset, by position in the prompt.
The row `STRATA_ROUTE_TAIL_SKIP=0` is the scale: it is how far the shipped default (route tail skip on, since 0.1.42)
is from the engine without it, on the same text.

| run against unset | KL 0-7 | KL 8-63 | KL 64-639 | KL 640+ | KL all | top-1 agreement |
|---|---:|---:|---:|---:|---:|---:|
| unset again (the floor) | 0.0000 | 0.0000 | 0.0004 | 0.0006 | 0.0005 | 99.81% |
| `=check` | 0.0012 | 0.0099 | 0.0123 | 0.0209 | 0.0165 | 96.77% |
| `STRATA_ROUTE_TAIL_SKIP=0` (scale) | 0.0042 | 0.1030 | 0.0641 | 0.0740 | 0.0705 | 93.93% |
| `=1` | 0.0120 | 0.2732 | 0.2113 | 0.8799 | 0.5557 | 81.42% |

`=check` is the same sums in another order: above the floor, as rounding differences grow along the prompt, and a
quarter of the scale. `=1` is the cost of the feature: about 8 times the scale over the whole text, most of it late in
the long prompts (640+). That is why it is opt-in, and why the follow-up keeps the heavy rows on time.

For scale, DETAILS.md's own numbers (a different protocol: greedy answers):

| reference (DETAILS.md, "Missed experts in a verify window") | mean KL |
|---|---|
| route tail skip (default since 0.1.42), RTX 3060 IQ3_XXS / RTX A4000 Q2_0, greedy | 0.0058 / 0.0032 |
| Q2_0 against IQ3_XXS, the same text | 0.1006 |
| the same engine run twice (the floor) | 0.0006-0.0007 |

#2690 (`STRATA_VERIFY_DEFER_CPU_BELOW`, stacked on this one) keeps the heavy CPU rows on time and cuts most of this deviation for a
small part of the speed.

## Two GPUs in one PC

Checked by reading the code and building it; not run, as we have no PC with two GPUs.

| setup | deferral | why |
|---|---|---|
| layer split over two cards (`--layer-split`, `--split-device`), serial windows | works | each stage is its own `Verifier` with its own layer range, flags, plan, rows and `def_parts_` (`generate.cpp` `stage_ver`, `set_stage`, per-stage `init` under its device); a stage defers its own layers but its last, so the hand-off between cards always follows a combine that waited for its rows; stage 0's `run` syncs its stream before `next_->run`; the pool routes each layer to its stage's plan (`drive_pool_split`) |
| layer split with `--pipeline-windows` | works | two verifiers per stage and parity, each with its own flags; `service` serves one layer at a time on the host thread and the pool call for a layer is synchronous, as in `run` |
| `--layer-split K --split-device 0` (both stages on one card) | works, serial | the same as above on one device (`ver_same`); upstream refuses it with `--pipeline-windows` |
| `--peer-device` | works | the peer's rows are written into the same mapped rows inside the pool call, before the layer's flag; the late addition waits for that flag. Upstream refuses `--peer-device` with a layer split |
| `--expert-cache-device1..3` (helper caches) | works | as the peer: the helper's rows land before the flag |
| `--remote-expert-opt` | not deferred | refused by `defer_ok` (its rows come back reduced through `remote_opt_->copy_rows`); one line says so |

Not verified: anything on real hardware with two GPUs (timing, a second card's driver), AMD multi-GPU.

## How it was checked

| check | how | result |
|---|---|---|
| default path byte-identical | upstream/main 296b59d2 and this branch, both built here, env unset, the teacher-forced run above | within the floor: this branch against upstream differs from position 5,033 of 5,257 on (KL 0.0001, top-1 99.92%), while the same binary run twice differs from position 1,494 on (KL 0.0005), so this run's floor is not zero; upstream against the follow-up branch, which contains this one, was byte-identical. An earlier run of this commit (2026-10-14): byte-identical to upstream, and to itself |
| `=check` within rounding of the default | `STRATA_VERIFY_DEFER_CPU=check` vs unset, the same run | KL 0.0165, top-1 96.77% (table above): a quarter of the scale row |
| the rename (`STRATA_VERIFY_DEFER` to `STRATA_VERIFY_DEFER_CPU`) changes no step | this branch's binary before and after it, `=1`, the same run | within the floor: KL 0.0002, top-1 99.81% |
| `=1` runs and its lines appear | `strata verify: STRATA_VERIFY_DEFER_CPU=1 (changes the output) ...`, `capturing the N-token window (...), deferred` | both lines; the 1-, 2-, 3-, 4- and 6-token windows deferred (earlier run of this commit) |
| CUDA build | Linux, CUDA 13.0, sm_75;86, Release, native experts | built |
| CPU tests | `conv_cache_test spec_prob_test batch_rows_test conversation_validation_test conversation_transfer_test`, `CUDA_VISIBLE_DEVICES=-1` | all pass |

What `=check` does not check: it adds the rows back in post(l), so it exercises the copy and the arithmetic of the late
addition but not where the wait sits in pre(l+1). Moving the late addition after the FFN-side read, or dropping its
wait, would still pass `=check`. That part rests on reading the code: a missing wait shows only when the CPU is slow
enough for the GPU to reach it first, and I have no test that forces it.

Not built: HIP (no ROCm toolchain here; the new kernels use only `fmaf`, `__expf`, `float4` and plain launches, as
other kernels in this file that already build on HIP), SYCL (none of its sources change), Metal (no Mac).
Not tested: `STRATA_VERIFY_DEVICE_PLAN=1` together with deferral, two GPUs (above), AMD.

## How to try it

```
STRATA_VERIFY_DEFER_CPU=1      # defer (changes the output)
STRATA_VERIFY_DEFER_CPU=check  # the same steps, the rows added back in the same layer (should match the default within rounding)
STRATA_VERIFY_PROFILE=1        # optional: "late rows" is the late wait and add
```

Set it in the engine's environment (for `serve/server.py`, in the environment the server starts the engine with). The
start-up log says `STRATA_VERIFY_DEFER_CPU=1 (changes the output)` and, at the first capture of each window size,
`capturing the N-token window (...), deferred` or why not.

Pre-existing, not changed here: with `STRATA_VERIFY_DEVICE_PLAN=1`, the PLE block and a stage starting at layer 0,
pre(1) copies the PLE rows without waiting for flag 1 when post(0) skipped its wait through the device plan
(`if (ar_on()) wait_flag_ge(m_flag_, 1, cs)` in pre). This change never defers layer 0 under PLE, so it does not
touch that path. A separate issue/PR is the right place for it.