#2690 verify: STRATA_VERIFY_DEFER_CPU_BELOW=<w>, a deferred layer keeps its heavier CPU rows on time (0.1 by default)

说明

Builds on #2689 (`STRATA_VERIFY_DEFER_CPU=1`). With `STRATA_VERIFY_DEFER_CPU` unset nothing changes.

**This PR changes what `STRATA_VERIFY_DEFER_CPU=1` does compared with #2689 alone.** There, every CPU row of a deferred
layer joins late. Here, by default only the rows of routing weight below 0.1 join late, and the others are waited for
in their own layer. `STRATA_VERIFY_DEFER_CPU_BELOW=1` gives #2689's behaviour back exactly.

## What it changes

With `STRATA_VERIFY_DEFER_CPU=1` as in #2689, layer l's CPU expert rows join the residual only before layer l+1's FFN
half, so layer l+1's mixer runs without any of them. That is where the deferral's deviation comes from, and the few
rows with a large routing weight carry most of what the mixer misses (Quality below). `STRATA_VERIFY_DEFER_CPU_BELOW=<w>`
(read with `=1` or `=check`) defers only the CPU rows whose routing weight is below `w` and keeps the others on time:

- In a window that defers, the doorbell publishes the routing weights too (`doorbell_publish`, the variant
  `--remote-expert-opt` uses), and `run` / `service` hand the pool the weights and a partial hook for the layers that
  are deferred.
- The pool computes first the experts of the CPU entries whose weight is >= w, calls the hook (it raises a new mapped
  flag, `h_pflag_`, to the layer's ring), then computes the rest. The rows themselves come out as in one pass: the
  pool cuts its phases by output rows, so only the order of the jobs changes.
- post(l) of a deferred layer waits for that flag and combines the heavy rows (new kernel `copy_rows_by_weight`, the
  plan's GPU rows zeroed as `copy_rows_from_mapped` does). The late addition in pre(l+1) takes the lighter ones.

So layer l+1's mixer misses only the light part of layer l's CPU share. The price is the wait for the heavy jobs in
post(l), which deferring every row hid behind the next mixer.

The values:

| `STRATA_VERIFY_DEFER_CPU_BELOW` | with `STRATA_VERIFY_DEFER_CPU=1` or `check` |
|---|---|
| unset or empty | 0.1, start-up line ends in `(the default)` |
| 0 < w < 1 | the split at w |
| w >= 1 | every CPU row deferred, no split: no partial flag, the same doorbell, graph and pool steps as #2689 (a row of weight exactly 1.0 is deferred too) |
| not a number above 0 | one line, then the default 0.1 |
| set, `STRATA_VERIFY_DEFER_CPU` unset or 0 | ignored, one line |

0.1 is the default because it is the setting measured below: on the RTX 3080 it kept the speed of deferring every row
at low context (+15.8% against +15.4%) and cut the deviation from the default about 6 times (KL 0.096 against 0.556).

The split is turned off at start-up, with one line saying so, and every CPU row joins late (as with `w` >= 1: no
partial flag allocated) in two cases: when the verifier takes the device plan (`STRATA_VERIFY_DEVICE_PLAN=1`), and with
`--peer-device` or a helper cache (`--expert-cache-device1..3`), whose rows land after the CPU's, so there is nothing to
wait for first. The engine tells the verifiers about the peer or helper before their init (`set_pool_gpu_helper`,
beside `set_peer_portable`). `run` and `service` raise the partial flag after every pool call, so a layer the pool did
not split (the stage's last layer; a window with more entries than the plan holds, said once) never leaves a GPU wait
behind. The new flag is released with the others before the engine ends (#267) and shows in the stall diagnostics.

Docs: a paragraph in the deferral's DETAILS.md section.

Where it came from: on our two-machine setup (the remote stage, #1748) the main machine was the bottleneck, and
deferring every CPU row moved the answers more than we liked; keeping the heavy rows on time kept the speed and most of
the answers there, and on one GPU too (below).

## What it leaves alone

- `STRATA_VERIFY_DEFER_CPU` unset: no flag is allocated, the doorbell is `doorbell_publish_res` as before, the window
  graph and the host loop are the default's; the pool computes the same `ps`/`split` booleans and runs its jobs in one
  pass as before.
- `STRATA_VERIFY_DEFER_CPU=1` with `STRATA_VERIFY_DEFER_CPU_BELOW=1`: the same graph and host steps as #2689's `=1`.
- With the split, windows that do not defer still use `doorbell_publish_res` and one pool pass.
- SYCL and Metal: not touched (SYCL has its own `verify.cpp` and `expert_source.cpp`; on Metal the new call is inside
  the deferral's `#if !defined(STRATA_GPU_METAL)` blocks). `verify_kernels.hpp` gains one declaration.

## Measured

GSQ-RCO IQ3_S (ISTA-DASLab), RTX 3080 10 GB alone (PCIe 4.0 x8, 13.0 GB/s) + Ryzen 9 5900XT, DDR4-3000, Linux;
`--serve` with the setup's config as it is (expert cache auto: 1,404 slots in every arm, the PCIe share automatic), one
fresh engine per arm, order unset, every row deferred, the default 0.1 twice, unset. Low context: 12 prompts x 200
tokens, three rounds, mean decode tok/s per round, the median of the three. High context: 3 prompts of ~29-30K tokens
read from 0, 200 tokens each, the median of the three. The binary was this branch before the review fixes that touch
only the peer, helper-cache and device-plan cases (see "How it was checked" for the run that compares the two), and
before the rename (the same steps under other names: the split at 0.1 was asked for explicitly, every row deferred was
the plain `=1`).

| arm | low context decode, tok/s | vs unset | high context decode, tok/s | vs unset |
|---|---:|---:|---:|---:|
| unset (two runs: 58.88, 58.49) | 58.69 | - | 52.10 | - |
| `STRATA_VERIFY_DEFER_CPU=1 STRATA_VERIFY_DEFER_CPU_BELOW=1` (every row, #2689's `=1`) | 67.74 | +15.4% | 61.00 | +17.1% |
| `STRATA_VERIFY_DEFER_CPU=1` (the default 0.1; two runs: 68.42, 67.52) | 67.97 | +15.8% | 58.45 | +12.2% |

At low context the default 0.1 keeps all of the gain of deferring every row (the two are within run-to-run spread). At
~30K it gives back part of it; the high-context medians are noisier (the first of the three prompts was the slowest in
every arm), so read that as "most of the gain", not as a precise ratio. Prompt reads are unchanged (956-963 tok/s at
~30K).

## Quality

Teacher-forced log-probabilities (`STRATA_LOGPOS`, top 20) on the same machine and model, 4 slices of plain text,
5,257 positions, read through the verify windows (`--short-read 100000`) so every position takes the decode path,
`--pcie-frac 0`; mean top-20 KL(unset || run) and top-1 agreement with unset, by position in the prompt. The row
`STRATA_ROUTE_TAIL_SKIP=0` is the scale: how far the shipped default (route tail skip on, since 0.1.42) is from the
engine without it, on the same text.

| run against unset | KL 0-7 | KL 8-63 | KL 64-639 | KL 640+ | KL all | top-1 agreement |
|---|---:|---:|---:|---:|---:|---:|
| unset again (the floor) | 0.0000 | 0.0000 | 0.0004 | 0.0006 | 0.0005 | 99.81% |
| `=check`, the default 0.1 | 0.0005 | 0.0177 | 0.0144 | 0.0122 | 0.0134 | 97.47% |
| `STRATA_ROUTE_TAIL_SKIP=0` (scale) | 0.0042 | 0.1030 | 0.0641 | 0.0740 | 0.0705 | 93.93% |
| `=1 BELOW=1` (every row, #2689's `=1`) | 0.0120 | 0.2732 | 0.2113 | 0.8799 | 0.5557 | 81.42% |
| `=1`, the default 0.1 | 0.0057 | 0.0587 | 0.1212 | 0.0793 | 0.0963 | 92.22% |

The default 0.1 cuts the deviation of deferring every row by about 6 times over the whole text (0.556 -> 0.096) and by
11 times late in the long prompts (640+: 0.880 -> 0.079), which puts it near the scale row: about as far from the
default as the route tail skip's own approximation, for the speed above. `=check` with the default adds the heavy rows
in the combine and the light ones right after it, the default's sums in another order: 0.013, the same order as #2689's
`=check` alone (0.017 on this text), and a fifth of the scale.

## Two GPUs in one PC

Checked by reading the code and building it; not run, as we have no PC with two GPUs.

| setup | the split | why |
|---|---|---|
| layer split over two cards (`--layer-split`, `--split-device`), serial windows | splits | each stage is its own `Verifier` with its own partial flag, plan sink and `cur_layer_`; `drive_pool_split` installs the stage's plan sink (and so its weights and partial hook) in the shared dispatch right before the layer's pool call, which runs on the host thread and calls the hook before it returns; a stage's last layer is never deferred, so no partial flag crosses the hand-off |
| layer split with `--pipeline-windows` | splits | the even and odd windows' verifiers have their own sinks (`split_drive`, `split_drive_b`), and `service` sets the weights and the hook per layer as `run` does |
| `--layer-split K --split-device 0` | splits, serial | as above on one card |
| `--peer-device` | off: every row deferred, as `BELOW=1` | the peer's rows land after the CPU's (`peer->finish`), so there is nothing to wait for first; `generate.cpp` sets `set_pool_gpu_helper("--peer-device")` before the verifiers' init, which turns the split off with one line and allocates no partial flag |
| `--expert-cache-device1..3` (helper caches) | off: every row deferred, as `BELOW=1` | as the peer |
| `--remote-expert-opt` | no deferral | refused by #2689's `defer_ok` |

Not verified: anything on real hardware with two GPUs, AMD multi-GPU.

## How it was checked

| check | how | result |
|---|---|---|
| default path byte-identical | upstream/main 296b59d2 and this branch, env unset, the teacher-forced run above | byte-identical (5,257 positions, top 20). The commit after the review fixes against this run's unset: from position 1,439 on, KL 0.0002, within the floor (the same binary run twice differs from position 1,494 on (KL 0.0005), so this run's floor is not zero) |
| every row deferred unchanged by this PR | #2689's binary and this one, `=1` there and every row here, the same run | within the floor: from position 2,401 on, KL 0.0002, top-1 99.81% |
| the heavy split is bookkeeping only | `=check` with 0.1 vs unset | KL 0.0134, top-1 97.47%: the order of #2689's `=check` (0.0165) |
| the review fixes leave the one-GPU path alone | the final commit and the measured one, `=1` with 0.1, the same run | within the floor: from position 733 on, KL 0.0002; against unset 0.0969 / 92.28% (measured commit: 0.0963 / 92.22%) |
| the split runs and its lines appear | `STRATA_VERIFY_DEFER_CPU_BELOW=0.1 (the default): a deferred layer waits for its CPU rows of routing weight >= 0.1 ...`; with BELOW=1 `... every CPU row joins late` | both lines at start-up (renamed binary); every window size captured deferred, 1- to 6-token windows |
| the rename and the default change no step | the renamed binary against the measured ones, the same teacher-forced run | unset byte-identical to the measured unset; the default against the old `HEAVY=0.1` KL 0.0001 and BELOW=1 against the old `=1` KL 0.0002, within the floor; against unset 0.0969 / 92.22% and 0.555 / 81.42%, as before |
| CUDA build | Linux, CUDA 13.0, sm_75;86, Release, native experts; Windows, MSVC 2022 + CUDA 13.0, sm_86 | built (both) |
| CPU tests | `conv_cache_test spec_prob_test batch_rows_test conversation_validation_test` (+ `conversation_transfer_test` on Linux), `CUDA_VISIBLE_DEVICES=-1` | all pass |

What the checks do not show: that post(l) waits for the heavy rows and not later. A missing or late partial flag ends in
a 20 s window timeout (the GPU waits for it), not in wrong rows; a flag raised too early would read rows the pool is
still writing. The flag is raised only after the heavy jobs' pool call returned, with the same fences as the layer's
own flag.

Not built: HIP (no ROCm toolchain here; the new kernel is a copy of `copy_rows_from_mapped_kernel` with one more test),
SYCL (none of its sources change), Metal (no Mac).

## How to try it

```
STRATA_VERIFY_DEFER_CPU=1                                     # defer the CPU rows of routing weight below 0.1 (the default)
STRATA_VERIFY_DEFER_CPU=1 STRATA_VERIFY_DEFER_CPU_BELOW=1     # defer every CPU row, as #2689 alone
STRATA_VERIFY_DEFER_CPU=check                                 # the check, split at 0.1 t