#3638 [Engine] 0.1.44.1, CUDA layer split (sm_120): after hours of serving, any batched prompt read over ~400 tokens ends non-finite, so the #879 guard's retry cannot recover it

本文

Follow-up to #879 (closed). This is the same rig as #3470.

## Setup
- **Build:** Strata v0.1.44.1 built locally (CUDA 13.0, `CMAKE_CUDA_ARCHITECTURES=120`, Release), Linux, driver 595.91.07.
- **Hardware:** 2x RTX 5060 Ti 16 GB, 48 GB RAM.
- **Model and split:** Qwen3.8-Flash-Next GSQ-RCO IQ3_S, layer split auto (K=24).
- **Args:** `--spec 4 --kv int8 --resident-experts --resident-budget-gib 32.14 --vram-reserve-mib 700
  --vram-reserve-later-mib 1265 --remote-expert-opt --pcie-frac 0.35 --spec-min-p 0.70`. Defaults otherwise:
  `--short-read 64`, prompt cache on, `STRATA_ROUTE_TAIL_SKIP=7`, no `--tail-role-token`.
- **Control vector:** loaded but off for every request here (`experimental_speed_projection: false`).
- **Process:** up 8.5 h at the first failure, 73 requests before it (prompts up to 159K tokens). Free VRAM throughout
  was CUDA0 268 MiB and CUDA1 666-684 MiB, so this is not the low-VRAM case from #3470.

## What happened in production
An agent conversation that had grown from 28K to 63K tokens without trouble sent its next request: 69,315 tokens, of
which 63,161 were reused from the cache and 6,154 were new. The client sent it 3 times, and the server retried each one
once after the guard fired: 6 attempts in all. Every attempt failed the same way:

```
strata serve: non-finite logits (#879) in a prompt window at position 69310 (row 0): ... the next request reads its prompt from token 0
strata serve: after non-finite logits (#879): dropped the live session, 5 checkpoints, ...; reading this prompt from token 0
```

- The first attempt reused 63,161 tokens. The other five re-read all 69,310 tokens from token 0 in a fresh session.
- Each failure was at row 0, position 69310: the last `<|im_start|>`, which opens the assistant header.
- With the prompt cache on, the prompt is split at that point (`turn_at`) and the short header goes through the verify
  windows. Those windows are the first logits check after the batched read, which doesn't check logits.

## Replays on the same process (each cold, from token 0, sampled with seed 1)
We recovered the request body from the client's failure dump. It renders to 69,311 tokens, 4 fewer than the original;
the client's dump differs slightly from what it sent. Each attempt below includes the server's own retry.

| arm | change | tokens | first logits check | result |
|---|---|---|---|---|
| a | none | 69,311 | prompt window at the last `<|im_start|>` (69306) | non-finite at 69306, row 0, both attempts |
| b1 | +3 / +11 / +59 whitespace tokens inside the last message (the header moves) | 69,314 / 69,322 / 69,370 | 69309 / 69317 / 69365 | non-finite at the new header position, row 0, every attempt |
| b2 | `"strata_checkpoint": false` (no `turn_at` split, so the header is read batched) | 69,311 | first decode window (69310) | non-finite in the **first decode** window, row 0 |
| c | the newest ~6.1K tokens (the last 2 messages) replaced by neutral text of equal token length | 69,311 | 69306 | non-finite at 69306, row 0 |
| a | none, repeated last | 69,311 | 69306 | non-finite at 69306, row 0 |

All 14 attempts were non-finite.

## Cold prompts of any content, same process
Next, prefixes of the same conversation and then unrelated text, each read from token 0 (`max_tokens` 64):

| prompt | tokens (batched before the header) | result |
|---|---|---|
| the conversation, messages 0-69 (the history the client had built up from the cache without trouble) | 63,162 (63,157) | non-finite at the header, both attempts |
| messages 0-13 / 0-5 / 0-1 (system prompt, tools, first user message) | 31,354 / 28,581 / 27,374 | non-finite at the header, every attempt |
| neutral filler text, one user message, no system prompt, no tools | 25,816 / 868 / 538 / 428 (25,811 / 863 / 533 / 423) | non-finite at the header, every attempt |
| the same filler | 362 / 164 / 32 | finite; a normal reply |

The threshold is a step between a 357-token and a 423-token batched read.

## When it flipped
The same conversation had batched reads of 6,478, 6,379 and 6,600 tokens succeed. The last of these started at
14:48:49 UTC. The five requests after it read 269-403 tokens each on top of the cache, below the threshold, and
succeeded by 14:49:47. After 3 minutes idle, the next request (6,154 tokens batched) failed at 14:52:42. The process
flipped in that 4-minute window. It was the only client then: plain text, no images, no sampling or control-vector
fields in the requests, and no kernel, Xid or driver messages in the journal.

## The same prompts on a fresh process
After a restart (same build, same args), each read cold from token 0: the 428-token filler, the 25,816-token filler,
messages 0-69 (63,162) and the full capture (69,311). All four came back finite and replied normally up to
`max_tokens` 64, with no #879 lines in the log.

## Reading of the results
- **Not the content:** unrelated filler fails the same way once the batched part is over about 400 tokens. The same
  conversation's earlier 6K-token batched reads succeeded, before the flip.
- **Not the window geometry:**
  - The failure follows the header when the header moves (b1).
  - It still fires when there is no prompt window at all and the first check is the first decode window (b2).
  - The non-finite value is already in the state when the batched read ends. The window only reveals it.
- **Something in the process's state:** the same build read prompts of up to 159K tokens cold earlier in this process's
  life. After 8.5 h and about 75 requests, any batched read above roughly 400 tokens produces non-finite logits, and
  short reads and decode are still fine.
- **Not recoverable by the retry:** a re-read from token 0 in the same process repeats the same computation and fails
  the same way (every attempt so far). The guard does what it says, but this class needs the retry to change something, for example:
  - read with `STRATA_ROUTE_TAIL_SKIP=0`;
  - compute the experts that are missing from the GPU on the CPU for the retry;
  - or at least report "deterministic at position P" so a client stops sending the same request.

## Still to test
- **What ages:** what changes in the 4-minute window, and whether adaptive expert swaps (`--adapt-swaps 0`) are involved.
- **What changes in the batched read at about 400 tokens**, and whether the expert placement after adaptive swaps is involved.
- **`STRATA_DBG_NAN=1`**, to find where the first non-finite value appears during the batched read.

## Questions
1. Is a non-finite residual at the end of a batched read a known path, separate from #879's MoE-output poisoning? What
   does the batched path switch between about 360 and 420 tokens (a kernel, a chunk or tile size, the expert path)?
2. Is there a per-request knob for a retry that changes the numerics, or a way to make the guard's retry do that?
3. Does the batched prompt path have a cheap non-finite check per chunk, so the failure can be placed, not just seen
   at the end?

For contrast: in our 0.1.40 failures (before the guard), the "!" run began 17-64 tokens into decode after a short
11-17 token read. That is a different signature from this one.