#2619 `gdn` state left by a fresh prefill is not bit-reproducible across runs at long prompts (2K and 40K are bit-stable, 120K is not)

Description

### Summary
Two independent runs of `tools/conversation_cache_soak.py` on the same box with the same config produced **different
`gdn` state fingerprints for the fresh-prefill (baseline) arm** at a 120K prompt, while the park/restore
(candidate) arm was bit-identical across both runs. Generation output was identical in both runs, so we saw no
semantic impact — but bit-reproducibility of the prefill path is lost. This looks like an engine-side determinism
question, not a conversation-cache defect: the cache is the *stable* side here.

### Environment
- GPU: Radeon VII 16 GB (`0x66af`) + Radeon Pro VII / Instinct MI50 16 GB (`0x66a1`), both gfx906, layer split
- ROCm: HIP 7.14 (hipcc 7.14.60850) from `mixa3607/rocm-gfx906:7.14-complete`; kernel 6.8.0-146-p2p; Ubuntu 24.04;
  `HSA_OVERRIDE_GFX_VERSION=9.0.6`
- Engine: v0.1.42 (`61b3fb5`), gfx906 opt-in build (`-DSTRATA_HIP_GFX906=ON`), working tree clean
- Model: Coder IQ1_M (`--native` pack), local NVMe
- Engine args: `--serve --kv int8 --kv-resident 32768 --max-context 131072 --layer-split 27 --prefill 4096
  --pipeline-windows 2 --ple-row-cache 8388608 --pcie-frac 0 --spec 2`
- Engine env: `STRATA_STATE_HASH=1 STRATA_SPLIT_MTP_BATCH=1 STRATA_STAGE_TRIM=1 STRATA_ARENA_MMAP=1
  STRATA_GFX906_ATTN_QUERY_SWIZZLE=1 STRATA_GFX906_ATTN_REDUCE12=1`
  (all verified present in `/proc/<engine-pid>/environ` inside the container)
- Experts: 100% resident (6,889 + 5,376 = 12,265 slots), `evictions=0` throughout

### Reproduction
`tools/conversation_cache_soak.py` **unmodified**. Two runs, same config, same box, back to back:

    python3 tools/conversation_cache_soak.py --prompt-tokens 2048,40000,120000 --cycles 30 --cache-mib 8192

Each run does an A/B/A: first the baseline arm with `--conversation-cache-mib 0` (fresh prefill, no cache), then the
candidate arm with `--conversation-cache-mib 8192`. Compare the `strata serve: STATE_HASH ...` lines that the
**baseline** arm emits, run 1 vs run 2.

### Observed — `gdn` fingerprint per prompt length, both arms, both runs

| `L` (prompt tokens) | run 1 baseline | run 2 baseline | run 1 candidate | run 2 candidate | baseline across runs |
|---|---|---|---|---|---|
| 2,026 (2K head) | `11588c1477c2b342` | `11588c1477c2b342` | `11588c1477c2b342` | `11588c1477c2b342` | identical |
| 2,062 (2K answer) | `7073c4a151834315` | `7073c4a151834315` | `7073c4a151834315` | `7073c4a151834315` | identical |
| 39,985 (40K head) | `ba8166e26a13cb48` | `ba8166e26a13cb48` | `ba8166e26a13cb48` | `ba8166e26a13cb48` | identical |
| 40,021 (40K answer) | `52f2e6a00b00ffda` | `52f2e6a00b00ffda` | `52f2e6a00b00ffda` | `52f2e6a00b00ffda` | identical |
| **119,987 (120K head)** | **`610f3b1ff5dde7ba`** | **`1dd3577d2a1918d5`** | `1dd3577d2a1918d5` | `1dd3577d2a1918d5` | **DIFFERS** |
| **120,023 (120K answer)** | **`a54d74e32544a13b`** | **`2261bd02e1845136`** | `2261bd02e1845136` | `2261bd02e1845136` | **DIFFERS** |

Every other fingerprinted component (`L`, `kv`, `ple`, `tail`, `dead`, `pooled`, `ple_prev`) is identical across
runs at every length, including 120K. `kv` in particular — the real attention K/V over `[0, L)`, including the
streamed host copy — is byte-identical across both arms and both runs at all lengths.

Also: within a single run the candidate arm emits the 120K `gdn` **10 times and always the same value**; only the
baseline (fresh prefill) differs between runs. So the non-determinism is on the prefill side, not the restore side.

### Output was identical, so this is a reproducibility issue, not (observed) a correctness one
Both runs returned the same known answers and the same continuation tokens at every length. We are not claiming
wrong output. We are claiming that "prefill the same prompt twice on the same box, same config → same `gdn`" does
not hold at 120K, which is what a bit-exactness / determinism expectation would require, and which the soak's
`state` gate implicitly assumes.

### What we ruled out, and what we did NOT establish
- **Not the conversation cache.** The candidate (park/restore) side is stable across runs; the baseline (fresh
  prefill) side is the one that moves. At 2K/40K both sides are stable.
- **`atomicAdd` in the GDN kernels: none.** `grep -c atomicAdd` is 0 in every GDN source file:
  `src/kernels/cuda/native_gdn.cu`, `src/kernels/cuda/fused_gdn.cu`, `src/kernels/cuda/native_gdn_preprocess.cu`,
  `src/kernels/cuda/gdn.cu`, `src/kernels/gdn_parity.cpp`, and all `gdn` headers. The only `atomicAdd` hits in any
  file with "gdn" in its name are 6 counter atomics inside `src/prefill/gdn_rec_parity.cu`, which is a parity test
  harness (`landed` / `late` / `apart` / `sitting` / `out`), not the state update.
- **`grep -i "non-determin"` over `src/`: 0 hits.**
- **We are explicitly NOT claiming the trigger is `prompt > --kv-resident`.** The 40K prompt (39,985 tokens) is
  also larger than `--kv-resident 32768`, so streaming KV is active there too, and it is bit-stable. Whatever
  distinguishes 120K from 40K is not pinned down by this data — that is an open question for you, not a conclusion
  from us.

### Confirmed code facts, for whoever picks this up
1. What the buffer is — `src/program/generate.cpp:3484`:
   ```cpp
   auto gdn_snapshot_bytes = [&](const strata::core::SessionState& s0) -> size_t {
       return (size_t) s0.gdn_alloc * ((size_t) g.ssm_state_size * g.ssm_v_heads * g.ssm_state_size +
                                       (size_t) g.ssm_conv_channels * (g.ssm_d_conv - 1)) * sizeof(float);
   };
   ```
   i.e. per GDN layer: the recursive matrix plus the conv rolling window, ×4 B, × `gdn_alloc` layers
   (21 here, the layer-split 27 carve).

2. The same buffer is already treated as containing non-semantic bytes by your own test —
   `src/core/conversation_snapshot_test.cpp:202`:
   ```cpp
   cuda_check(cudaMemset(ss.gdn_state,91,sizes.gdn));
   ```
   a canary fill, which is the reason we did not immediately call the mismatch a cache bug.

3. The per-layer debug hash already exists on both paths: `src/program/generate.cpp:11789` (serve) and
   `src/prefill/prefill.cpp:4096` ("the GDN states as the prompt path leaves them"), gated on
   `STRATA_STATE_HASH_GDN`.

### Suggested next step
Run the same prompt twice through the **prefill** path above the resident window with `STRATA_STATE_HASH_GDN=1` and
diff the per-layer hashes; if it reproduces, the `gdn` comparison in `conversation_cache_soak.py` /
`conversation_cache_parity.py` should compare **cross-run stability of the restore side** rather than equality to a
single baseline sample, in the streamed region. We can run this on our box if you want a second data point, but we
did not want to file a mechanism claim we cannot support.