#2500 Long generation ratchets to !×256 at the default 20,000 spin bound — 2× Arc Pro B60 (xe), 4,331 host-served experts, engine 0.1.44

Descripción

The server's repeat-stop message points at #606; this is a different failure — the #606 tell is
checked in the signature below.

## What happens

On a two-card layer split with 4,331 experts served by the host, long generations collapse into a
single-token repeat: initial lucid output, partial draft-accept throughout, then `!`×256 at some window, ended
by `repeat_stop_tokens`. Short requests in the same session are clean. The outcome follows the config,
not the drive or the card state — same box, same boot, same model:

| engine config | result |
|---|---|
| `--ple-io direct`, default 20,000 bound | garbage from the first window, **6/6 requests over two days**, byte-identical fingerprint; drafts accepted 100% (124/124, 126/126); 21.7–43.5 tok/s |
| direct, `STRATA_SPIN_MAX=200000` | clean: 334 + 4,876 tok, 65.9/66.4 % accepted, 15.3/28.3 tok/s |
| direct, `STRATA_SPIN_MAX=40000` | clean: 2,705 tok (541 windows), 65.9 %, 24.7 tok/s |
| direct, `STRATA_SPIN_MAX=30000` | req 1 clean (637 tok); req 2 lucid to ~5,688 tok, then `!`×256 (5,944 total; 31.5 tok/s) |
| `--ple-io mmap`, default 20,000 bound, short | clean: 635 + 483 tok, 65.9/66.4 %, 17.3/26.4 tok/s |
| `--ple-io mmap`, 20,000 bound, long (3 requests) | req 1 clean (810 tok, 72.2 %); req 2 lucid to ~3,000 tok, then `!`×256 (3,037 total; 30.8 tok/s; 68.5 %); req 3 ("continue" on the ratcheted conversation) lucid to ~2,660 tok, then `!`×256 (5,889 total; 34.4 tok/s; 69.9 %) |
| `--ple-io mmap`, 20,000 bound, long, second independent run (3 requests) | req 1 clean (564 tok, 67.0 %); req 2 lucid to ~6,004 tok, then `!`×256 (6,260 total; 31.2 tok/s; 69.0 %); req 3 ("continue") lucid to ~1,176 tok, then `!`×256 (1,432 total; 36.5 tok/s; 73.0 %) |

Token positions are exact (server CLI log); window counts are estimates at ~5 tok/window, so: first
ratchet at window 1 (direct 20k), ~608 and ~1,200 (mmap 20k, two independent runs), ~1,138 (direct 30k),
0 over 541 windows (direct 40k).
Single events, so the rates are wide-bracket estimates, not a CDF.

## Signature (why this is not #606)

- Lucid prefix, then a step into the repeat; draft-accept stays partial (68–72 %) up to the ratchet —
  not the 100 %-of-everything collapse.
- **A fresh short prompt answers cleanly before and after** — the #606 tell. The q8_1 clamps are in
  0.1.44.
- The ratchet is per-conversation: continuing a ratcheted conversation ratchets again (req 3 above); a
  fresh conversation starts clean.
- Drive state exonerated within the measured range: the drive was in its fast state (PLE random 4 KiB
  65,267–66,155 rows/s) in every run, garbage included. The one slow-state observation (13,955 rows/s,
  cold) predates all of them.
- mmap does not remove the tail: two independent long runs, first ratchet at ~window 608 and ~1,200 vs
  ~1,138 at direct 30k — so it is not only PLE direct-IO drive contention. I have not attributed the
  residual.

## Environment

- 2× Intel Arc Pro B60 24 GB (8086:e211), xe driver, `--layer-split 25` (card 0: layers 0–24; card 1:
  25–47 + head). PCIe probes at start: 3.1 GB/s (card 0), 14.0 GB/s (card 1).
- Ryzen 5 5600XT (AVX2-only — engine line "this CPU has no AVX-512 … run on AVX-2"), 64 GB DDR4.
- NVMe: WDC WDS100T2B0B (WD Black SN770 1 TB), DRAM-less. Measured: PLE random 4 KiB 13,955 rows/s
  (slow, once, cold) / 65,267–66,155 (fast, every run); expert shard fill 0.45–0.62 GB/s.
- CachyOS, kernel 7.2.9-1-cachyos. Compute runtime (NEO) 26.35.39758; `sycl-ls`:
  `[level_zero:gpu] Intel(R) Arc(TM) Pro B60 Graphics 20.1.0 [1.17.39758]` on both cards;
  oneAPI/SPCL 2026.21.7.0.24_160000. rusticl 26.2.4-arch3.1 present, not selected.
- Engine **0.1.44 at `ed4773eb`**, SYCL AOT build (`bmg-g21`), `sycl/serve/server_intel.py`, one slot.

Engine config (the long run; the baseline differs only in `--ple-io direct` / the spin bound):

```
--pack …/pack-ud-iq4_xs --native …/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf
--expert-profile …/expert-profile-ud-iq4_xs.bin --expert-cache auto --stream-experts
--ple-io mmap --layer-split 25 --prefill auto --max-context 32768 --vram-reserve-mib 1024
--spec 4 --spec-min-p 0.5 --mtp …/mtp/rt
```

env: `STRATA_VERIFY_DEVICE_PLAN=1 STRATA_VERIFY_NO_HOST=0 STRATA_STAGE_TRIM=1
STRATA_STAGER_THREADS=12 SYCL_CACHE_PERSISTENT=0 ONEAPI_DEVICE_SELECTOR=level_zero:*`
(no `STRATA_SPIN_MAX` in any mmap run)

Load: 8,036 slots / 18.08 GiB on card 0; 4,764 of the missing experts mirrored in 10.63 GiB pinned
host memory; 7,445 of 11,776 profiled pairs on card 1; **4,331 experts served by the host**;
1,633 MiB of VRAM free; verify windows captured, no faults before the first request.

All runs under one boot (`uptime -s` 2026-10-09 19:37:32); `dmesg` clear and no devcoredump through
the last run.

Prompts (identical first prompt in every run, same thinking setting): "Tell me about the seven seas."
/ "Expand, and be detailed." / "continue".

## Not tested

Slow drive state; `--ple-io mmap` + `STRATA_SPIN_MAX=40000`; 40k over more than 541 windows; one card.

## Notes

1. The verify path has no per-window "wait ran to its bound" report (contrast `ep_check` in #1798):
   the first expired window here was silent, and everything above was reconstructed from the output.
   A one-line report, or an env switch to enable one, would make this class self-diagnosing.
2. For a layout with a large host-served share on xe, is `STRATA_SPIN_MAX=40000` a sound permanent
   setting, or should the bound follow the host-served count / mirror size?

Full engine logs and per-request server CLI output for every row of the table are available on request.

All test configs run natively and data collected on host. Data collated and summarized by Qwen 3.8.