#2778 Batched prompt read stalls (#1118 family) + prefill copy_i32 illegal memory access, engine 0.1.44

说明

Engine: 0.1.44 (BUILD.json → version: 0.1.44; the stall dump's PDB path is C:\Users\AI-Server\Desktop\Strata\GitHub\wt-144-rel\build-release-zip\strata.pdb, i.e. the official release build)
Server: serve/server.py from the same release
OS: Windows 11, WDDM
GPU: single RTX 2080 Ti 22 GB (sm_75) - no layer split, no --pipeline-windows, no --resident-experts
CPU: Threadripper 3970X, 32 physical cores, no AVX-512; 96 GiB DDR4-3200
Model: Swift-Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS, tools/iq_pack.py native pack

---

Summary

Two separate problems, found while benchmarking --pool-workers:

1. A long prompt stalls in reading the prompt (batched) when another slot is decoding. The #29 watchdog fires at 60 s and stops the engine; the server reloads the model (~2–3 min). Root cause identified and a workaround confirmed - it is #1118.
2. prefill copy_i32: an illegal memory access was encountered on the third round of 16K prompts at --batch 2. The engine is then left at live.state = "unloaded" for 6+ minutes and does not recover on its own. Not resolved.

---

1. Long-prompt stall in the batched reader - same family as #1118

Symptom

A ~16K-token prompt stops advancing partway through the read. The log repeats the same line every 10 s for 130 s+:

[strata] reading the prompt: 14,757 of 14,764 tokens, 131 s so far
[strata] answering: 3 of max 192 tokens, 0.0 tok/s

The watchdog (#29) then stops the engine:

strata serve: no progress for 60 s during a request (reading the prompt (batched), done up to token 14757) - stopping the engine so the server starts it again (issue #29)
strata serve: stall report (engine 0.1.44): stage "reading the prompt (batched), done up to token 14757" for 60 s; 0 layers served since the last finished step (0 = stopped, more = slow)
  wrote the thread stacks to H:\Strata\strata-stall-28168.dmp (attach it to the issue)

From the outside: the port stays listening, the request hangs forever, and the server reloads the model.

A second occurrence stalled at token 7,304 (strata-stall-34384.dmp) - so it happens at different offsets, not one fixed position.

Reproduction

The trigger is one slot decoding while a long prompt is being read - exactly what #1118 describes. Note this made it easy to misdiagnose: a *bare* single 16K request with no other traffic in flight completes fine. It stalls when the harness has another request (a warm-up, or a second concurrent request) running alongside. Read as a pair, the two ingredients are:

- "parallel": 2 (--batch 2)
- one request decoding in a slot while a second, long prompt is admitted and read

Workaround that works: STRATA_BATCH_DECODE_SHARE=0

Adding it to the config's env block fixes it on this machine:

| 16K-token prompt | before | after |
|---|---|---|
| single, no other traffic | stalls 130 s+ | 11.6 s, prefill 1600 tok/s |
| two concurrent | stalls → engine unloaded | 23.2 s, prefill 1613–1630 tok/s |

Re-measured a second time: single 16K in 11.8 s, prefill 1605 tok/s. Consistent.

The cost is the one #1118 already documents: while a prompt is read, the other slot's decode drops to 5–11 tok/s. Acceptable here.

One detail worth flagging: this box already had STRATA_BATCH_YIELD_CHUNKS=0 set in the config, and that alone did not prevent the stall. STRATA_BATCH_DECODE_SHARE is a *different* variable - setting the yield-chunk count to 0 is not a substitute for turning the decode share off.

Binary strings confirm both variables and the log templates exist in 0.1.44:

STRATA_BATCH_DECODE_SHARE
STRATA_BATCH_YIELD_CHUNKS
BSTOP / BYIELD
strata batch: the prompt read gives way at %lld of %lld tokens; slot %d holds it (copied in %.1f ms)
strata batch: BYIELD %d not taken at %lld of %lld%s%s

Question

Is STRATA_BATCH_DECODE_SHARE meant to be user-facing, or is it an internal knob that happened to be the right lever here? If the latter, would the signal path in the yield window be worth hardening so that a prompt read cannot wait on it indefinitely (rather than requiring the variable)?

---

2. prefill copy_i32: an illegal memory access at --batch 2 (unresolved)

With problem 1 worked around, a new failure shows up on the third back-to-back round of two concurrent 16K prompts:

strata verify: captured the batch window over slots 0,0,1,1
strata batch: 116 windows, avg 3.28 rows, 39.05 ms/window = run 35.17 (CUDA0 GPU-reach wait 31.15 + CPU experts 1.80) + commit 1.38 + emit 2.32 + adapt wait 0.18; per stage (ms/window): CUDA0 wait 31.15 + pool 1.80; per layer-window: CPU experts 0.59, VRAM hits 39.48, PCIe 0.00; routed entries per window: VRAM 1894.9 (98.5%), PCIe 0.1 (0.0%), CPU 29.2 (1.5%); adaptive tier (async): 116 rounds started, 419 experts swapped in; 27.6 rows/s over 13828 ms of wall time (admissions included)
prefill copy_i32: an illegal memory access was encountered

After that: live.state = "unloaded", and it stayed there for 6+ minutes while I polled /metrics every 10 s (40 polls, all unloaded). Both in-flight requests got HTTP 503. The server did not restart the engine on its own; I had to kill the processes and relaunch.

The same log also shows the adaptive tier running far off its steady state:

strata serve: asynchronous adaptive tier: 322 rounds, 3888 experts swapped in, 1449.5 ms per round (start to flip, between windows) (arena, copy-in only, batch 0, wait 1, pick inline 0.10 ms/round)

Steady state on this box is ~20 ms/round; 1449.5 ms/round suggests it was not converging.

Relation to existing reports

- #1341 - your reply there notes prefill copy_i32 is the sticky cudaGetLastError() mis-attribution (src/kernels/cuda/qsa.cu:77) and names no real kernel.
- #2157 - you note prefill.cpp:4678 is only the catch of the whole prompt run, so it names no kernel either.

This box is single card, no layer split, no --pipeline-windows, so it is neither of the multi-GPU paths above. It looks like a third manifestation, on the batched path, at --batch 2 with --kv-resident 32768. Not deterministic: 2 of 3 rounds passed, the third failed. 8K prompts at --batch 2 did not trigger it in ~20 runs.

Question

Would STRATA_IO_PREFETCH=1 or STRATA_KV_WIDE_HOST=0 be worth trying here (both were levers in #1407 / #2157), or is there a more targeted STRATA_* switch for the batched prefill path you would rather see tried first? Happy to run any reproducer you name.

---

Config

--pack <swift-iq3_xxs> --native <IQ3_XXS gguf> --ple-gguf <same file>
--expert-profile <expert-profile.bin> --expert-cache auto
--prefill auto:32768 --spec 4 --mtp <rt>
--max-context 131072 --kv int8 --kv-resident 32768
--vision --vram-reserve-mib 1664
--spec-min-p 0.90 --adapt-every 1 --adapt-swaps 160 --adapt-decay 0.97 --adapt-async 1
--pcie-frac 0.15 --batch-mtp --batch 2
env: STRATA_BF16_TC=1, STRATA_SELECT_SIMT=1, STRATA_ROUTE_RESIDENT=1,
     STRATA_BATCH_YIELD_CHUNKS=0, STRATA_BATCH_DECODE_SHARE=0   (last one added as the workaround)

/metrics at idle: pool_workers=31, expert_slots=6918, batch_slots=2, batch_groups=1, version=0.1.44.

Engine start lines of note:

strata generate: expert arena read unbuffered (0 of 32 probe reads from the file cache; 8.7 GiB available, 70.7 GiB of files)
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: token graph hit path: 6918 resident experts, decided on the device

Not an all-resident / zero-doorbell session: zero-doorbell appears 0 times in the engine log, so #1025's doorbell fix is not in play here.

---

Dumps available

- strata-stall-28168.dmp (399,852 B, 2026-10-16 20:16:47) - single 16K, stalled at token 14,757
- strata-stall-34384.dmp (399,012 B, 2026-10-16 20:04:50) - stalled at token 7,304

Happy to attach either or both.

---

Two notes that cost me time, in case they help the docs

1. The stall report and the dump path are written to strata-<model>.log, not to the server's srv.out. I read srv.out first, saw no no progress line, and concluded the watchdog had never fired - it had, with the full stall report and dump path, in the engine's own log.
2. live.state = "unloaded" for 6+ minutes is not self-recovering on this box. Every client times out and the engine only comes back after a manual kill + relaunch. If that is expected behaviour after this class of error, a note in TROUBLESHOOTING.md would save someone the same minutes.[

[strata-stall-28168.dmp](https://github.com/user-attachments/files/33552047/strata-stall-28168.dmp)
[strata-stall-34384.dmp](https://github.com/user-attachments/files/33552048/strata-stall-34384.dmp)
[strata-swift-iq3_xxs.log](https://github.com/user-attachments/files/33552049/strata-swift-iq3_xxs.log)

](url)