#5564 [Engine] Qwen3.8-Flash-Next IQ3_XXS 0.1.58: prefill stuck at ~111K tokens — cancel leaves the engine busy-waiting, GPU 0%, one request slot — reproduced on 2× RTX 2080 Ti sm_75

open · @Poxiaoz · 3 コメント · GitHub で見る

BenchmarksServer & APIMulti-GPUNVIDIA / CUDAModels & quantsLinux

本文

**Regression from the 0.1.55 / 0.1.58 line — the same symptom family reported in #4284 and #5404, but on
sm_75 (2× RTX 2080 Ti 22 GB, Linux, engine 0.1.58 from `strata-qwen3.8-flash-next`).**

## Rig

- 2× RTX 2080 Ti 22 528 MiB (sm_75, no NVLink), driver 610.57.04
- Intel Xeon E5-2696 v3 (18C/36T), 62.6 GiB RAM
- Proxmox LXC v103, Ubuntu container; engine `0.1.58`, pack `gsq_iq3_xxs` (GSQ-RCO IQ3_XXS
  Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS, 2 shards 75.8 GB), `--layer-split 26` explicit
- config: `--max-context 262144`, `--prefill 8192`, `--prefill-pipe 1`, `--batch 1`
  (added during this report), `--spec 4 --spec-min-p 0.5 --mtp-max-t 4`, `STRATA_EXP_MODE=8`
- not used: `--vision`

## Reproduction

From the engine log, after `--prefill 8192` was changed to the `--batch 1` config (the switch from `--parallel =0`):

```
Nov 08 05:49:31 reading the prompt: 45,626 of 111,349 tokens, 12 s
Nov 08 05:49:51 reading the prompt: 94,778 of 111,349 tokens, 32 s
Nov 08 05:49:58 done: 0 tokens in 39 s (cancel, cancel=True)         ← first cancel
Nov 08 05:50:05 reading the prompt: 111,162 of 111,349 tokens, 7 s   ← next one read from the start, not resumed
Nov 08 05:50:06 thinking: 3 of max 32768 tokens, 47.2 tok/s, 8 s
Nov 08 05:50:08 writing a tool call: run_code: 211 of max 32768 ...   ← last log line; engine stuck ever since
```

Symptoms of the stuck engine (measured on the live PID at 2026-11-08 ~05:56 local):

| observable | value |
| --- | --- |
| `/health` | HTTP 200 (serves a stub, not the model) |
| `/v1/status`, `/v1/messages`, `/v1/chat/completions` | curl timeouts, 0 bytes |
| engine process | alive, RSS ~54 GB, %CPU ~230-350 (busy-wait in `futex_do_wait`) |
| GPU util | 0 % on both cards, VRAM 21.7/21.5 GB (model resident) |
| engine log | **last line 05:50:08, no further output for 12+ min** |
| dmesg | NVMe PCIe AER (correctable), no OOM, no D-state thread, no GPU fault |

The engine does not self-kill ("no progress for 60 s" watchdog) — it stays alive with 0 GPU activity and
never writes another log line. `--batch 1` does **not** recover it: a same-config restart is the only way
to get the slot back.

## What "cancel" leaves behind (the suspected root cause)

1. `--prefill 8192` chunked the read; the first cancel arrives mid-chunk (token ~94K of 111K).
2. The engine stops the chunk, and the *next* request starts reading the prompt **from token 0** (111,162
   of 111,349 at 7 s) — the cancel does not resume from the partial progress.
3. After that second request reaches the tool-call stage (`writing a tool call: run_code: 211 tokens`)
   the engine goes silent and never returns: no `done`, no `cancel`, no watchdog.

So: a client cancel that lands during a chunked `reading the prompt` leaves the engine in a state where a
**follow-up request is not served at all**, the process busy-waits on a futex, and the only recovery is a
full service restart. This matches #4284 ("Deadlock on ~25K-token context") except that #4284 reproduces
without a cancel and on sm_120; here a cancel mid-prefill is what triggers it.

## Workaround / mitigation

- Restart the service to recover (health takes ~70 s to come back).
- After restart, `/v1/chat/completions` answers. Smoke tests after restart:
  - short prompt (58 tok) → 200 in ~1.2 s, `done: 8 tokens ... cancel=False`
  - 12K-ish prompt → 200
  - GPU util after a reply → not stuck at 0; VRAM unchanged.
- Not yet verified whether the cancel is avoidable: with `--batch 1` a cancel still lands mid-chunk when a
  second request arrives; the first request's cancellation should cancel the in-flight chunk, not the whole
  session.

## Questions for the maintainers

1. Is a cancelled `reading the prompt (chunked)` allowed to leave the engine in a state that loads 0 tokens
   and then never serves the next request before restart? Should the engine resume the unread chunk after a
   cancel rather than re-reading from token 0?
2. Is the silent stop after `writing a tool call` the same failure as #4284 (busy-wait + no watchdog)? The
   engine here did **not** print "no progress for 60 s" — it just stopped.
3. If `--batch 1` is meant to be safe under cancellation, the cancel path needs to release the slot;
   currently the slot is held while the process hangs.

関連リンク

インストール・モデル・リリースへのサイト内リンク。