#5678 [Engine] Qwen3.8-Flash-Next IQ3_S on sm_121 (unified memory): prefill stalls at 33,372 of 33,379 tokens, just past the 32,768 chunk; STOP not honoured, restart never finishes

open · @aibix0001 · 4 コメント · GitHub で見る

BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsLinux

本文

### Area
[Engine]

### Desktop app, Terminal or source?
Source (setup.sh or a build)

### What happened
I sent a single, fresh 33,379-token prompt (no conversation reuse, `cache_n` 0) to `serve/server.py`. It stopped advancing right after the first 32,768-token prefill chunk:

```
[strata] reading the prompt: 33,379 tokens, 10 s so far
[strata] reading the prompt: 32,768 of 33,379 tokens, 20 s so far
[strata] reading the prompt: 33,372 of 33,379 tokens, 22 s so far
[strata] reading the prompt: 33,372 of 33,379 tokens, 32 s so far
... (same line every 10 s) ...
[strata] reading the prompt: 33,372 of 33,379 tokens, 92 s so far
[strata] the engine did not finish the request after it was stopped (STOP) within 300 s; the server ended the engine. The engine and the server lost step (issue #481; ...)
[strata] done: 0 tokens in 400 s (no speed) (error, cancel=True)
[strata] the engine had stopped (exit code -9); starting it again (a minute or two) ...
[strata] still starting (20 s) - please wait ...
...
[strata] still starting (323 s) - please wait ...
```

- The engine's own "no progress for 60 s" watchdog did not fire. The prompt stayed at 33,372 for 70+ s until my client cancelled the request.
- The STOP was not honoured within 300 s, so the server killed the engine (`-9`).
- The restarted engine stayed in "still starting" for more than 20 minutes (`/v1/status` → `"loaded": false`) until I stopped the server by hand. A fresh server start afterwards loaded normally (~7 min, as before) and served requests again.

Expected: the prompt completes as the previous requests did, or the engine's watchdog stops it and the restart comes back.

Not purely length-dependent: on the same server, just before this request, a prompt of the **same length (33,379 tokens, different nonce, `max_tokens` 8)** completed (`33,378 of 33,379 tokens, 27 s` → `done: 8 tokens`). A 33,181-token request with a 256-token answer had also completed (prompt 963 tok/s, decode 53.2 tok/s).

Possibly related, but these have different triggers: #4284 (deadlock at ~25K tokens in a long session), #5564 (stuck prefill after a cancel), #481 / #1317 (server-engine lost step).

### Smallest steps or command that reproduce it
Seen once. I did not retry the same length afterwards (I moved the long cell to 30K, which ran fine in both power modes).

1. `./setup.sh --setup --family qwen --model IQ3_S --context 65536 --vision no --yes --no-start --data-dir <dir>` (unified memory detected; config below)
2. Start `serve/server.py --engine strata --config strata-iq3_s.json` (RLIMIT_MEMLOCK unlimited, n-gram table locked).
3. Send fresh prompts the way `tools/dgx_bench.py` does (`calibrate()` + one measured request per cell): cells 4096, 16384 and 32768 tokens at T=0 with 256-token answers. The hang came on the measured request of the 32768 cell (prompt_n 33,379), in its second round.

### Reproduction status
Reproduced locally (observed once)

### Strata version
Source checkout f33c8c3 (2026-11-09). Engine 0.1.59 built with `spark/build_engine.sh` (`BUILD_VISION=0 BUILD_DSV41=0`). `BUILD.json`: `{"version": "0.1.59", "platform": "linux-aarch64", "archs": "121", "cuda": "13.4", "portable": true}`. **Note: CUDA 13.4, not the 13.0 release toolkit** (the build script says 13.4 "has not been tried").

### Graphics card, memory and system
ASUS ProArt P16 H7607BA laptop, NVIDIA RTX Spark N1X: Blackwell GPU, compute capability 12.1, 48 SMs, 20 Arm cores, 120 GB unified memory (GB10 class). Arch Linux ARM (Omarchy) with a desktop session running, kernel 7.3.0-41, NVIDIA open modules + GSP 620.70.06. AC power, ASUS thermal policy "balanced" (~32 W GPU budget). GPU max 70 °C during this run.

### Model, size and settings
Qwen3.8-Flash-Next GSQ-RCO IQ3_S (ISTA-DASLab, both shards verified by setup.sh). Engine args as written by setup.sh (paths shortened):

```
--pack <data>/packs/iq3_s --native <data>/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf
--ple-gguf <data>/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile data/expert-profile.bin
--expert-cache auto --prefill auto:32768 --spec 4 --spec-min-p 0.5 --mtp <data>/mtp/rt --max-context 65536 --kv int8
--mmap-experts --mtp-q4 all --mtp-hnorm stream --ple-io ram --kv-resident 32768
```

Engine start lines: all 24,576 experts in the GPU cache (46.84 GiB), "KV streaming: 32768 of 65536 cells per QSA layer in VRAM", "PLE table locked in RAM", and the five sm_121 speed switches on by default (PF_FUSED PREFILL_NVFP4 HC_UPMIX SIDE_MIXER SPEC_COST).

The stall position (32,768 + 604 tokens) sits just past both the prefill chunk (`--prefill auto:32768`) and the VRAM-resident KV window (`--kv-resident 32768`). That is a correlation only, not a diagnosis.

### Logs
The server log excerpt is above. I can attach the full server log and the engine log (`strata-iq3_s.log`) if useful. The engine log has no "no progress" line for this request.

---
These measurements were collected with the help of an AI agent (Claude) on the owner's machine. Every log line quoted here was copied from the actual logs.

関連リンク

インストール・モデル・リリースへのサイト内リンク。