#5678 [Engine] Qwen3.8-Flash-Next IQ3_S on sm_121 (unified memory): prefill stalls at 33,372 of 33,379 tokens, just past the 32,768 chunk; STOP not honoured, restart never finishes
open · @aibix0001 · 4 comments · View on GitHub
BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsLinux
Description
### Area
[Engine]
### Desktop app, Terminal or source?
Source (setup.sh or a build)
### What happened
I sent a single, fresh 33,379-token prompt (no conversation reuse, `cache_n` 0) to `serve/server.py`. It stopped advancing right after the first 32,768-token prefill chunk:
```
[strata] reading the prompt: 33,379 tokens, 10 s so far
[strata] reading the prompt: 32,768 of 33,379 tokens, 20 s so far
[strata] reading the prompt: 33,372 of 33,379 tokens, 22 s so far
[strata] reading the prompt: 33,372 of 33,379 tokens, 32 s so far
... (same line every 10 s) ...
[strata] reading the prompt: 33,372 of 33,379 tokens, 92 s so far
[strata] the engine did not finish the request after it was stopped (STOP) within 300 s; the server ended the engine. The engine and the server lost step (issue #481; ...)
[strata] done: 0 tokens in 400 s (no speed) (error, cancel=True)
[strata] the engine had stopped (exit code -9); starting it again (a minute or two) ...
[strata] still starting (20 s) - please wait ...
...
[strata] still starting (323 s) - please wait ...
```
- The engine's own "no progress for 60 s" watchdog did not fire. The prompt stayed at 33,372 for 70+ s until my client cancelled the request.
- The STOP was not honoured within 300 s, so the server killed the engine (`-9`).
- The restarted engine stayed in "still starting" for more than 20 minutes (`/v1/status` → `"loaded": false`) until I stopped the server by hand. A fresh server start afterwards loaded normally (~7 min, as before) and served requests again.
Expected: the prompt completes as the previous requests did, or the engine's watchdog stops it and the restart comes back.
Not purely length-dependent: on the same server, just before this request, a prompt of the **same length (33,379 tokens, different nonce, `max_tokens` 8)** completed (`33,378 of 33,379 tokens, 27 s` → `done: 8 tokens`). A 33,181-token request with a 256-token answer had also completed (prompt 963 tok/s, decode 53.2 tok/s).
Possibly related, but these have different triggers: #4284 (deadlock at ~25K tokens in a long session), #5564 (stuck prefill after a cancel), #481 / #1317 (server-engine lost step).
### Smallest steps or command that reproduce it
Seen once. I did not retry the same length afterwards (I moved the long cell to 30K, which ran fine in both power modes).
1. `./setup.sh --setup --family qwen --model IQ3_S --context 65536 --vision no --yes --no-start --data-dir <dir>` (unified memory detected; config below)
2. Start `serve/server.py --engine strata --config strata-iq3_s.json` (RLIMIT_MEMLOCK unlimited, n-gram table locked).
3. Send fresh prompts the way `tools/dgx_bench.py` does (`calibrate()` + one measured request per cell): cells 4096, 16384 and 32768 tokens at T=0 with 256-token answers. The hang came on the measured request of the 32768 cell (prompt_n 33,379), in its second round.
### Reproduction status
Reproduced locally (observed once)
### Strata version
Source checkout f33c8c3 (2026-11-09). Engine 0.1.59 built with `spark/build_engine.sh` (`BUILD_VISION=0 BUILD_DSV41=0`). `BUILD.json`: `{"version": "0.1.59", "platform": "linux-aarch64", "archs": "121", "cuda": "13.4", "portable": true}`. **Note: CUDA 13.4, not the 13.0 release toolkit** (the build script says 13.4 "has not been tried").
### Graphics card, memory and system
ASUS ProArt P16 H7607BA laptop, NVIDIA RTX Spark N1X: Blackwell GPU, compute capability 12.1, 48 SMs, 20 Arm cores, 120 GB unified memory (GB10 class). Arch Linux ARM (Omarchy) with a desktop session running, kernel 7.3.0-41, NVIDIA open modules + GSP 620.70.06. AC power, ASUS thermal policy "balanced" (~32 W GPU budget). GPU max 70 °C during this run.
### Model, size and settings
Qwen3.8-Flash-Next GSQ-RCO IQ3_S (ISTA-DASLab, both shards verified by setup.sh). Engine args as written by setup.sh (paths shortened):
```
--pack <data>/packs/iq3_s --native <data>/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf
--ple-gguf <data>/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile data/expert-profile.bin
--expert-cache auto --prefill auto:32768 --spec 4 --spec-min-p 0.5 --mtp <data>/mtp/rt --max-context 65536 --kv int8
--mmap-experts --mtp-q4 all --mtp-hnorm stream --ple-io ram --kv-resident 32768
```
Engine start lines: all 24,576 experts in the GPU cache (46.84 GiB), "KV streaming: 32768 of 65536 cells per QSA layer in VRAM", "PLE table locked in RAM", and the five sm_121 speed switches on by default (PF_FUSED PREFILL_NVFP4 HC_UPMIX SIDE_MIXER SPEC_COST).
The stall position (32,768 + 604 tokens) sits just past both the prefill chunk (`--prefill auto:32768`) and the VRAM-resident KV window (`--kv-resident 32768`). That is a correlation only, not a diagnosis.
### Logs
The server log excerpt is above. I can attach the full server log and the engine log (`strata-iq3_s.log`) if useful. The engine log has no "no progress" line for this request.
---
These measurements were collected with the help of an AI agent (Claude) on the owner's machine. Every log line quoted here was copied from the actual logs.
Related on this site
Quick links to related pages.