#5936 [Engine] GB10 (sm_121): batched prompt attention hangs at context >= 64 tokens, watchdog kills the engine (Edge0-145B-NVFP4, 0.1.55) [AI-generated report]
open · @Dinsmoor · 1 コメント · GitHub で見る
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows
本文
> **This report is completely AI generated.** It was written by an AI coding assistant (Claude) while testing on the reporter's own hardware, and is submitted by the reporter. The measurements and log lines below come from real runs on that machine, but the wording has not been hand-checked. ## Summary On a DGX Spark (GB10, sm_121), Edge0-145B-NVFP4 hangs while reading any prompt of about 72 tokens or more. The GPU never finishes the attention/router step, the engine's own watchdog (#29) ends it after 60 s, and the reload takes about 6 minutes. Coding agents send prompts of thousands of tokens, so the model is unusable on this machine as shipped. ## Environment - DGX Spark (GB10, sm_121, aarch64, 121 GB unified memory), NVIDIA driver 615.71.09 - `strata 0.1.55`, Edge0 repo at `b7900cb`, model Edge0-145B-NVFP4, launched with `edge0-infer serve` - Template defaults (`--expert-cache auto --prefill auto --spec 4 --mtp --kv int8`) plus `--vram-reserve-mib 40000`, `--pool-workers 4`, `--context 131072` ## Behavior - Prompts of 66 tokens or fewer work, including back-to-back requests. - Prompts of about 72 tokens or more hang: ``` strata serve: stall report (engine 0.1.55): stage "reading the prompt (batched): waiting for the GPU (attention, router) at layer 1 of the prompt chunk from token 0" for 60 s; 0 layers served since the last finished step ``` - The engine exits with code -6, the server restarts it, and the client gets a 503 after about 66 s. - With `--prefill 32` the stall reads "from token 64": the first two chunks pass and the chunk starting at position 64 hangs. A 66-token prompt (last chunk is 2 tokens at position 64) passes, and a 72-token prompt (last chunk is 8 tokens) hangs. So multi-token prompt attention at context >= 64 seems to be what fails. - At the stall every expert-pool worker is sleeping, nothing is waiting on disk, the host is idle, and the last decode verify window finished normally. It looks like a GPU kernel that never completes, not a CPU, disk or memory stall. ## Repro 1. Start the model with the settings above. 2. POST `/v1/chat/completions` with `max_tokens: 8`, thinking disabled, and a user message of about 40 repeated words (about 72 tokens). 3. It fails after about 66 s with a 503. A 20-word prompt (46 tokens) returns in about 1 s. ## Already ruled out (each still hangs) - `STRATA_PF_FUSED=0` with `STRATA_PREFILL_NVFP4=fp16` (`=0` is rejected: "expected w4a8, w4a4 or fp16") - `STRATA_PREFILL_CPU_SHARE=0` - `STRATA_SM121_PERF=0` together with `STRATA_PROMPT_ATTN_IMMA=0` - `STRATA_PROMPT_ATTN_OLD=1` - `--prefill 32` (template-locked, so I edited the template value) - `--kv fp16` instead of int8 - `--pool-workers 4` - a larger `--vram-reserve-mib`, so not every expert is resident Not yet tried: `--spec 0` / no `--mtp`, and the `STRATA_ATTN_*` kernel switches. ## Expected A 1,500-token prompt completes. A single 1,493-token request stalled the same way here. ## Related #3979 (prefill stall every 60 s), #29 (watchdog), #267 (verify window). <details><summary>Engine log excerpt (redacted: paths and username removed)</summary> ``` 58530:strata serve: no progress for 60 s during a request (reading the prompt (batched): waiting for the GPU (attention, router) at layer 1 of the prompt chunk from token 64) - stopping the engine so the server starts it again (issue #29) 58531:strata serve: stall report (engine 0.1.55): stage "reading the prompt (batched): waiting for the GPU (attention, router) at layer 1 of the prompt chunk from token 64" for 60 s; 0 layers served since the last finished step (0 = stopped, more = slow) 69155:strata serve: no progress for 60 s during a request (reading the prompt (batched): waiting for the GPU (attention, router) at layer 1 of the prompt chunk from token 0) - stopping the engine so the server starts it again (issue #29) 69156:strata serve: stall report (engine 0.1.55): stage "reading the prompt (batched): waiting for the GPU (attention, router) at layer 1 of the prompt chunk from token 0" for 60 s; 0 layers served since the last finished step (0 = stopped, more = slow) 79900:strata serve: no progress for 60 s during a request (reading the prompt (batched): waiting for the GPU (attention, router) at layer 1 of the prompt chunk from token 0) - stopping the engine so the server starts it again (issue #29) 79901:strata serve: stall report (engine 0.1.55): stage "reading the prompt (batched): waiting for the GPU (attention, router) at layer 1 of the prompt chunk from token 0" for 60 s; 0 layers served since the last finished step (0 = stopped, more = slow) ---- strata generate: expert cache auto: 53.09 GiB free, 40000 MiB reserved (+281 MiB for the draft head, +256 MiB for what is allocated after the cache) -> 7493 slots strata generate: session is up (engine 0.1.55) prefill: the CPU share is ON by default for prompt chunks below 1024 tokens (the idle CPU takes some of the experts the GPU would stream; answers can differ slightly from 0.1.40.3, mean KL ~0.004). STRATA_PREFILL_CPU_SHARE=0 turns it off. strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones) strata serve: STRATA_ROUTE_TAIL_SKIP is off by default with NVFP4 experts (it moved an NVFP4 pack's answers by about as much as its quantization does); STRATA_ROUTE_TAIL_SKIP=7 turns it on strata serve: adaptive overlap: off (STRATA_ADAPT_OVERLAP=1 turns it on) strata serve: adaptive tier: blocking, and the asynchronous tier is tried once the session shows the swaps hold the windows up (--adapt-async 0 or STRATA_ADAPT_ASYNC=0: never) strata serve: stall report (engine 0.1.55): stage "reading the prompt (batched): waiting for the GPU (attention, router) at layer 1 of the prompt chunk from token 0" for 60 s; 0 layers served since the last finished step (0 = stopped, more = slow) ---- [2660] K func=0xbea664c38ac0 [2661] K func=0xbea664b9d2a4 [2662] K func=0xbea664b58224 [2663] K func=0xbea664b23a30 [2664] K func=0xbea664c772a4 strata verify: captured the 4-token window (upload no error, sync no error) K0-STAGE-RESET t=1034541038.372 h_seq=48 K0-LAUNCH-QUERY NOTREADY(busy) strata serve: prompt 46 tokens = 0 reused + 46 read in 929 ms (49.5 tok/s), 2 generated in 156 ms (12.8 tok/s), drafts accepted 3 of 3, 1 checkpoints strata serve: decode expert cache hit rate: 100.0% (170 hits / 170 lookups) strata serve: conversation cache: parked 50 tokens in 53.7 ms; parked=1 bytes=237595276 evictions=0 snapshot_bytes=237595276 reused_kv_bytes=0 strata serve: no progress for 60 s during a request (reading the prompt (batched): waiting for the GPU (attention, router) at layer 1 of the prompt chunk from token 0) - stopping the engine so the server starts it again (issue #29) strata serve: stall report (engine 0.1.55): stage "reading the prompt (batched): waiting for the GPU (attention, router) at layer 1 of the prompt chunk from token 0" for 60 s; 0 layers served since the last finished step (0 = stopped, more = slow) VRAM free when this request began: 8139 MiB expert pool: epoch 924, batch epoch 924: 15 of 15 jobs claimed, 15 done; 4 of 4 workers parked, 4 sleeping; mode 0 expert pool threads: w0=sleeping w1=sleeping w2=sleeping w3=sleeping; host idle for 60424 ms verify window (last window, not the current stage): 4 tokens at position 46, host at layer step 48; the GPU rang 48; flags: served 48, plan (A) 48, copies (B) 48 threads waiting on the disk (state D): 0 of 32 memory: 80880 MiB resident, 0 MiB in swap, 43896 MiB RAM available; 3 major page faults so far 2 s later: expert pool: epoch 924, batch epoch 924: 15 of 15 jobs claimed, 15 done; 4 of 4 workers parked, 4 sleeping; mode 0 expert pool threads: w0=sleeping w1=sleeping w2=sleeping w3=sleeping; host idle for 62425 ms verify window (last window, not the current stage): 4 tokens at position 46, host at layer step 48; the GPU rang 48; flags: served 48, plan (A) 48, copies (B) 48 threads waiting on the disk (state D): 0 of 32 memory: 80880 MiB resident, 0 MiB in swap, 43912 MiB RAM available; 3 major page faults so far strata: released the verify window's GPU waits (#267): the GPU finished in 0 ms ``` </details>
関連リンク
インストール・モデル・リリースへのサイト内リンク。