#6605 [Engine] Inkling-2-Mini (CUDA, RTX 5090): prefill hangs for prompts above ~20-27k tokens, GPU idle, no token generated

open · @sfrmattos · 0 comentários · No GitHub

Server & APINVIDIA / CUDAModels & quantsLinux

Descrição

Problem: strata-inkling stops making progress while reading a long prompt. The request never produces a token
and never errors; /metrics stays at "reading the prompt" with GPU utilization 0%. Short prompts are fine.

Environment: Strata 0.2.19 / strata-inkling 0.1.65 (linux-x86_64-cuda-cuda13), Linux x86_64,
RTX 5090 32 GB, 61 GB RAM. Inkling-2-Mini SQ-20, --max-context 262144, --kv fp8, --parallel 1,
vision on (GPU reader), DSpark drafter. VRAM in use after load: ~30.4 of 32.6 GB.

Reproduce: serve with the settings above, then POST /v1/chat/completions with a single user message of
repeated filler text of increasing length (max_tokens 100, no tools), straight to the server (no agent client).

Expected: the prompt is read and the answer starts.

Actual (prompt tokens -> result):

```text
~2.1k  -> ok, 1.0 s
~8.4k  -> ok, 2.0 s
~17.9k -> ok, 3.6 s
~27k   -> no response in 40-60 s, repeated on a fresh restart each time
~48k   -> no response in 100 s
```

/metrics during the hang: state "reading", phase "reading the prompt", prompt_tokens 32298, elapsed 522 s,
generated 0, gpu_util 0. A stuck request also keeps a client connection open until the server is restarted.

Other observations:
- Asking for 6.3 GB less memory by moving experts to RAM (--vram-experts-gb 13.5 --ram-experts-gb 2.2)
  did not change anything: the engine reported "experts=3072 VRAM / 0 host" and VRAM use stayed the same,
  and the hang stayed too. So the split is ignored for SQ-20 here.
- --kv fp16 at 256k does not start on this card: CUDA error 2 (out of memory) at ink_cuda.cuh:54
  (same location as #6587 on another card). The launcher also does not pass its own --kv to this engine
  (only --set kv=fp8 reaches it).
- Not tested: other quants, --context 128k, other KV modes, other GPUs.

Reproduction status: reproduced locally, several times. Cause unknown; the idea that low VRAM headroom is
involved is a guess, not confirmed.

Related: #6587 (same OOM site), #6598 (long-prompt crash on another serve path).

No site

Links install, modelos, releases.