#6611 Inkling 2 Mini (0.1.65, Windows CUDA, 16 GB): empty answers from hidden thinking; experts forced into VRAM (SQ-15 pages, hybrid crashes)
open · @burakwbulut-arch · 1 commentaires · Sur GitHub
Description
**Setup:** Windows 11, RTX 5060 Ti 16 GB (sm_120, driver 621.24), 96 GB RAM, Strata 0.2.19 / `strata-inkling` 0.1.65 `windows-x86_64-cuda-cuda13`, installed with `setup.py --family inkling-2-mini --model SQ-15 --context 8192 --vision no --yes`.
### 1. Empty `content` on longer prompts (hidden thinking span)
With the default server config, a translation request with ~360–1300 prompt tokens comes back as `content: ""` with `finish_reason: "length"`. With a larger `max_tokens` (1500) the answer appears after ~800 completion tokens. Short prompts ("Merhaba, nasılsın?") work.
`inkling_dialect.template_kwargs` forces `reasoning_effort="none"` ("Thinking effort level: 0"), but the model still writes a `<|content_thinking|>` span. `_emit_typed` silently drops it, so the client gets nothing and the token budget is spent. Passing `reasoning_effort` / `chat_template_kwargs` changes nothing.
Local workaround that fixed it: in `render_prompt`, when there are no tools, append `<|message_model|><|content_thinking|><|end_message|><|message_model|><|content_text|>` to the prompt. After that the model answers right away. Without a system prompt it sometimes writes its reasoning as plain text, though.
### 2. All experts forced into VRAM: SQ-15 pages on a 16 GB card, the hybrid path crashes
- The log always shows `experts=3072 VRAM / 0 host`, even when the config has `--vram-experts-gb 10.00 --ram-experts-gb 2.30` (setup wrote `12.28 / 0.00`).
- With SQ-15, VRAM sits at 15.8 / 16.3 GB. `--bench --prompt-len 1024 --gen 256` gives **prefill 156 ppts, decode 18.5 tps** (24.6 with `--kv fp8`). During the run the GPU shows 100 % utilization at only ~68 W (TBP ~180 W), which looks like WDDM sysmem paging.
- `--bench ... --vram-reserve-mib 2500` switches to the `inkling2-ep` path (`2802 VRAM, 270 host`) and then exits with `0xC0000005` (access violation).
- SQ-10 on the same card benches at **prefill 1606 ppts, decode 86.5 tps**, so the kernels themselves are fine. But only 0.64 GiB stays free after load (`Auto-sizing max_ctx ... -> 2816`). With the default `kv` the context is clamped to 256, and a longer prompt then crashes the server (`CUDA error 700 ... ink_cuda_backend.cu:277`, plus cuBLAS errors 13/14 at `ink_runner.cuh:738/955`) instead of being rejected.
The release notes say SQ-15 fits 100 % in VRAM on 16 GiB cards, and that SQ-10 runs in hybrid RAM+VRAM mode from 4 GiB. On this card SQ-15 needs more VRAM than the card has, and the hybrid path crashes.
**Expected:**
- `--vram-experts-gb` / `--ram-experts-gb` are respected, or the experts that don't fit are moved to host automatically, as the MoE design should allow.
- A prompt longer than the clamped context is refused rather than crashing the engine.
Happy to run any diagnostic build or test.
Sur le site
Liens install, modèles, releases.