#6609 [Engine] Inkling-2-Mini on RTX 3090 24 GB: long prompts OOM at ink_cuda.cuh:54 unless --max-context is small (SQ-20 and SQ-15)
open · @warc0s · 1 コメント · GitHub で見る
BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsLinux
本文
### Area [Engine] (speed, kernels, models, the server and API) ### Desktop app, Terminal or source? Terminal (the `strata` command) ### What happened On an RTX 3090 24 GB, Inkling-2-Mini only reads long prompts when `--max-context` is small. With a larger `--max-context`, a prompt of ~7k tokens or more makes the engine run out of memory at `ink_cuda.cuh:54` (or hang in "reading the prompt"), even though the same pack handles a 28k prompt at 32K context. It looks like the engine sizes its buffers from the free VRAM for the full `--max-context` at start, leaving no headroom for the prefill workspace. Related: #6605 (prefill hang on an RTX 5090, split ignored) and #6587 (OOM at `ink_cuda.cuh:54`). This adds data from an sm_86 card, with both SQ-20 and SQ-15 and several context sizes. All runs: `strata launch --family inkling-2-mini --model <pack> --context <ctx> --vision gpu --parallel 1`, DSpark on, one request at a time, prompts sent straight to `/v1/chat/completions` (no agent client), `max_tokens` 64. | Pack | `--max-context` | VRAM in use after load | Longest prompt that worked | First failure | |---|---|---|---|---| | SQ-20 | 262144 | — | — | OOM at load: `ink_cuda.cuh:54` (auto-sizing line: `max_ctx 262144 -> 135296 for parallel=1 (4.84 GiB free VRAM)`) | | SQ-20 | 131072 | — | — | OOM at load: `ink_moe_cuda.cuh:117` (with `--parallel 3` chosen by setup: `ink_cuda.cuh:54`) | | SQ-20 | 65536 | 23.6 GB | 7.2k tokens | 28k-token prompt: stuck at "reading the prompt: 7,168 of 28,449 tokens" with GPU at 0 %, engine log ends with OOM at `ink_cuda.cuh:54` | | SQ-20 | 32768 | 22.4 GB | 7.2k tokens | 28k-token prompt: OOM at `ink_cuda.cuh:54` (VRAM went 22.4 -> 24.1 GB) | | SQ-15 | 131072 | 22.8 GB | 7.2k tokens | 28k-token prompt: OOM at `ink_cuda.cuh:54` | | SQ-15 | 32768 | 17.7 GB | **28.4k tokens, OK** (8.1k, 9.8k, 14.2k and 28.4k all fine, prefill 552–591 tok/s, VRAM peak 19.9 GB) | — | Expected: either the engine keeps enough headroom for prefill at the context it accepted, or it starts with a smaller context / refuses the prompt with an error. Other things seen on the same machine: 1. **The VRAM/RAM split is ignored for SQ-20** (as in #6605). Setup passes `--vram-experts-gb 12.49 --ram-experts-gb 3.19` at 256K, but the engine loads `experts=3072 VRAM / 0 host`. 2. **`--vram-reserve-mib` changes backend.** With `--set vram-reserve-mib=3000` the engine switches to `inkling2-ep` and does honour the split (`experts: 2683 VRAM (GPU0=12.49 GiB), 389 host`), starts at 262144 and reports `READY 262144`. But decode drops to ~3.5–6.4 tok/s, a short chat answer comes back with empty `content`, and a 7.2k-token prompt still ends in OOM at `ink_cuda.cuh:54` (exit code -11), after which the server restarts the engine. 3. **The server does not notice the dead engine in the hang case.** After the 28k prompt above, `/v1/models` still answered, `strata launch --status` kept saying `ready`, a new request never got a token (GPU idle), and `strata launch --stop --preset <name>` returned `Could not stop <name>: server_stop_failed`. The `strata-inkling` process had to be killed by hand. 4. Setup's fit for Inkling seems to leave the vision shard out of the VRAM budget: in `strata_control/inkling_fit.py`, `compute()` calls `non_expert_gb(ctx, parallel, vision=False)`, so `--vision gpu` still gets all 15.68 GB of SQ-20 experts placed in VRAM at 128K. (I read this in the code; I did not test whether changing it fixes the OOM.) Note on speed figures: this card runs with a 160 W power limit, so decode speeds here (~37 tok/s SQ-20, ~29 tok/s SQ-15 after a long prompt) are not representative of a stock 3090. The failures above do not depend on speed. ### Smallest steps or command that reproduce it 1. `strata launch --model sq-15 --family inkling-2-mini --context 128k --vision gpu --parallel 1 --no-tui` 2. Send one chat completion with a ~28k-token user message (varied filler text) and `max_tokens` 64. 3. The engine exits with `strata-inkling CUDA error 2 (out of memory) at .../inkling2/cuda/ink_cuda.cuh:54`. 4. Same steps with `--context 32k`: the same prompt is answered. ### Reproduction status Reproduced locally ### Strata version Strata 0.2.19 · engine 0.1.65 (`strata-inkling` 0.1.65 linux-x86_64-cuda-cuda13) ### Graphics card, memory and system RTX 3090 24 GB (sm_86, PCIe 3.0 x8, 160 W power limit, driver 595.91.07) · 256 GB RAM · Ubuntu 26.04.1 LTS (kernel 7.0) ### Model, size and settings Inkling-2-Mini SQ-20 and SQ-15 · 32K–256K context · pictures on (GPU reader) · parallel 1 · DSpark ### Logs Engine log, SQ-20 at 256K (default placement): ```text [InkCUDA] Loaded Inkling-2-Mini-SQ-20.strata (SQ-20): trunk=3.67 GB (1.22s), experts=3072 VRAM / 0 host (5.08s, hyb=0, promo=0, pf_buf=0MB), sm120_tc=0, screen=1 [InkCUDA] Auto-sizing max_ctx 262144 -> 135296 for parallel=1 (4.84 GiB free VRAM) strata-inkling CUDA error 2 (out of memory) at /data/split-pkg/rc219-r2/src/engine/src/inkling2/cuda/ink_cuda.cuh:54 ``` Engine log, SQ-20 at 256K with `--set vram-reserve-mib=3000`: ```text [inkling2-ep] Loaded strata (SQ-20) | gpus=1 (p2p=pinned, ep_split=0.50) | experts: 2683 VRAM (GPU0=12.49 GiB, GPU1=0.00 GiB), 389 host (doorbell=off, numa=1, thr=0) | kv=fp16 INFO engine=strata-inkling-0.1.65 arch=inkling2_loop backend=cuda-ep1 model=Inkling-2-Mini-SQ-20.strata layers=24 visits=48 vocab=200058 chunk=512 batch_slots=1 slot_read=1 kv_dtype=fp16 drafter=dspark images=1 image_pad=200054 parallel=64 READY 262144 stop hold holdb ``` Server log, SQ-20 at 64K, 28k-token prompt (no further progress; GPU utilization 0 %): ```text [strata] reading the prompt: 7,168 of 28,449 tokens, 433 s so far ... [strata] reading the prompt: 7,168 of 28,449 tokens, 543 s so far ``` Server log, SQ-20 at 256K with the `inkling2-ep` backend, 7.2k-token prompt: ```text [strata] the engine stopped unexpectedly (exit code -11). The engine stopped because the graphics card ran out of memory (its log says: strata-inkling CUDA error 2 (out of memory) at /data/split-pkg/rc219-r2/src/engine/src/inkling2/cuda/ink_cuda.cuh:54). [strata] the engine had stopped (exit code -11); starting it again (a minute or two) ... ```
関連リンク
インストール・モデル・リリースへのサイト内リンク。