#6587 [Engine] Inkling-2-Mini SQ-15 never starts on RTX 3060 12 GB: CUDA error 2 (out of memory) at ink_cuda.cuh:54 even after auto-sizing

open · @gillesgagniard · 1 comentarios · En GitHub

Setup & installServer & APINVIDIA / CUDAModels & quantsLinux

Descripción

### Area

[Engine]

**Problem:** Inkling-2-Mini SQ-15 never starts on an RTX 3060 12 GB. The engine loads the weights, prints an auto-sizing line reducing the context from 131072 to ~72,100, then exits with `CUDA error 2 (out of memory)` at `ink_cuda.cuh:54`. All 6 attempts on 2026-11-22 between 14:13 and 14:28 failed the same way.

**Environment:**
- Strata Terminal 0.2.19, source checkout at commit `c6f6427d` (2026-11-22)
- Engine `0.1.65-linux-x86_64-cuda-cuda13` (from the component path in the log)
- Arch Linux, kernel 7.3.5-zen1-1-zen
- GPU: NVIDIA GeForce RTX 3060 12 GB, driver 620.73.04, CUDA UMD 13.5
- 62 GB RAM, 31 GB swap
- Model: `Inkling-2-Mini-SQ-15.strata` (14.35 GB) + `Inkling-2-Mini-Vision.strata` (637 MB), arch `inkling2_loop`
- Served config: `parallel=1`, `gpu 0`, drafting `dspark`, `--max-context 131072 --vram-experts-gb 5.50 --ram-experts-gb 6.78 --vision …` (these expert budgets were chosen by setup)

**Reproduce:**
1. From a source checkout at `c6f6427d`, run `./setup.sh`.
2. Set up Inkling-2-Mini SQ-15 and let it start serving on port 8080.
3. The engine fails at startup every time. The generated run script and the recorded engine command line are in the evidence below.

**Expected:** setup's auto-sizing picks a context that fits the free VRAM and the engine starts, or setup stops with a clear "does not fit" message and what to change.

**Actual:** after a successful weight load, the engine exits with the exact error:

```text
strata-inkling CUDA error 2 (out of memory) at /data/split-pkg/rc219-r2/src/engine/src/inkling2/cuda/ink_cuda.cuh:54
```

What I observed across the six attempts:
- The weights load fine every time: `trunk=2.07 GB`, experts split between VRAM and host, `pf_buf=64MB`, `sm120_tc=0`, `screen=0`.
- The auto-sizing line always reports `1.73 GiB free VRAM` and cuts `max_ctx 131072 -> 72064..72320 for parallel=1`; the OOM then follows anyway.
- One attempt ran with `--max-context 31072` (14:18:05, no auto-sizing line printed) and hit the same error at the same source line, so it is not only the 128K KV-cache size.
- The expert VRAM share varies between attempts (1528–1864 experts in VRAM) with the same result.

This is not a broken CUDA setup or a busy card: the same GPU and driver serve Qwen3.8-Flash-Next IQ3_S fine (that engine holds ~10.1 GiB VRAM in `nvidia-smi`), and during the Inkling attempts the only other VRAM users were KDE and Firefox (~1.1 GiB total).

**Evidence:** startup log, six attempts (private paths sanitized to `/home/USER`; the exact typed setup-wizard choices were not captured beyond what is recorded here):

```text
[strata] 2026-11-22 14:13:27 engine started: .../components/shapes/inkling2/engine/0.1.65-linux-x86_64-cuda-cuda13/strata-inkling --serve --native /home/USER/Strata-data/models/inkling-2-mini/Inkling-2-Mini-SQ-15.strata --max-context 131072 --vision /home/USER/Strata-data/models/inkling-2-mini/Inkling-2-Mini-Vision.strata --vram-experts-gb 5.50 --ram-experts-gb 6.78 --draft dspark
[InkCUDA] Loaded /home/USER/Strata-data/models/inkling-2-mini/Inkling-2-Mini-SQ-15.strata (SQ-15): trunk=2.07 GB (0.22s), experts=1549 VRAM / 1523 host (3.71s, hyb=1, promo=0, pf_buf=64MB), sm120_tc=0, screen=0
[InkCUDA] Auto-sizing max_ctx 131072 -> 72064 for parallel=1 (1.73 GiB free VRAM)
strata-inkling CUDA error 2 (out of memory) at /data/split-pkg/rc219-r2/src/engine/src/inkling2/cuda/ink_cuda.cuh:54
[strata] 2026-11-22 14:17:24 engine started: (same command)
[InkCUDA] Loaded ... (SQ-15): trunk=2.07 GB (0.22s), experts=1550 VRAM / 1522 host (2.93s, hyb=1, promo=0, pf_buf=64MB), sm120_tc=0, screen=0
[InkCUDA] Auto-sizing max_ctx 131072 -> 72128 for parallel=1 (1.73 GiB free VRAM)
strata-inkling CUDA error 2 (out of memory) at /data/split-pkg/rc219-r2/src/engine/src/inkling2/cuda/ink_cuda.cuh:54
[strata] 2026-11-22 14:18:05 engine started: (same, but --max-context 31072)
[InkCUDA] Loaded ... (SQ-15): trunk=2.07 GB (0.22s), experts=1549 VRAM / 1523 host (2.90s, hyb=1, promo=0, pf_buf=64MB), sm120_tc=0, screen=0
strata-inkling CUDA error 2 (out of memory) at /data/split-pkg/rc219-r2/src/engine/src/inkling2/cuda/ink_cuda.cuh:54
[strata] 2026-11-22 14:18:56 engine started: (same command as 14:13)
[InkCUDA] Loaded ... (SQ-15): trunk=2.07 GB (0.24s), experts=1568 VRAM / 1504 host (2.93s, hyb=1, promo=0, pf_buf=64MB), sm120_tc=0, screen=0
[InkCUDA] Auto-sizing max_ctx 131072 -> 72128 for parallel=1 (1.73 GiB free VRAM)
strata-inkling CUDA error 2 (out of memory) at /data/split-pkg/rc219-r2/src/engine/src/inkling2/cuda/ink_cuda.cuh:54
[strata] 2026-11-22 14:27:42 engine started: (same command)
[InkCUDA] Loaded ... (SQ-15): trunk=2.07 GB (1.38s), experts=1528 VRAM / 1544 host (6.94s, hyb=1, promo=0, pf_buf=64MB), sm120_tc=0, screen=0
[InkCUDA] Auto-sizing max_ctx 131072 -> 72064 for parallel=1 (1.73 GiB free VRAM)
strata-inkling CUDA error 2 (out of memory) at /data/split-pkg/rc219-r2/src/engine/src/inkling2/cuda/ink_cuda.cuh:54
[strata] 2026-11-22 14:28:17 engine started: (same command)
[InkCUDA] Loaded ... (SQ-15): trunk=2.07 GB (0.20s), experts=1864 VRAM / 1208 host (2.13s, hyb=1, promo=0, pf_buf=64MB), sm120_tc=0, screen=0
[InkCUDA] Auto-sizing max_ctx 131072 -> 72320 for parallel=1 (1.73 GiB free VRAM)
strata-inkling CUDA error 2 (out of memory) at /data/split-pkg/rc219-r2/src/engine/src/inkling2/cuda/ink_cuda.cuh:54
```

**Reproduction status:** reproduced locally, 6 for 6 attempts on 2026-11-22.

Related to the Inkling-2-Mini support thread (#6442); this is a startup failure report, not a support request.

En el sitio

Enlaces a install, modelos, releases.