#6644 [Engine] strata-inkling 0.1.65: Inkling-2-Mini SQ-15 on a 12 GB RTX 3060 — OOM after load (default backend), empty answers + segfault in gemm_sk_q8_1 (hybrid backend)

open · @ludiazv · 3 コメント · GitHub で見る

Setup & installServer & APINVIDIA / CUDAModels & quants

本文

Setup
- Strata v0.2.19, strata-inkling 0.1.65 built from source (cmake --build build --target strata-inkling, -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="75;86", CUDA 13.2.86), served by terminal/serve/server.py
- RTX 3060 12 GB (12288 MiB, PCIe 3.0 x16, nothing else on the card), driver 595.104.02, Debian 13, Xeon E5-2696 v4 (AVX2, no AVX-512), 90 GB RAM
- Inkling-2-Mini-SQ-15.strata (SHA-256 matches inkling_family.py), text only, --max-context 65536, "drafting": "dspark", "parallel": 1

1. Default CUDA backend: out of memory right after loading
strata-inkling --serve --native Inkling-2-Mini-SQ-15.strata --max-context 65536 --vram-experts-gb 7.65 --ram-experts-gb 4.63 --draft dspark
[InkCUDA] Loaded ... (SQ-15): trunk=2.07 GB, experts=2001 VRAM / 1071 host (hyb=1, promo=0, pf_buf=64MB), sm120_tc=0, screen=0
strata-inkling CUDA error 2 (out of memory) at engine/src/inkling2/cuda/ink_cuda.cuh:54
- --vram-experts-gb / --ram-experts-gb (what setup's inkling_fit.place() writes) are ignored here: 7.0, 6.6, 6.2 and 5.8 all load the same 2001 experts in VRAM.
- Same OOM with drafting off, with --max-context 32768, and with --chunk 256.

2. EP/hybrid backend (--vram-reserve-mib 2048): starts, then empty answers and a segfault
[inkling2-ep] Loaded strata (SQ-15) | gpus=1 | experts: 1536 VRAM (GPU0=6.00 GiB), 1536 host (doorbell=off, numa=1, thr=0) | kv=fp16
- A short chat request ("Name three primary colors.", max_tokens 24) returns "".
- A ~40K-token prompt kills the engine (exit -11), every time:
strata-inkling[...]: segfault at 900 ip ... error 4 in strata-inkling[b3f16, ...]
$ addr2line -f -C -e strata-inkling 0xb3f16
strata::quant::sk::gemm_sk_q8_1(unsigned char const*, strata::quant::sk::SkCode, strata::quant::sk::BlockQ8_1_G64 const*, float*, unsigned long, unsigned long, unsigned long)
- Same with --vram-reserve-mib 3072 --vram-experts-gb 5.5, with --cpu-threads 6 (doorbell=on) and with --kv fp8.

Expected: SQ-15 in the "3x loop-pinned hybrid" mode that inkling_fit.py recommends for 12 GiB cards.

Happy to run a debug build or more variants.

関連リンク

インストール・モデル・リリースへのサイト内リンク。