#6645 [Engine] strata-inkling: --kv k8v8 and --kv k8v4 SIGSEGV immediately after load on Linux RTX 3090 (sm_86)
open · @danbunnell · 0 comments · View on GitHub
BenchmarksNVIDIA / CUDAModels & quantsLinux
Description
### Area [Engine] (speed, kernels, models, the server and API) ### Desktop app, Terminal or source? Other ### What happened Running `strata-inkling --bench` with `--kv k8v8` or `--kv k8v4` on a Linux RTX 3090 (`sm_86`) switches to the `[inkling2-ep]` backend and immediately exits with `SIGSEGV` (`rc=-11`) right after printing the `Loaded` line. This reproduces on both: - Official `Inkling-2-Mini-SQ-15.strata` (all `3072 VRAM / 0 host` experts loaded cleanly on GPU0, ~14.6 GiB total VRAM used) across `--prompt-len 128` through `245760`, both `--draft off` and `--draft dspark`, and both `--kv k8v8` and `--kv k8v4`. - `unsloth/Inkling-2-Mini-GGUF/Inkling-2-Mini-UD-IQ4_XS.gguf` (`2688 VRAM / 384 host` experts, with or without `--vram-reserve-mib 512` / `--cpu-threads 16`). By contrast, `--selfcheck --native Inkling-2-Mini-SQ-15.strata --kv k8v8` passes (`PASS backend=cuda-ep1`), and default `--kv fp16` / `--kv fp8` (which use `[InkCUDA]`) run `--bench` cleanly at short/32K contexts. Expected: `--kv k8v8` and `--kv k8v4` run `--bench` without crashing on `sm_86` (this is also needed to fit 128K–256K+ contexts on 24 GB cards alongside #6609). ### Smallest steps or command that reproduce it 1. Download `strata-inkling-0.1.65-linux-x86_64-cuda-cuda13.tar.gz` from `v0.2.19` and `Inkling-2-Mini-SQ-15.strata`. 2. Run: ```sh LD_LIBRARY_PATH=./bin ./bin/strata-inkling --bench --native ./Inkling-2-Mini-SQ-15.strata --kv k8v8 --draft off --prompt-len 128 --gen 128 --parallel 1 ``` 3. Same crash with `--kv k8v4`. ### Reproduction status Reproduced locally ### Strata version Strata 0.2.19 · strata-inkling 0.1.65 (commit c6f6427d9de9, linux-x86_64-cuda-cuda13) ### Graphics card, memory and system RTX 3090 24 GB (sm_86) · 32 GB RAM · Linux x86_64 ### Model, size and settings Inkling-2-Mini SQ-15 (14.35 GB) and UD-IQ4_XS GGUF · --kv k8v8 and --kv k8v4 · --parallel 1 · --draft off / dspark ### Logs ```text [inkling2-ep] Loaded strata (SQ-15) | gpus=1 (p2p=pinned, ep_split=0.50) | experts: 3072 VRAM (GPU0=11.44 GiB, GPU1=0.00 GiB), 0 host (doorbell=off, numa=1, thr=0) | kv=k8v8 # exits with signal 11 (SIGSEGV), peak VRAM 14,977 MiB ``` Same with `--kv k8v4`: ```text [inkling2-ep] Loaded strata (SQ-15) | gpus=1 (p2p=pinned, ep_split=0.50) | experts: 3072 VRAM (GPU0=11.44 GiB, GPU1=0.00 GiB), 0 host (doorbell=off, numa=1, thr=0) | kv=k8v4 # exits with signal 11 (SIGSEGV), peak VRAM 14,976 MiB ``` With `UD-IQ4_XS.gguf`: ```text [inkling2-ep] Loaded gguf (GGUF-NVFP4) | gpus=1 (p2p=pinned, ep_split=0.50) | experts: 2688 VRAM (GPU0=11.19 GiB, GPU1=0.00 GiB), 384 host (doorbell=off, numa=1, thr=0) | kv=k8v8 # exits with signal 11 (SIGSEGV), peak VRAM 17,534 MiB ```
Related on this site
Quick links to related pages.