#6598 [Engine] serve Inkling-2-Mini via serve/server.py — SIGSEGV on any prompt above ~40 tokens (--bench is fine)
open · @bhic1973 · 1 コメント · GitHub で見る
BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
本文
### Area
[Engine] (speed, kernels, models, the server and API)
### Desktop app, Terminal or source?
Source (START-HERE.bat, setup.sh or a build)
### What happened
On Strata v0.2.19 with the prebuilt `strata-inkling` 0.1.65 shape engine (RTX 3090 24 GB, sm_86, CUDA 13 build), Inkling-2-Mini loads and answers short prompts, but the engine dies with `SIGSEGV` (`exit code -11`) on the first request whose prompt is longer than roughly 40 tokens. The server then restarts the engine and the next request fails the same way, so the native Inkling path is unusable for real prompts.
The same binary, model, GPU, KV type and build settings complete `--bench` correctly at prompt lengths from 32 to 2048, and `--selfcheck` passes — so this is specific to the `--serve` path, not to the model, the GPU or the kernels.
A second anomaly in the same path: requests through `serve` are ~7–8x slower than `--bench` on identical settings (19 vs 1547 tok/s prefill, 8 vs 42–72 tok/s decode).
Observed:
| request | result |
|---|---|
| `serve`, prompt 36 tokens | OK, content returned (~8 tok/s decode) |
| `serve`, prompt 73 tokens | **SIGSEGV**, exit -11 |
| `serve`, prompt ~145 tokens | **SIGSEGV**, exit -11 |
| `serve`, prompt ~721 tokens | **SIGSEGV**, exit -11 |
| `--bench --prompt-len 32..2048` | OK, no crash |
Expected: the request returns a completion and the engine keeps serving.
### Smallest steps or command that reproduce it
1. Install the official pack without starting it (config below is exactly what setup writes):
```bash
python terminal/setup.py --family inkling-2-mini --model SQ-20 \
--context 131072 --vision yes --models-dir /path/to/models --no-start --yes
```
2. Start the server with that config, unchanged:
```bash
terminal/.venv/bin/python terminal/serve/server.py --engine strata \
--config terminal/strata-inkling-2-mini-sq-20.json --host 127.0.0.1 --port 8080
```
3. Short prompt (37 tokens) — succeeds:
```bash
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model":"x",
"messages":[{"role":"user","content":"Le moteur de recherche moderne indexe des documents, calcule la pertinence, puis classe les resultats pour repondre a la question de l utilisateur avec des exemples concrets."}],
"max_tokens":16,"temperature":0}'
```
4. Same sentence twice (73 tokens) — the engine segfaults:
```bash
python3 - <<'PY' > /tmp/body.json
import json
p = ("Le moteur de recherche moderne indexe des documents, calcule la pertinence, "
"puis classe les resultats pour repondre a la question de l utilisateur avec des exemples concrets. ")
json.dump({"model":"x","messages":[{"role":"user","content":p*2}],"max_tokens":16,"temperature":0}, open("/tmp/body.json","w"))
PY
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' --data @/tmp/body.json
```
5. Control, same binary, no crash:
```bash
EXE=~/.local/share/strata/components/shapes/inkling2/engine/0.1.65-linux-x86_64-cuda-cuda13/strata-inkling
$EXE --bench --native /path/to/Inkling-2-Mini-SQ-20.strata --prompt-len 2048 --gen 8 --gpus 1 --kv fp8
```
Config written by setup (unchanged when reproducing):
```json
{
"exe": "component:strata-inkling",
"args": [
"--native", "/path/to/models/inkling-2-mini/Inkling-2-Mini-SQ-20.strata",
"--max-context", "131072",
"--vision", "/path/to/models/inkling-2-mini/Inkling-2-Mini-Vision.strata",
"--vram-experts-gb", "15.68",
"--ram-experts-gb", "0.00"
],
"cwd": ".../terminal",
"tokenizer": ".../packs/inkling-2-mini-sq-20/tokenizer",
"model_name": "inkling-2-mini-sq-20",
"arch": "inkling2_loop",
"log": ".../terminal/strata-inkling-2-mini-sq-20.log",
"lib_dirs": [],
"port": 8080,
"drafting": "dspark",
"parallel": 1,
"gpu": 0,
"gpus_asked": true
}
```
Also reproduced with: vision off, `--vram-experts-gb` absent, the community `Inkling-2-Mini-UD-Q4_K_XL.gguf` (17.63 GB) with `--kv fp8 --max-context 131072` and with `--kv fp16 --max-context 32768`, and the CUDA 12.9 build of the same 0.1.65. `--parallel 1` in every run.
### Reproduction status
Reproduced locally
### Strata version
Strata 0.2.19 (`git describe` v0.2.19, commit `c6f6427d9`) · engine 0.1.65 · `strata-inkling` build_id `c6f6427d9de9+cuda13.0` (archs sm_75/80/86/89/120) · CUDA 12.9 build tested too
### Graphics card, memory and system
RTX 3090 24 GB · 60 GB RAM · Ubuntu 24.04.5 LTS, kernel 7.0.0-43-generic, driver 580.196.07, Python 3.12.12
### Model, size and settings
Inkling-2-Mini SQ-20 (`.strata`, 19 346 685 952 B, sha256 `0bbfd52344f0dd4779a7807b5f73554a812f24c035e6877b63cd0d499f7e1b97`) + Vision shard (636 526 592 B) · 128K context · pictures on · `--kv fp8` · DSpark · `--parallel 1`. Also the community `Inkling-2-Mini-UD-Q4_K_XL.gguf` (17.6 GB).
### Logs
Kernel, five crashes, same faulting offset (`b97bb`) in the engine binary:
```
<hostname> kernel: strata-inkling[25929]: segfault at 900 ip 00005c96f927b7bb sp 00007ffc51f619c0 error 4 in strata-inkling[b97bb,5c96f91c9000+179000] likely on CPU 2 (core 2, socket 0)
<hostname> kernel: strata-inkling[26110]: segfault at 900 ip 00005e938dba37bb sp 00007ffe2424e9c0 error 4 in strata-inkling[b97bb,5e938daf1000+179000] likely on CPU 2 (core 2, socket 0)
<hostname> kernel: strata-inkling[26153]: segfault at 900 ip 00005d86b78337bb sp 00007ffcbecbfb00 error 4 in strata-inkling[b97bb,5d86b7781000+179000] likely on CPU 0 (core 0, socket 0)
<hostname> kernel: strata-inkling[26609]: segfault at 900 ip 00005ec7c53a17bb sp 00007fff4f7c2ec0 error 4 in strata-inkling[b97bb,5ec7c52ef000+179000] likely on CPU 0 (core 0, socket 0)
<hostname> kernel: strata-inkling[26728]: segfault at 900 ip 000056fa3b0ec7bb sp 00007fff0a5a9d40 error 4 in strata-inkling[b97bb,56fa3b03a000+179000] likely on CPU 2 (core 2, socket 0)
```
Server log:
```
[strata] the engine stopped unexpectedly (exit code -11). The usual cause is running out of RAM: Linux then ends the biggest program (check: sudo dmesg | grep -i -E 'killed process|out of memory'); Windows slows down instead. Close other programs or use a smaller model (Q2_0 / IQ2_XS). The next request starts the engine again. Its log: /path/to/terminal/strata-inkling-2-mini-sq-20.log
[strata] done: 0 tokens in 8 s (no speed) (error, cancel=False)
```
Engine log (the start line repeats after each restart):
```
[strata] 2026-11-22 14:24:18 engine started: .../strata-inkling --serve --native .../Inkling-2-Mini-SQ-20.strata --max-context 131072 --vision .../Inkling-2-Mini-Vision.strata --vram-experts-gb 15.68 --ram-experts-gb 0.00 --kv fp8 --vram-reserve-mib 700 --draft dspark
[inkling2-ep] Loaded strata (SQ-20) | gpus=1 (p2p=pinned, ep_split=0.50) | experts: 3072 VRAM (GPU0=14.60 GiB, GPU1=0.00 GiB), 0 host (doorbell=off, numa=1, thr=0) | kv=fp8
```
`--selfcheck` and `--bench` on the same model:
```
[InkCUDA] Loaded .../Inkling-2-Mini-SQ-20.strata (SQ-20): trunk=3.67 GB (0.37s), experts=3072 VRAM / 0 host (1.46s, hyb=0, promo=0, pf_buf=0MB), sm120_tc=0, screen=1
[selfcheck] PASS backend=cuda (NVIDIA GeForce RTX 3090 (sm_86, 82 SMs, 23.6 GiB VRAM, sm120_tc=0))
[bench] prompt=32 | prefill=441.7 ppts | decode=71.6 tps
[bench] prompt=64 | prefill=750.7 ppts | decode=62.3 tps
[bench] prompt=128 | prefill=1063.0 ppts | decode=60.5 tps
[bench] prompt=512 | prefill=1570.9 ppts | decode=44.8 tps
[bench] prompt=1024 | prefill=1565.3 ppts | decode=49.4 tps
[bench] prompt=2048 | prefill=1546.9 ppts | decode=42.4 tps
```
No OOM in the kernel log; host RAM stayed at ~2 GB of 60 GB. VRAM in `--serve` mode sat at 17 525 MiB of 24 576.
### Related issues
I searched before filing; these are close but none matches sm_86 + `--serve` + `--bench` fine:
- #6580 — HIP gfx1200, illegal memory access in batched prefill (GGUF); short prompts fine.
- #6569 — SQ-10 segfault in CPU expert computation on a 4070 SUPER; reproduces in `--bench` at 128 tokens.
- #6590 — sm_75 KV variants segfault / CUDA 700 at full context.
- #6591 — sm_75 decode falling with context growth.
Note on the second anomaly: in `--serve` the engine reads the trunk once (`trunk=3.67 GB (0.37s)`) and VRAM plateaus at ~17.5 GiB, while `--bench` reaches full speed with the same load line, so trunk placement alone does not explain the 8x difference. Not claiming a cause — just what the logs show.
Helper scripts used for this report can be provided on request.
関連リンク
インストール・モデル・リリースへのサイト内リンク。