#6590 [Engine] sm_75 inkling path: --kv k8v8/k8v4 segfault, CUDA 700 when prompt+gen fills --max-context, DSpark OOM at 131K

open · @qqkane · 2 comentarios · En GitHub

BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsLinux

Descripción

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Other

(Prebuilt component `strata-inkling` 0.1.65 from the v0.2.19 release tarball `strata-inkling-0.1.65-linux-x86_64-cuda-cuda13.tar.gz`, run directly from the CLI. No desktop app, no `strata` command, no source build.)

### What happened

Two runtime failures on sm_75 (RTX 2080 Ti 22 GB) with the `Inkling-2-Mini-SQ-15` pack. Loading and self-check are fine (`--selfcheck` PASS, `--info` reports the right shape, all 3072 experts land in VRAM), so these are runtime bugs rather than a build or placement problem. `--kv fp16` / `auto` work; the two problems below are the KV paths.

**A. `--kv k8v4` and `--kv k8v8` segfault on any input.** Even the smallest bench crashes. The engine switches to a different backend (`[inkling2-ep] Loaded … kv=k8v8`) right after expert placement and dies with a core dump. Reproduced at `--prompt-len 512`, `8192` and `32768`, for both dtypes. `--kv fp8` starts fine but dies at 32K (see B).

**B. `CUDA error 700` (illegal memory access) when the sequence exactly fills `--max-context`.** This is not an out-of-memory condition: VRAM peak was 20125 MiB of 21504 MiB, and the identical prompt succeeds when given headroom.

| command (`--bench`, fp16, gen 32) | result |
|---|---|
| `--prompt-len 24576` (auto ctx) | works — prefill 998.6 ppts, decode 32.7 tps, peak 19485 MiB |
| `--prompt-len 32768` (auto ctx) | **CUDA error 700**, peak 20125 MiB |
| `--prompt-len 32768 --max-context 32768` | **CUDA error 700**, peak 20125 MiB |
| `--prompt-len 32768 --max-context 65536` | **works** — prefill 907.4 ppts, decode 34.9 tps |

So the KV index runs past the reservation when prompt+gen reaches the window, instead of clamping the request or refusing it with a clear message. The same failure appears at 32K with `--kv fp8 --max-context 196608` (`CUDA error 700` at `ink_cuda_backend.cu:277`), so it is not tied to one dtype.

Also, for the record: `--draft auto` (DSpark) at `--max-context 131072` dies with `CUDA error 2 (out of memory)` while the same setup with `--draft off` starts fine, so the drafter appears to need roughly 5 GB beyond the non-draft footprint. Not a blocker, but worth a fit note — at 32K the drafter starts without complaint.

### Smallest steps or command that reproduce it

```bash
# release v0.2.19, prebuilt engine, no source build
cd ~/inkling-test && export LD_LIBRARY_PATH=$PWD CUDA_VISIBLE_DEVICES=0

# health check: both pass
./strata-inkling --info Inkling-2-Mini-SQ-15.strata
./strata-inkling --selfcheck --native Inkling-2-Mini-SQ-15.strata --gpus 1

# A: core dump
./strata-inkling --bench --native Inkling-2-Mini-SQ-15.strata --gpus 1 \
    --parallel 1 --prompt-len 512 --gen 32 --kv k8v8

# B: fails, then passes with one extra flag
./strata-inkling --bench --native Inkling-2-Mini-SQ-15.strata --gpus 1 \
    --parallel 1 --prompt-len 32768 --gen 32 --kv fp16 --max-context 32768
./strata-inkling --bench --native Inkling-2-Mini-SQ-15.strata --gpus 1 \
    --parallel 1 --prompt-len 32768 --gen 32 --kv fp16 --max-context 65536
```

### Reproduction status

Reproduced locally

### Strata version

Release v0.2.19, prebuilt component `strata-inkling` 0.1.65 (build `c6f6427d9de9+cuda13.0`, `archs 75;80;86;89;120`, `min_driver 580.65`), CUDA 13.0, no source build. The GGUF/q36 engine is not involved — this is the `inkling2_loop` path.

### Graphics card, memory and system

RTX 2080 Ti 22 GB (sm_75, 68 SMs, 21504 MiB) · driver 615.71.09 · CUDA 13.0 · Manjaro Linux, kernel 6.18.55 · one GPU only (`--gpus 1`, the second card in the machine was not used).

### Model, size and settings

`Inkling-2-Mini-SQ-15.strata`, 14348316672 B, sha256 `2ea32eed533afba257e2cbee7b073dc2c3da80fe55d1d2d794779b5498e30737` (matches the published `SHA256SUMS.txt`) · `arch=inkling2_loop layers=24 visits=48 hidden=2304 vocab=200058 experts=128` · trunk 2.07 GB, 3072 experts in VRAM, 0 host, `hyb=0` · `--parallel 1`, `--draft off` unless stated.

### Logs

```
$ ./strata-inkling --bench --native ... --prompt-len 512 --gen 32 --kv k8v8
[inkling2-ep] Loaded strata (SQ-15) | gpus=1 (p2p=pinned, ep_split=0.50) | experts: 3072 VRAM (GPU0=11.44 GiB, GPU1=0.00 GiB), 0 host (doorbell=off, numa=1, thr=0) | kv=k8v8
Segmentation fault (core dumped)
```

```
$ ./strata-inkling --bench --native ... --prompt-len 32768 --gen 32 --kv fp16 --max-context 32768
[InkCUDA] Loaded Inkling-2-Mini-SQ-15.strata (SQ-15): trunk=2.07 GB (1.86s), experts=3072 VRAM / 0 host (10.04s, hyb=0, promo=0, pf_buf=0MB), sm120_tc=0, screen=1
strata-inkling CUDA error 700 (an illegal memory access was encountered) at /data/split-pkg/rc219-r2/src/engine/src/inkling2/cuda/ink_cuda_backend.cu:277
```

```
$ ./strata-inkling --bench --native ... --prompt-len 32768 --gen 32 --kv fp16 --max-context 65536
[bench] backend=cuda format=strata parallel=1 gpus=1 kv=fp16 prompt=32768 gen=32 | prefill=907.4 ppts (36.11s) | decode=34.9 tps agg (34.9 tps/slot, 0.92s)
```

```
$ ./strata-inkling --serve --native ... --max-context 131072 --parallel 1 --draft auto
[InkCUDA] Loaded Inkling-2-Mini-SQ-15.strata (SQ-15): trunk=2.07 GB (1.72s), experts=3072 VRAM / 0 host (9.71s, hyb=0, promo=0, pf_buf=0MB), sm120_tc=0, screen=1
strata-inkling CUDA error 2 (out of memory) at /data/split-pkg/rc219-r2/src/engine/src/inkling2/cuda/ink_cuda.cuh:54
```

Related: #5876 (KV reserved for the full `--max-context` before expert placement, q36 engine) and #2197 (sm_75 kernel series).
Two notes that may help triage: this build has no env-level Turing tuning (`strings strata-inkling` shows only `STRATA_NUMA_POOL` — no `STRATA_SM75_PERF`, which #2197 proposes as a default), and `--kv k8v4/k8v8` silently changing backend to `inkling2-ep` on a single GPU looks like the reason the crash is a segfault rather than a clean error.

Decode-speed numbers with `--kv fp16` (97 tps on an empty context down to 33 tps at 24K, with prefill flat at ~1 ppts) are reported separately.

En el sitio

Enlaces a install, modelos, releases.