#6633 [Engine] Inkling-2-Mini (.strata): DSpark speculative decoding never engages (drafting: off, drafts_offered: 0)

open · @ByteBrainMaster · 2 Kommentare · Auf GitHub

BenchmarksServer & APINVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Beschreibung

**Problem:** `strata-inkling` 0.1.65 never runs speculative decoding on the native `.strata` Inkling-2-Mini packs. `/metrics` reports `drafter: "dspark"` but `drafting: "off"`, and `drafts_offered` / `drafts_accepted` stay at 0 during long generations, although the pack is documented to have the DSpark drafter built in and the published speed figures are measured with drafting on.

**Environment:**
- Windows 11 10.0.26300; GPU NVIDIA RTX 3080 Ti Laptop, 16 GB, sm_86, driver 621.24
- CPU i9-12900H, 47.7 GB RAM
- Strata app 0.2.19, control 0.1.65, engine `components/shapes/inkling2/engine/0.1.65-windows-x86_64-cuda-cuda13`
- Packs: `Inkling-2-Mini-SQ-15.strata` (14.3 GB), `Inkling-2-Mini-SQ-20.strata` (19.3 GB), `Inkling-2-Mini-Vision.strata`
- Run config: `--max-context 32768 --vision ... --vram-experts-gb 11.41 --ram-experts-gb 4.27 --draft dspark`

**Reproduce:**
1. `strata launch --family inkling-2-mini --model sq-20 --no-tui --yes --context 32k --vision cpu`
2. Send a 400-token request (`POST /v1/chat/completions`).
3. While it generates, read `GET /metrics`: `engine.drafter`, `engine.drafting`, `totals.drafts_offered`, `totals.drafts_accepted`.

**Expected:** `drafting` on and `drafts_offered` > 0. The model card says *"Every pack has a DSpark draft model built in"* and its speed table is *"with Strata's built-in draft decoding on (W=4 Beam / DSpark)"*. `strata drafting plan` for this family also answers *"Inkling-2-Mini has its DSpark Block-8 drafter built into the model file; no separate drafter download is needed."*

**Actual:** `drafting: off`, `drafts_offered: 0`, `drafts_accepted: 0` throughout the generation (three samples while `live.state = generating`):

```
state=generating  drafter=dspark  drafting=off  offered=0  accepted=0
state=generating  drafter=dspark  drafting=off  offered=0  accepted=0
state=generating  drafter=dspark  drafting=off  offered=0  accepted=0
```

The OpenAI response carries no draft counters at all, where the Qwen engine on the same box returns them:

```json
{"cache_n":0,"prompt_n":25,"prompt_ms":863.2,"prompt_per_token_ms":34.528,
 "predicted_n":400,"predicted_ms":24199.2,"predicted_per_token_ms":60.498,
 "predicted_per_second":16.5}
```

(Qwen for comparison: `"draft_n":720,"draft_n_accepted":477`.)

`strata-inkling.exe --info` on the same pack reports the drafter as present:

```
arch=inkling2_loop format=strata build=SQ-20 layers=24 visits=48 hidden=2304 vocab=200058
experts=128 eos=200006 drafter=dspark images=1 image_pad=200054 parallel=64
```

and the engine start line carries the flag:

```
strata-inkling.exe --serve --native ...Inkling-2-Mini-SQ-20.strata --max-context 32768
  --vision ...Inkling-2-Mini-Vision.strata --vram-experts-gb 11.41 --ram-experts-gb 4.27 --draft dspark
```

**Evidence:** three attempts to supply a drafter explicitly, all with a verified file, none of which changed `drafting`:

1. `--dspark` → `dspark-Inkling-2-Mini-Q8_0.gguf` from `unsloth/Inkling-2-Mini-GGUF`, 383.6 MB, sha256 `2ec78510da837e8fc3be54a14757a0c5fbd2d8d9e85a0d92853fafdf674f3773` (matches the HF LFS oid). Flag visible in the start line; engine starts normally; `drafting: off`.
2. `--dspark` → `mtp.safetensors` from `thinkingmachines/Inkling-2-Mini`, 606.3 MB, sha256 `7c3b651317f01f73bcf4e1dceba66ef8b7e1a1eac6bc171e882e690a4b10a1d2` (matches). Same result.
3. `--draft auto` instead of `--draft dspark`. Same result.

A/B against drafting turned off entirely (`--drafting off`), same prompt, SQ-15:

| Config | decode |
|---|---|
| default (`--draft dspark`) | 33.4 / 33.3 / 33.2 tok/s |
| `--drafting off` | 34.4 / 33.9 / 33.7 tok/s |

Identical, which shows the drafting path contributes nothing today (and also means the A/B cannot demonstrate a gain, because the "on" arm is not on either).

There is no `--beam` or draft-width flag in `strata-inkling --help`, and none in the binary's flag strings, although the model card refers to **"W=4 Beam / DSpark"**.

**Reproduction status:** reproduced locally on this machine (SQ-15 and SQ-20, three parameter combinations, two verified drafter files). Not tested on Linux or other cards.

**Question:**
1. Is DSpark wired up for the `.strata` path in 0.1.65, or does it need something the CLI does not expose (the `W=4 Beam` width, a block that must be extracted from the pack first, `--parallel 1` only, a minimum context)?
2. If it is wired up, what condition moves `drafting` from `off` to on? None of `--draft dspark`, `--draft auto` or `--dspark <verified file>` changed it.
3. If it is not wired up for `.strata` yet, could `strata drafting` / `strata launch` say so, and could the docs' speed figures note that they include drafting? I can run more A/Bs or send the full engine log.

Mehr auf der Site

Links zu Install, Modellen, Releases.