Issues / #474

#474 strata serve: mtp: the draft head does not fit — no guidance when MTP draft head exceeds VRAM on smaller GPUs

closed · @saikiran121111 · 2 commentaires · Sur GitHub

Setup & installServer & APINVIDIA / CUDAModels & quantsWindowsLinux

Description

What happened

The engine crashes on startup with this error when the MTP draft layer's token-subset head can't fit in VRAM:

strata serve: mtp: the draft head does not fit
RuntimeError: the engine exited before it was ready

This is a silent failure — the user gets no hint about what to change. On a 12 GB RTX 3060 with IQ3_S at 262K context, only ~1,500 MiB is free when MTP tries to bind; the default CJK draft vocab (106K tokens) needs ~348 MiB of contiguous VRAM and fails.

The fix already exists but isn't surfaced

setup.py and setup.sh already accept --draft-vocab with three choices:

┌───────────────┬─────────┬─────────────────┐
 │    choice     │ tokens  │ draft head VRAM │
├───────────────┼─────────┼─────────────────┤
│cjk (default) │ 106,299 │ ~348 MiB        │
├───────────────┼─────────┼─────────────────┤
│     cyrillic      │ 57,793  │      ~189 MiB        │
├───────────────┼─────────┼─────────────────┤
│       en            │ 40,525  │       ~133 MiB        │
└───────────────┴─────────┴─────────────────┘

Switching to --draft-vocab en resolved it immediately. The issue is that setup picks the default without considering VRAM headroom, and the engine's error message doesn't mention the flag.

Two small improvements that would help

1. Engine: when MTP's bind() fails with "the draft head does not fit", print a hint: "Try a smaller draft vocabulary: --draft-vocab en (English/code, ~110 MiB less VRAM) or --draft-vocab cyrillic. Change it in setup with --draft-vocab."
2. Setup / engine startup: when --vram-reserve-mib leaves less headroom than the draft head needs, pick en (or cyrillic) automatically and note it, rather than silently using the largest vocab regardless of GPU size.
3. setup.sh / START-HERE.bat: mention --draft-vocab as a tuning option for GPUs under 14 GB VRAM.

Environment

- Strata engine v0.1.34, model: Qwen3.8-Flash-Next IQ3_S, context 262144
- GPU: NVIDIA GeForce RTX 3060 12 GB
- OS: Linux

Sur le site

Liens install, modèles, releases.