问题 / #474
#474 strata serve: mtp: the draft head does not fit — no guidance when MTP draft head exceeds VRAM on smaller GPUs
closed · @saikiran121111 · 2 评论 · 去 GitHub 看
Setup & installServer & APINVIDIA / CUDAModels & quantsWindowsLinux
说明
What happened The engine crashes on startup with this error when the MTP draft layer's token-subset head can't fit in VRAM: strata serve: mtp: the draft head does not fit RuntimeError: the engine exited before it was ready This is a silent failure — the user gets no hint about what to change. On a 12 GB RTX 3060 with IQ3_S at 262K context, only ~1,500 MiB is free when MTP tries to bind; the default CJK draft vocab (106K tokens) needs ~348 MiB of contiguous VRAM and fails. The fix already exists but isn't surfaced setup.py and setup.sh already accept --draft-vocab with three choices: ┌───────────────┬─────────┬─────────────────┐ │ choice │ tokens │ draft head VRAM │ ├───────────────┼─────────┼─────────────────┤ │cjk (default) │ 106,299 │ ~348 MiB │ ├───────────────┼─────────┼─────────────────┤ │ cyrillic │ 57,793 │ ~189 MiB │ ├───────────────┼─────────┼─────────────────┤ │ en │ 40,525 │ ~133 MiB │ └───────────────┴─────────┴─────────────────┘ Switching to --draft-vocab en resolved it immediately. The issue is that setup picks the default without considering VRAM headroom, and the engine's error message doesn't mention the flag. Two small improvements that would help 1. Engine: when MTP's bind() fails with "the draft head does not fit", print a hint: "Try a smaller draft vocabulary: --draft-vocab en (English/code, ~110 MiB less VRAM) or --draft-vocab cyrillic. Change it in setup with --draft-vocab." 2. Setup / engine startup: when --vram-reserve-mib leaves less headroom than the draft head needs, pick en (or cyrillic) automatically and note it, rather than silently using the largest vocab regardless of GPU size. 3. setup.sh / START-HERE.bat: mention --draft-vocab as a tuning option for GPUs under 14 GB VRAM. Environment - Strata engine v0.1.34, model: Qwen3.8-Flash-Next IQ3_S, context 262144 - GPU: NVIDIA GeForce RTX 3060 12 GB - OS: Linux
本站相关内容
相关页面的快捷入口。