#5841 [Engine] --mtp-q4 all overflows the draft head when the output head is IQ4_XS (GB10 / Qwen3.8-Flash-Next IQ2_XS): "mtp: writing the draft head", engine exits 1
closed · @tutumomo · 1 commentaires · Sur GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quants
Description
## Problem With `--mtp-q4 all` on a model whose output head is IQ4_XS, the engine exits at load with ``` strata mtp: draft head over 106299 tokens (137.9 MiB) strata serve: mtp: writing the draft head ``` (exit 1, nothing serves). `--mtp-q4 proj` and `--mtp-q4 head` avoid it; drafting off avoids it. ## Cause `MtpDrafter::make_q4_head()` (`src/core/mtp.cpp`) rewrites the gathered draft-head subset in place as Q4_0 rows: ```cpp const size_t in_row = head_->row_bytes(), out_row = (size_t) (N / 32) * 18; ... cudaMemcpy(dhead_, dst.data(), dst.size(), cudaMemcpyHostToDevice) ``` but `bind()` allocated `dhead_` for the head's own row size: ```cpp const int64_t row_bytes = (int64_t) head->row_bytes(); cudaMalloc((void**) &dhead_, (size_t) (n_dvocab_ * row_bytes)) ``` A Q4_0 row is `n_embd / 32 * 18` bytes. IQ4_XS is `n_embd / 256 * 136` bytes, which is **smaller** at `n_embd = 2560`: | format | bytes per row (n_embd 2560) | bytes for 106299 rows | | --- | --- | --- | | IQ4_XS | 1360 | 137.9 MiB | | Q4_0 | 1440 | 146.0 MiB | So the H2D copy writes 8.1 MiB past the end of `dhead_` and fails; the comment "the Q4_0 rows replace the subset in place (they are smaller)" only holds for formats at least as wide as Q4_0 (Q6_K, Q8_0, ...). IQ4_XS also passes `native_mmvq_supported`, so `--mtp-q4 all` is selected on this model by the GB10 defaults. ## Repro DGX Spark (GB10, sm_121, aarch64), Ubuntu 26.04, CUDA 13.0.88, engine 0.1.59.1 built from `spark/build_engine.sh`: ``` strata launch --family swift --model IQ2_XS \ --model-path <Qwen3.8-Flash-Next IQ2_XS GGUF folder> \ --context 128k --drafting mtp --vision on --access tailscale --port 8081 ``` The launch passes `--mtp <dir> --mtp-q4 all --mtp-hnorm stream` (the GB10 defaults). The engine dies every start; drafting off loads. ## Fix (tested here) Size the buffer for the larger of the two rows and account for it in `vram_`: ```cpp const int64_t q4_row_bytes = (q4_head_ && g_->n_embd % 32 == 0) ? (g_->n_embd / 32) * 18 : 0; const int64_t head_cap = std::max(row_bytes, q4_row_bytes); cudaMalloc((void**) &dhead_, (size_t) (n_dvocab_ * head_cap)) ... vram_ += (uint64_t) (n_dvocab_ * head_cap) + raw.size(); ``` `gather_rows` still fills `n_dvocab_ * row_bytes` at the front, and `make_q4_head` overwrites from the front with the wider Q4_0 rows, so nothing else changes. **Measured on:** DGX Spark GB10 (sm_121, 121 GiB unified), Qwen3.8-Flash-Next-GSQ-RCO-abliterated IQ2_XS (SC117), Strata 0.2.13 / engine 0.1.59.1, 128K context, `--vision on`: - before: engine exit 1 at "mtp: writing the draft head", 4/4 starts - after: `strata mtp: --mtp-q4: draft head over 106299 tokens as Q4_0 (146.0 MiB read per step, was 137.9)`, loads in ~4 min, serves - decode, same prompt, 256 new tokens: drafting off 37.3 tok/s (3 runs: 37.3 / 37.4 / 37.3); MTP on 41.5 / 53.6 / 53.7 tok/s, drafts accepted 120/196, 112/168, 131/224 (61%, 67%, 58%) ## Note (not a Strata bug, but blocks building on this OS) Ubuntu 26.04 ships glibc 2.43, whose `rsqrt`/`rsqrtf` are declared `noexcept`; CUDA 13.0's `crt/math_functions.h` declares them without it, so `nvcc` fails with `error: exception specification is incompatible with that of previous function "rsqrt"`. `-Xcompiler -U_GNU_SOURCE` "fixes" that but then breaks libstdc++ headers (`fwide`, `uselocale`, `pthread_mutex_timedlock` undefined). I built against a symlink copy of the toolkit with `noexcept(true)` added to those two declarations in the copy only; the system toolkit was left untouched.
Sur le site
Liens install, modèles, releases.