#5841 [Engine] --mtp-q4 all overflows the draft head when the output head is IQ4_XS (GB10 / Qwen3.8-Flash-Next IQ2_XS): "mtp: writing the draft head", engine exits 1

closed · @tutumomo · 1 コメント · GitHub で見る

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quants

本文

## Problem

With `--mtp-q4 all` on a model whose output head is IQ4_XS, the engine exits at load with

```
strata mtp: draft head over 106299 tokens (137.9 MiB)
strata serve: mtp: writing the draft head
```

(exit 1, nothing serves). `--mtp-q4 proj` and `--mtp-q4 head` avoid it; drafting off avoids it.

## Cause

`MtpDrafter::make_q4_head()` (`src/core/mtp.cpp`) rewrites the gathered draft-head subset in place as Q4_0 rows:

```cpp
const size_t in_row = head_->row_bytes(), out_row = (size_t) (N / 32) * 18;
...
cudaMemcpy(dhead_, dst.data(), dst.size(), cudaMemcpyHostToDevice)
```

but `bind()` allocated `dhead_` for the head's own row size:

```cpp
const int64_t row_bytes = (int64_t) head->row_bytes();
cudaMalloc((void**) &dhead_, (size_t) (n_dvocab_ * row_bytes))
```

A Q4_0 row is `n_embd / 32 * 18` bytes. IQ4_XS is `n_embd / 256 * 136` bytes, which is **smaller** at `n_embd = 2560`:

| format | bytes per row (n_embd 2560) | bytes for 106299 rows |
| --- | --- | --- |
| IQ4_XS | 1360 | 137.9 MiB |
| Q4_0 | 1440 | 146.0 MiB |

So the H2D copy writes 8.1 MiB past the end of `dhead_` and fails; the comment "the Q4_0 rows replace the subset in place (they are smaller)" only holds for formats at least as wide as Q4_0 (Q6_K, Q8_0, ...). IQ4_XS also passes `native_mmvq_supported`, so `--mtp-q4 all` is selected on this model by the GB10 defaults.

## Repro

DGX Spark (GB10, sm_121, aarch64), Ubuntu 26.04, CUDA 13.0.88, engine 0.1.59.1 built from `spark/build_engine.sh`:

```
strata launch --family swift --model IQ2_XS \
  --model-path <Qwen3.8-Flash-Next IQ2_XS GGUF folder> \
  --context 128k --drafting mtp --vision on --access tailscale --port 8081
```

The launch passes `--mtp <dir> --mtp-q4 all --mtp-hnorm stream` (the GB10 defaults). The engine dies every start; drafting off loads.

## Fix (tested here)

Size the buffer for the larger of the two rows and account for it in `vram_`:

```cpp
const int64_t q4_row_bytes = (q4_head_ && g_->n_embd % 32 == 0) ? (g_->n_embd / 32) * 18 : 0;
const int64_t head_cap = std::max(row_bytes, q4_row_bytes);
cudaMalloc((void**) &dhead_, (size_t) (n_dvocab_ * head_cap))
...
vram_ += (uint64_t) (n_dvocab_ * head_cap) + raw.size();
```

`gather_rows` still fills `n_dvocab_ * row_bytes` at the front, and `make_q4_head` overwrites from the front with the wider Q4_0 rows, so nothing else changes.

**Measured on:** DGX Spark GB10 (sm_121, 121 GiB unified), Qwen3.8-Flash-Next-GSQ-RCO-abliterated IQ2_XS (SC117), Strata 0.2.13 / engine 0.1.59.1, 128K context, `--vision on`:

- before: engine exit 1 at "mtp: writing the draft head", 4/4 starts
- after: `strata mtp: --mtp-q4: draft head over 106299 tokens as Q4_0 (146.0 MiB read per step, was 137.9)`, loads in ~4 min, serves
- decode, same prompt, 256 new tokens: drafting off 37.3 tok/s (3 runs: 37.3 / 37.4 / 37.3); MTP on 41.5 / 53.6 / 53.7 tok/s, drafts accepted 120/196, 112/168, 131/224 (61%, 67%, 58%)

## Note (not a Strata bug, but blocks building on this OS)

Ubuntu 26.04 ships glibc 2.43, whose `rsqrt`/`rsqrtf` are declared `noexcept`; CUDA 13.0's `crt/math_functions.h` declares them without it, so `nvcc` fails with `error: exception specification is incompatible with that of previous function "rsqrt"`. `-Xcompiler -U_GNU_SOURCE` "fixes" that but then breaks libstdc++ headers (`fwide`, `uselocale`, `pthread_mutex_timedlock` undefined). I built against a symlink copy of the toolkit with `noexcept(true)` added to those two declarations in the copy only; the system toolkit was left untouched.

関連リンク

インストール・モデル・リリースへのサイト内リンク。