#5913 [Engine] qwen35moe 19B-A3B: htod copy overruns a 57,802,752-byte buffer on a specific 19,431-token prompt

closed · @trutto · 3 comentários · No GitHub

BenchmarksServer & APINVIDIA / CUDAModels & quantsLinux

Descrição

### Area
[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?
Terminal (the `strata` command)

### What happened
I ran `strata serve` on a qwen35moe 19B-A3B GGUF and sent a single 19,431-token prompt. The request
ends with `finish=error`; the log shows a host→device copy that writes 106,496 bytes past a 57,802,752-byte
buffer. Non-stream client gets HTTP 500, stream client sees the connection close after ~11 s. The server
stays up and other requests work; the failure is deterministic on retry.

The failure is content/state dependent, not purely length dependent: the same engine/flags with
**the same 19,431 prompt-token count** built from Strata's own repo files (markdown + Rust) succeeds,
and a pure-English filler prompt at 19,430/19,440/19,450 tokens succeeds.

Workarounds, each of which avoids it with the same prompt:
`--chunk 2048` · `--mtp 0` · `--ctx 32768` (any one alone is enough; default chunk 4096 + MTP 2 +
ctx 65536 fails).

### Smallest steps or command that reproduce it
1. ```
strata serve --model Groot-19B-A3B-TTQ-3.56bpw-IQ3_XXS.gguf --gpu \
  --ctx 65536 --kv q8 --mtp 2 --addr 127.0.0.1:9144 --api-key <key>
```
2. POST one user message to `/v1/chat/completions` (no tools, temperature 0). The prompt is
   61,094 characters of mixed technical markdown + shell/Python source, with a one-sentence needle
   inserted at 90 % depth and a short question appended. Server logs `prompt=19431 tok`.
3. The request fails; the log contains the `htod` line below and `finish=error`.

Retried 3/3 times, stream and non-stream, and with `--chunk 2048/4096/8192` (all fail at the default
chunk setting).

### Reproduction status
Reproduced locally

### Strata version
Strata 1.2.1 (`strata --version`)

### Graphics card, memory and system
NVIDIA RTX 5050 Laptop GPU, 8 GB VRAM (8151 MiB), driver 615.78.08 · 30 GB RAM · Arch-based Linux (Omarchy)

### Model, size and settings
`Groot-19B-A3B-TTQ-3.56bpw-IQ3_XXS.gguf` — qwen35moe, 19B total / 3B active, 40 layers,
128 experts/top-8, hybrid Delta/Attn, MTP nextn head.
sha256 `4816aa9ad85f98016b73b956b6ce2162a2c38c412b5ed35fb8ee484a7a37cfe6`.
Context 65536, `--kv q8`, MTP 2, experts streamed from host (defaults otherwise).

### Logs
```
request 1: htod: 450560 bytes at 57458688 into 57802752
request: POST /v1/chat/completions key=admin model=Groot-... prompt=19431 tok stream=false tools=0
done: key=admin model=Groot-... prompt=19431 tok (0 cached) -> 0 tok in 11.16 s, decode 0.0 tok/s, finish=error
```

### Additional evidence

Same prompt, one flag changed:

| variant | result |
|---|---|
| ctx 65536, kv q8, **MTP 2**, chunk 4096 (default) | **fails** |
| ctx 65536, kv **f16**, MTP 2, chunk 4096 | **fails** |
| ctx 65536, kv q8, **--chunk 2048** | OK (prefill 1498 tok/s, decode 51.9 tok/s) |
| ctx 65536, kv q8, **--mtp 0** | OK (prefill 1622, decode 58.7) |
| **--ctx 32768**, kv q8, MTP 2 | OK (prefill 1558, decode 60.9) |

Buffer arithmetic for the two failing variants — both writes end at the **same address** into a
destination that is 106,496 bytes too small:

| kv | source offset | copy size | write end | destination | overflow |
|---|---:|---:|---:|---:|---:|
| q8  | 57,458,688 | 450,560 | 57,909,248 | 57,802,752 | 106,496 |
| f16 | 57,008,128 | 901,120 | 57,909,248 | 57,802,752 | 106,496 |

Source start and copy length scale with the KV dtype (44 tokens' worth), which looks like a
token-count-dependent staging buffer sized slightly short rather than a bad offset.

I can provide the exact failing prompt (61 KB) or the request JSON privately if it helps; it contains
local config text, so I did not attach it. I am also happy to run diagnostic variants on this box.

### Related
Same model shape as #5551 (qwen35moe 19B-A3B bring-up), but a different failure mode: this is a
host→device buffer overflow during prefill, reproducible with a public TTQ GGUF, not NaN/MMVQ.

No site

Links install, modelos, releases.