#5913 [Engine] qwen35moe 19B-A3B: htod copy overruns a 57,802,752-byte buffer on a specific 19,431-token prompt
closed · @trutto · 3 comentarios · En GitHub
BenchmarksServer & APINVIDIA / CUDAModels & quantsLinux
Descripción
### Area [Engine] (speed, kernels, models, the server and API) ### Desktop app, Terminal or source? Terminal (the `strata` command) ### What happened I ran `strata serve` on a qwen35moe 19B-A3B GGUF and sent a single 19,431-token prompt. The request ends with `finish=error`; the log shows a host→device copy that writes 106,496 bytes past a 57,802,752-byte buffer. Non-stream client gets HTTP 500, stream client sees the connection close after ~11 s. The server stays up and other requests work; the failure is deterministic on retry. The failure is content/state dependent, not purely length dependent: the same engine/flags with **the same 19,431 prompt-token count** built from Strata's own repo files (markdown + Rust) succeeds, and a pure-English filler prompt at 19,430/19,440/19,450 tokens succeeds. Workarounds, each of which avoids it with the same prompt: `--chunk 2048` · `--mtp 0` · `--ctx 32768` (any one alone is enough; default chunk 4096 + MTP 2 + ctx 65536 fails). ### Smallest steps or command that reproduce it 1. ``` strata serve --model Groot-19B-A3B-TTQ-3.56bpw-IQ3_XXS.gguf --gpu \ --ctx 65536 --kv q8 --mtp 2 --addr 127.0.0.1:9144 --api-key <key> ``` 2. POST one user message to `/v1/chat/completions` (no tools, temperature 0). The prompt is 61,094 characters of mixed technical markdown + shell/Python source, with a one-sentence needle inserted at 90 % depth and a short question appended. Server logs `prompt=19431 tok`. 3. The request fails; the log contains the `htod` line below and `finish=error`. Retried 3/3 times, stream and non-stream, and with `--chunk 2048/4096/8192` (all fail at the default chunk setting). ### Reproduction status Reproduced locally ### Strata version Strata 1.2.1 (`strata --version`) ### Graphics card, memory and system NVIDIA RTX 5050 Laptop GPU, 8 GB VRAM (8151 MiB), driver 615.78.08 · 30 GB RAM · Arch-based Linux (Omarchy) ### Model, size and settings `Groot-19B-A3B-TTQ-3.56bpw-IQ3_XXS.gguf` — qwen35moe, 19B total / 3B active, 40 layers, 128 experts/top-8, hybrid Delta/Attn, MTP nextn head. sha256 `4816aa9ad85f98016b73b956b6ce2162a2c38c412b5ed35fb8ee484a7a37cfe6`. Context 65536, `--kv q8`, MTP 2, experts streamed from host (defaults otherwise). ### Logs ``` request 1: htod: 450560 bytes at 57458688 into 57802752 request: POST /v1/chat/completions key=admin model=Groot-... prompt=19431 tok stream=false tools=0 done: key=admin model=Groot-... prompt=19431 tok (0 cached) -> 0 tok in 11.16 s, decode 0.0 tok/s, finish=error ``` ### Additional evidence Same prompt, one flag changed: | variant | result | |---|---| | ctx 65536, kv q8, **MTP 2**, chunk 4096 (default) | **fails** | | ctx 65536, kv **f16**, MTP 2, chunk 4096 | **fails** | | ctx 65536, kv q8, **--chunk 2048** | OK (prefill 1498 tok/s, decode 51.9 tok/s) | | ctx 65536, kv q8, **--mtp 0** | OK (prefill 1622, decode 58.7) | | **--ctx 32768**, kv q8, MTP 2 | OK (prefill 1558, decode 60.9) | Buffer arithmetic for the two failing variants — both writes end at the **same address** into a destination that is 106,496 bytes too small: | kv | source offset | copy size | write end | destination | overflow | |---|---:|---:|---:|---:|---:| | q8 | 57,458,688 | 450,560 | 57,909,248 | 57,802,752 | 106,496 | | f16 | 57,008,128 | 901,120 | 57,909,248 | 57,802,752 | 106,496 | Source start and copy length scale with the KV dtype (44 tokens' worth), which looks like a token-count-dependent staging buffer sized slightly short rather than a bad offset. I can provide the exact failing prompt (61 KB) or the request JSON privately if it helps; it contains local config text, so I did not attach it. I am also happy to run diagnostic variants on this box. ### Related Same model shape as #5551 (qwen35moe 19B-A3B bring-up), but a different failure mode: this is a host→device buffer overflow during prefill, reproducible with a public TTQ GGUF, not NaN/MMVQ.
En el sitio
Enlaces a install, modelos, releases.