Issues / #420

#420 0.1.32 aborts with mmq.cuh:1555 fatal error (J_best=0) on a 77-token prompt, SM 12.0, 2 GPUs, Swift IQ3_XXS

closed · @qni-live · 3 comentarios · En GitHub

Setup & installMulti-GPUNVIDIA / CUDAModels & quants

Descripción

Engine 0.1.32 aborts on the first longer request. Engine 0.1.30 with the same settings, model and pack runs fine.

Setup
- 2x RTX PRO 4500 Blackwell (SM 12.0, 32 GB each), driver 595.84, CUDA 13.3 toolkit, Ubuntu 26.04
- Engine built from the v0.1.32 source tarball with `setup.py --build` (arch 120, source built on this PC, not the ready-made engine)
- Model: Swift 1.5 IQ3_XXS pack, `--layer-split 25`, `--kv k8v4`, `--spec 5 --spec-min-p 0.5`, MTP, `--prefill auto`, vision on
- Same config file as the 0.1.30 run that works (only `exe` points at the 0.1.32 build)

What happens
- Server starts and prints `ready`.
- First request: a chat completion with a 77-token prompt (`max_tokens 256`, temperature 0).
- The server prints `reading the prompt: 77 tokens, 10 s so far`, then the engine aborts:

```
J_best=0
.../third_party/llama.cpp/ggml/src/ggml-cuda/template-instances/../mmq.cuh:1555: fatal error
```

- Happened in 2 of 2 runs with that request.
- Three much shorter requests ("Say hi. 1/2/3", about 20 tokens, `max_tokens 20`) in the same setup did answer, also with `--prefill 512` and with `--spec 4`. I did not find the exact prompt length where it starts, and I did not test other models or one GPU.

Not tested: whether it is my local build or the release code; whether one GPU or another quantization shows it.

Logs available: server log with the abort, the config file. I can try a given length or flag if you tell me what is useful.

En el sitio

Enlaces a install, modelos, releases.