Pull requests / #1790
#1790 sycl mtp: slot drafters copy the shared draft head's type (#1121 on the SYCL port, as #1110 did for CUDA)
closed · @leadgtr7 · 0 comentarios · En GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsWindows
Descripción
## Title Issue: Resolves #1121 for the SYCL port (the CUDA source was fixed in #1110) ## Summary `--batch-mtp` still ends the Intel (SYCL) engine on the first batch admission with `strata batch: MTP admission for slot 0 failed: mtp: unsupported native MMVQ GGML type`. The cause is the one #1110 fixed in `src/core/mtp.cpp`: a slot drafter that shares the main drafter's draft-vocab head copies `dhead_`, `dvocab_` and `n_dvocab_` but not `dhead_type_`, which stays `-1`. `sycl/src/core/mtp.cpp` never got that line. This adds it. Nothing else changes; without `--batch-mtp` no drafter shares a head, so that path is untouched. ## What changed - `sycl/src/core/mtp.cpp`, `MtpDrafter::bind`: `dhead_type_ = shared->dhead_type_;` next to the other three copies (same line and comment as the CUDA source). ## How it was checked Arc Pro B65 32 GB (xe, PCIe Gen4 x16), engine 0.1.41 built with `sycl/tools/build.sh` (AOT `bmg-g31`), Swift 1.5 IQ3_S GSQ-RCO (all 512 experts; ~12.5k in VRAM, the rest in the pinned host mirror), `--mtp rt --spec 4`, `rt/draft_vocab.bin` present, `"parallel"` 2 and 4. Server with #1721's `_pump` fix (needed for any `"parallel"` on SYCL). | | result | |---|---| | without this line, `--batch-mtp`, parallel 2 | engine exits at the first admission (the error above), both clients get nothing | | without this line + `STRATA_MTP_FULL_HEAD=1` (the workaround in #1121) | works; single-slot decode 34.4 tok/s vs 38.4 without the variable (the full head costs ~10%) | | **with this line**, `--batch-mtp`, parallel 2, no env | works: 199 batch windows, avg 2.60 rows, both replies coherent | | **with this line**, parallel 4 | works: 175 windows, avg 6.83 rows | Throughput on this card is a separate story and not this PR's claim (4 concurrent 300-token requests: 29.5 tok/s queued on one slot, 31.9 with 2 slots + `--batch-mtp`, 32.4 with 4; the batch windows wait on mirrored experts over PCIe). The time to first token drops from up to 29 s to under 6 s with 4 slots. Not tested: CUDA/HIP (untouched), greedy token equality of batched vs solo (on SYCL that already differs without `--batch-mtp`; noted on #1721). ## Extra Notes Only the SYCL port is touched. Assisted-by: Claude Code.
En el sitio
Enlaces a install, modelos, releases.