Pull requests / #2889
#2889 q36: parallelise the expert copy at load
open · @ggeorgovassilis · 0 comentários · No GitHub
Server & APINVIDIA / CUDAModels & quants
Descrição
## What Parallelise the routed-expert copy in `place_experts()` (`src/qwen35/q36_engine.cu`). The load's hot loop copies the routed experts from the GGUF mapping into the host arena, one expert at a time, on a single thread: ```cpp for (int64_t e = 0; e < ne; ++e) expert_blob(L, e, L.hblobs + (size_t) e * L.stride); ``` This PR splits that loop into disjoint per-expert chunks across threads, with a new `STRATA_Q36_PARALLEL_PLACE` switch (`0` = serial, `N` = `N` threads, unset = one thread per logical core). Each expert's source and destination ranges are disjoint, so the placed bytes are identical at any thread count — byte-identical by construction. ## Why On an RTX 4060 Laptop (8 GB) / Ryzen 7 7735HS, loading `Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf` (22.85 GB) was dominated by this single-threaded copy (~300 MB/s, where the machine's memory bandwidth is ~36–44 GB/s). Measured: the copy drops **~64 s → ~16 s at 8 threads, ~13 s at 16 threads** (~4×), and the total load drops from ~104 s to ~58 s. Greedy-decode output is byte-identical (parallel vs serial). ## What it leaves alone - The dense-weight load (`load()`), the `cudaHostAlloc` pin of the arena, and the all-in-VRAM placement path are untouched. - The full-arena `std::memset` before the copy is unchanged (it is not part of this loop). ## Testing - **Build:** compiles cleanly with CUDA 12 and CUDA 13 (`strata` target). - **Tests:** `tools/test_setup_*.py` 38/40 pass (2 pre-existing network/config failures in `test_setup_engine_hash.py` / `test_setup_golden.py`, unrelated to this change); `serve/test_*.py` 784/785 pass (1 flaky `test_keepawake` timing test), 11 skipped. - **Byte-identity:** `--tokens "1,2,3" --n 24` prints identical `gen:` tokens with `STRATA_Q36_PARALLEL_PLACE` unset vs `=0`. - **Smoke:** the model loads and answers a trivial prompt (`"hi"`). - Note: this branch is based on `upstream/main`, whose q36 driver does not yet carry the `STRATA_Q36_SINGLE_STAGE` staging-buffer fix, so on an 8 GB card it needs a reduced context / `--expert-vram 0` to fit; the smoke test was run on the fork's build that includes that fix.
No site
Links install, modelos, releases.