Pull requests / #2889

#2889 q36: parallelise the expert copy at load

open · @ggeorgovassilis · 0 comments · View on GitHub

Server & APINVIDIA / CUDAModels & quants

Description

## What

Parallelise the routed-expert copy in `place_experts()` (`src/qwen35/q36_engine.cu`).

The load's hot loop copies the routed experts from the GGUF mapping into the host arena, one
expert at a time, on a single thread:

```cpp
for (int64_t e = 0; e < ne; ++e) expert_blob(L, e, L.hblobs + (size_t) e * L.stride);
```

This PR splits that loop into disjoint per-expert chunks across threads, with a new
`STRATA_Q36_PARALLEL_PLACE` switch (`0` = serial, `N` = `N` threads, unset = one thread per
logical core). Each expert's source and destination ranges are disjoint, so the placed bytes
are identical at any thread count — byte-identical by construction.

## Why

On an RTX 4060 Laptop (8 GB) / Ryzen 7 7735HS, loading `Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf`
(22.85 GB) was dominated by this single-threaded copy (~300 MB/s, where the machine's memory
bandwidth is ~36–44 GB/s). Measured: the copy drops **~64 s → ~16 s at 8 threads, ~13 s at 16
threads** (~4×), and the total load drops from ~104 s to ~58 s. Greedy-decode output is
byte-identical (parallel vs serial).

## What it leaves alone

- The dense-weight load (`load()`), the `cudaHostAlloc` pin of the arena, and the
  all-in-VRAM placement path are untouched.
- The full-arena `std::memset` before the copy is unchanged (it is not part of this loop).

## Testing

- **Build:** compiles cleanly with CUDA 12 and CUDA 13 (`strata` target).
- **Tests:** `tools/test_setup_*.py` 38/40 pass (2 pre-existing network/config failures in
  `test_setup_engine_hash.py` / `test_setup_golden.py`, unrelated to this change);
  `serve/test_*.py` 784/785 pass (1 flaky `test_keepawake` timing test), 11 skipped.
- **Byte-identity:** `--tokens "1,2,3" --n 24` prints identical `gen:` tokens with
  `STRATA_Q36_PARALLEL_PLACE` unset vs `=0`.
- **Smoke:** the model loads and answers a trivial prompt (`"hi"`).
- Note: this branch is based on `upstream/main`, whose q36 driver does not yet carry the
  `STRATA_Q36_SINGLE_STAGE` staging-buffer fix, so on an 8 GB card it needs a reduced context /
  `--expert-vram 0` to fit; the smoke test was run on the fork's build that includes that fix.

Related on this site

Quick links to related pages.