#6399 [Engine] Prefetch routed expert weight rows into L1 on GB10
open · @InB4DevOps · 0 comentarios · En GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows
Descripción
## Problem and result The grouped expert kernels normally read a batch of weight rows when they start computing its dot products. Requesting those rows earlier improved end-to-end primary decode on one DGX Spark by **7.20% and 7.54%** in two independent 12-pair batches. Each batch's lower 95% confidence bound exceeded +3%. | Prompt | Batch 1: median stock → PR, tok/s | Paired gain [95% CI] | Batch 2: median stock → PR, tok/s | Paired gain [95% CI] | |---|---:|---:|---:|---:| | Code review, 2,028 tokens (primary) | 66.21 → 70.96 | +7.20% [6.87, 7.53] | 66.15 → 71.22 | +7.54% [7.21, 7.88] | | Short, 111 tokens | 61.39 → 65.92 | +7.41% [7.19, 7.64] | 61.42 → 65.82 | +4.47% [−2.19, 11.58] | | Long, 8,168 tokens | 58.33 → 62.03 | +6.42% [5.95, 6.89] | 58.25 → 62.05 | +6.71% [6.20, 7.23] | ## Related discussion Scope: #6398 (maintainer agreement pending). #4811 proposes copying gate/up tiles into shared memory. This PR uses cache hints for both projections, with no shared weight tile. Both were tested against stock separately; no additive benefit is claimed. #6375 changes hyper-connections and is not in this branch. ## Approach For `__CUDA_ARCH__ == 1210`, at the start of the fused gate/up and grouped down kernels, threads issue `prefetch.global.L1` hints at 128-byte intervals across the first 16 weight rows. While computing the first group chunk they hint the next batch of rows. The down kernel checks that the full batch exists. The packed weights, shared-memory allocation, arithmetic and kernel launch geometry stay the same. No flag is required. ## Validation **Passed:** Ubuntu 24.04.5 LTS/aarch64, NVIDIA GB10 (128 GB unified memory; no separate VRAM), driver 580.178.04, CUDA 13.0.88. Base `3142b66b96e9c0e775e30f620b11b191a1427c2a`, Strata 0.2.17 / engine 0.1.63. Stock was built with `sh engine/spark/build_engine.sh`; the candidate recompiled `iq_kernels.cu` using the stock Ninja compile command and linked against the same remaining objects. The submitted CUDA file at `0db3f0755` is byte-identical to the measured source. - `screen.py iqaheadL1x128 iqaheadL1x128 4` (local HTTP driver): +5.52% primary, −2.07% short, +6.97% long. These pairs were not reused for confirmation. - `confirm.py iqaheadL1x128` and `report.py iqaheadL1x128`: 48 distinct warm server processes; alternating AB/BA, three warmups per process, greedy Qwen3.8-Flash-Next IQ3_S, 131,072-token context, int8 KV, all 24,576 expert slots resident, default DFlash2. Each scored response generated 256 tokens with no prompt reuse. All paired input and output IDs matched. Every primary run accepted 178 of 277 offered drafts. Table reports geometric mean paired rate changes and two-sided Student-t intervals on log ratios; no pairs excluded. - `validate.py iqaheadL1x128`: 49 raw expert-intermediate/output comparisons (seven real layer formats × seven token-group sizes) matched; all 160 verifier-logit records (660,535,040 bytes) matched byte for byte, as did the primary state fingerprint. **Prefill follow-up:** [96 new paired runs on this commit, including every measurement](https://github.com/Niko1221/Strata/pull/6399#issuecomment-7044974904). The 1% slowdown threshold was ruled out for the short prompt only. The upper 95% bounds on prompt time were −0.12% (111 tokens), +1.19% (2,028), and +2.05% (8,168). The latter two are inconclusive, not established regressions. **Original decode confirmation, prefill results:** the 1% prefill non-regression guard did not clear for short prompts in either batch or for the primary in batch 2. Primary prompt time changed by +0.26% [−0.45, +0.96] then +3.05% [−3.61, +10.17]; short prompt time by +1.69% [−1.49, +4.98] then +1.34% [−7.36, +10.86]. Long prefill cleared the guard in both batches. **Not run locally:** HIP or SYCL builds, other GPUs, other packs, multi-GPU, or a direct comparison against #4811. Ready for maintainer review of the GB10 default; scope agreement is still pending. The follow-up above did not rule out a 1% prefill slowdown for the primary or long prompt. CUDA 12.9/13.0, HIP, SYCL and Metal builds passed in CI; the [Windows terminal/tools failure also occurs on the exact base commit](https://github.com/Niko1221/Strata/pull/6399#issuecomment-7041901567). <details><summary>Individual primary decode measurements (tok/s, stock → PR)</summary> | Pair | Order | Batch 1 | Batch 2 | |---:|---|---:|---:| | 1 | AB | 66.22 → 70.82 | 65.65 → 70.93 | | 2 | BA | 66.47 → 71.13 | 66.00 → 71.25 | | 3 | AB | 66.21 → 71.03 | 66.06 → 71.39 | | 4 | BA | 66.20 → 71.02 | 66.12 → 71.04 | | 5 | AB | 66.56 → 70.81 | 66.20 → 70.72 | | 6 | BA | 66.00 → 70.70 | 66.11 → 71.42 | | 7 | AB | 65.82 → 70.91 | 66.34 → 70.90 | | 8 | BA | 66.05 → 71.49 | 66.60 → 71.20 | | 9 | AB | 66.26 → 70.73 | 66.26 → 71.47 | | 10 | BA | 65.90 → 71.10 | 66.54 → 71.42 | | 11 | AB | 66.47 → 71.11 | 66.17 → 70.84 | | 12 | BA | 66.09 → 70.58 | 66.00 → 71.37 | </details>
En el sitio
Enlaces a install, modelos, releases.