#6566 [Engine] Pair GDN recurrence inputs and tile prefill kernels in the SYCL engine on Battlemage (BMG)
open · draft · @xiangyuT · 0 コメント · GitHub で見る
Server & APIMulti-GPUModels & quantsLinux
本文
## Problem and result Pairing consecutive GDN inputs reduces repeated loads during prompt recurrence. Tiling the projection, convolution and residual updates reduces the time spent in the rest of the GDN prefill path. On one B70 with the default mirror configuration, 32K engine prefill fell from **22.058 to 20.900 s (-5.25%)**. Client TTFT went from 22.096 to 20.939 s and TPOT from 11.532 to 11.525 ms/token (-0.06%). Both runs left `STRATA_LEND_MIRROR` and the GDN dispatch overrides unset. For this streamed model, prefill borrowed 2,401 cache slots. The engine mirrored 1,200 of them in pinned host memory, using its [default cap](https://github.com/Niko1221/Strata/blob/6e38f588a646c0ee6948a74afbe670eb75896106/engine/sycl/src/program/generate.cpp#L5367-L5373). | Input tokens | Clean prefill (s) | PR prefill (s) | Prefill change | Client TTFT (s), clean → PR | TPOT (ms/token), clean → PR | TPOT change | Equal 256-token replies | |---:|---:|---:|---:|---:|---:|---:|---:| | 14,192 | 10.034 | 9.392 | -6.40% | 10.063 → 9.421 | 12.010 → 12.001 | -0.07% | 3/3 | | 27,000 | 19.226 | 18.257 | -5.04% | 19.262 → 18.293 | 11.810 → 11.806 | -0.03% | 3/3 | | 30,000 | 20.730 | 19.616 | -5.38% | 20.767 → 19.653 | 12.047 → 12.041 | -0.05% | 3/3 | | 32,768 | 22.058 | 20.900 | -5.25% | 22.096 → 20.939 | 11.532 → 11.525 | -0.06% | 3/3 | Each length used three distinct fresh inputs, with 256 generated tokens per reply. The clean reference is the existing run of the same unmodified main binary on the same B70, with matching engine settings and request files. Both processes excluded three 32K warmups before the measured 32K block. The clean process had also exercised shorter lengths before that block. TTFT measures time to first nonempty response text. TPOT is the time between the first and last nonempty response text divided by 255. <details><summary>Individual requests with the default mirror</summary> Values follow the same three input IDs. Parentheses give the maximum minus the minimum. | Run | Input tokens | Engine prefill (s) | TTFT (s) | TPOT (ms/token) | Total (s) | |---|---:|---|---|---|---| | Clean | 14,192 | 10.467 / 9.921 / 9.714 (0.753) | 10.496 / 9.950 / 9.743 (0.753) | 12.103 / 11.785 / 12.143 (0.357) | 13.584 / 12.956 / 12.841 (0.743) | | PR | 14,192 | 9.074 / 9.280 / 9.820 (0.746) | 9.103 / 9.310 / 9.849 (0.746) | 12.102 / 11.779 / 12.124 (0.344) | 12.191 / 12.315 / 12.942 (0.752) | | Clean | 27,000 | 19.076 / 19.584 / 19.018 (0.565) | 19.111 / 19.619 / 19.055 (0.564) | 11.891 / 11.878 / 11.662 (0.229) | 22.144 / 22.649 / 22.030 (0.619) | | PR | 27,000 | 18.348 / 18.397 / 18.025 (0.371) | 18.383 / 18.433 / 18.062 (0.371) | 11.889 / 11.872 / 11.659 (0.230) | 21.416 / 21.462 / 21.036 (0.426) | | Clean | 30,000 | 21.001 / 20.622 / 20.568 (0.433) | 21.038 / 20.659 / 20.605 (0.433) | 11.927 / 12.283 / 11.931 (0.356) | 24.081 / 23.792 / 23.649 (0.432) | | PR | 30,000 | 19.527 / 19.320 / 20.000 (0.680) | 19.564 / 19.358 / 20.037 (0.679) | 11.925 / 12.274 / 11.924 (0.350) | 22.606 / 22.489 / 23.079 (0.590) | | Clean | 32,768 | 22.114 / 21.794 / 22.267 (0.473) | 22.153 / 21.832 / 22.304 (0.472) | 11.628 / 11.198 / 11.768 (0.570) | 25.119 / 24.689 / 25.306 (0.617) | | PR | 32,768 | 20.514 / 21.228 / 20.958 (0.714) | 20.553 / 21.267 / 20.996 (0.713) | 11.624 / 11.194 / 11.756 (0.562) | 23.519 / 24.122 / 23.995 (0.603) | </details> <details><summary>Supplementary comparison with the full 2,401-slot mirror</summary> Both arms set `STRATA_LEND_MIRROR=2401` to mirror every borrowed cache slot. This controlled comparison measures GDN prefill with the full expert-cache loan available in pinned host memory. Both processes excluded three 32K warmups. | Input tokens | Clean prefill (s) | PR prefill (s) | Prefill change | Client TTFT (s), clean → PR | TPOT (ms/token), clean → PR | TPOT change | Equal 256-token replies | |---:|---:|---:|---:|---:|---:|---:|---:| | 14,192 | 6.999 | 6.469 | -7.57% | 7.028 → 6.498 | 12.012 → 12.014 | +0.02% | 3/3 | | 27,000 | 13.354 | 12.317 | -7.77% | 13.390 → 12.353 | 11.808 → 11.806 | -0.02% | 3/3 | | 30,000 | 14.605 | 13.463 | -7.82% | 14.643 → 13.501 | 12.051 → 12.050 | -0.01% | 3/3 | | 32,768 | 15.645 | 14.408 | -7.90% | 15.683 → 14.447 | 11.533 → 11.534 | +0.00% | 3/3 | The three values below follow matching input IDs. Parentheses give the maximum minus the minimum. | Run | Input tokens | Engine prefill (s) | TTFT (s) | TPOT (ms/token) | Total (s) | |---|---:|---|---|---|---| | Clean | 14,192 | 7.012 / 6.997 / 6.988 (0.024) | 7.042 / 7.026 / 7.017 (0.024) | 12.115 / 11.783 / 12.138 (0.355) | 10.132 / 10.032 / 10.114 (0.100) | | PR | 14,192 | 6.477 / 6.466 / 6.464 (0.013) | 6.506 / 6.495 / 6.493 (0.013) | 12.116 / 11.785 / 12.142 (0.357) | 9.597 / 9.501 / 9.591 (0.096) | | Clean | 27,000 | 13.359 / 13.344 / 13.360 (0.016) | 13.394 / 13.381 / 13.396 (0.015) | 11.894 / 11.873 / 11.658 (0.236) | 16.428 / 16.410 / 16.370 (0.058) | | PR | 27,000 | 12.320 / 12.315 / 12.316 (0.005) | 12.356 / 12.351 / 12.352 (0.005) | 11.894 / 11.868 / 11.656 (0.237) | 15.391 / 15.379 / 15.325 (0.066) | | Clean | 30,000 | 14.610 / 14.613 / 14.593 (0.020) | 14.648 / 14.652 / 14.629 (0.023) | 11.936 / 12.286 / 11.930 (0.356) | 17.692 / 17.787 / 17.673 (0.114) | | PR | 30,000 | 13.463 / 13.469 / 13.459 (0.010) | 13.500 / 13.506 / 13.496 (0.009) | 11.935 / 12.285 / 11.929 (0.356) | 16.545 / 16.640 / 16.540 (0.100) | | Clean | 32,768 | 15.646 / 15.642 / 15.646 (0.004) | 15.685 / 15.680 / 15.685 (0.005) | 11.632 / 11.199 / 11.769 (0.569) | 18.652 / 18.537 / 18.687 (0.150) | | PR | 32,768 | 14.406 / 14.410 / 14.408 (0.005) | 14.445 / 14.449 / 14.447 (0.004) | 11.628 / 11.200 / 11.773 (0.573) | 17.411 / 17.306 / 17.450 (0.144) | </details> ## Related discussion Maintainer scope agreement for the proposed automatic Battlemage dispatch is pending. ## Approach - On Level Zero Battlemage with subgroup 16, the 128-wide, 16-key/48-value-head recurrence reads two successive inputs into registers and updates the FP32 state in token order. Automatic selection starts at 128 tokens. An odd final token uses one update. Existing recurrence variant selection takes precedence. With `STRATA_SYCL_GDN_PREFILL_REG_QK` set, values beginning with 1 request register Q/K on supported inputs. Other explicit values select the column kernel. - From 512 rows, eligible layers use a four-token BF16 projection (48 outputs, 2,560 input columns, stride 96, row count divisible by four), fused convolution and L2 normalization, and an aligned four-stream residual write with RMSNorm. Existing convolution settings retain priority, including `STRATA_GDN_CONV_SERIAL` over `STRATA_GDN_CONVL2` and an explicit `STRATA_GDN_CONVL2=0`. Tile selection uses the queue's architecture, shapes and alignment. Other device families and ineligible shapes use the existing paths. The 512-row tile threshold comes from a B70 crossover check of the same kernel sources. At 512 rows, the separate and tiled BF16 projections took 0.0409 and 0.0221 ms, convolution plus L2 took 0.1207 and 0.0931 ms, and residual plus RMSNorm took 0.1336 and 0.1144 ms. The end-to-end table uses the complete PR with automatic dispatch. The selector checks cover convolution at 64, 511 and 512 rows and explicit serial/separate overrides. Earlier profiling also recorded the new convolution and residual kernels separately. <details><summary>Development measurements supporting the tile threshold</summary> The first table lists every process median used in the summary. The second lists all nine samples from a separate diagnostic run with the same kernel implementations and compiler flags. Its only addition printed the stored samples after the timing loop, before sorting. | Operation | Rows | Ordinary, processes 1 / 2 / 3 (ms) | Max−min (ms) | Tiled, processes 1 / 2 / 3 (ms) | Max−min (ms) | |---|---:|---|---:|---|---:| | projection | 512 | 0.040887 / 0.040919 / 0.040711 | 0.000208 | 0.022115 / 0.021783 / 0.022341 | 0.000558 | | projection | 1,024 | 0.075876 / 0.075658 / 0.075468 | 0.000408 | 0.041375 / 0.042408 / 0.042064 | 0.001033 | | projection | 8,192 | 0.649472 / 0.648599 / 0.648815 | 0.000873 | 0.244516 / 0.245024 / 0.245368 | 0.000852 | | conv | 512 | 0.121252 / 0.120238 / 0.120721 | 0.001014 | 0.093063 / 0.094112 / 0.093088 | 0.001049 | | conv | 1,024 | 0.227856 / 0.228617 / 0.227406 | 0.001211 | 0.190304 / 0.192931 / 0.190861 | 0.002627 | | conv | 8,192 | 1.869502 / 1.853806 / 1.851281 | 0.018221 | 1.337943 / 1.338117 / 1.333760 | 0.004357 | | residual | 512 | 0.133634 / 0.133743 / 0.132332 | 0.001411 | 0.114356 / 0.114379 / 0.113796 | 0.000583 | | residual | 1,024 | 0.262012 / 0.262244 / 0.262059 | 0.000232 | 0.225277 / 0.226437 / 0.225139 | 0.001298 | | residual | 1,025 | 0.262570 / 0.263007 / 0.262769 | 0.000437 | 0.225886 / 0.226159 / 0.225995 | 0.000273 | | residual | 8,192 | 2.028981 / 2.033383 / 2.028712 | 0.004671 | 1.749265 / 1.750543 / 1.748100 | 0.002443 | Additional nine-launch capture, in milliseconds: | Operation | Rows | Implementation | Samples 1–9 (ms) | Max−min (ms) | |---|---:|---|---|---:| | projection | 512 | ordinary | 0.041693 / 0.041270 / 0.041623 / 0.041199 / 0.041060 / 0.042509 / 0.040468 / 0.041101 / 0.040991 | 0.002041 | | projection | 512 | tiled | 0.022637 / 0.022010 / 0.021538 / 0.021174 / 0.022604 / 0.021467 / 0.021555 / 0.021631 / 0.021997 | 0.001463 | | projection | 1,024 | ordinary | 0.075793 / 0.075313 / 0.074929 / 0.075491 / 0.075464 / 0.074810 / 0.075745 / 0.073793 / 0.075897 | 0.002104 | | projection | 1,024 | tiled | 0.041471 / 0.041669 / 0.047056 / 0.043013 / 0.042353 / 0.042312 / 0.042398 / 0.042231 / 0.041046 | 0.006010 | | projection | 8,192 | ordinary | 0.649282 / 0.648316 / 0.648719 / 0.649287 / 0.647497 / 0.648853 / 0.648509 / 0.648157 / 0.647492 | 0.001795 | | projection | 8,192 | tiled | 0.245342 / 0.244998 / 0.245417 / 0.244110 / 0.244548 / 0.244571 / 0.245504 / 0.246388 / 0.245544 | 0.002278 | | conv | 512 | ordinary | 0.122051 / 0.121070 / 0.119575 / 0.120795 / 0.120390 / 0.119503 / 0.121120 / 0.120816 / 0.119086 | 0.002965 | | conv | 512 | tiled | 0.095368 / 0.087640 / 0.093990 / 0.089669 / 0.094564 / 0.090419 / 0.098125 / 0.085885 / 0.095438 | 0.012240 | | conv | 1,024 | ordinary | 0.227738 / 0.226016 / 0.237709 / 0.226425 / 0.238912 / 0.228712 / 0.236809 / 0.226152 / 0.235689 | 0.012896 | | conv | 1,024 | tiled | 0.194394 / 0.174859 / 0.190886 / 0.175776 / 0.194288 / 0.178616 / 0.191364 / 0.175700 / 0.192856 | 0.019535 | | conv | 8,192 | ordinary | 1.877893 / 1.854272 / 1.875517 / 1.848548 / 1.877254 / 1.847565 / 1.873078 / 1.852742 / 1.882493 | 0.034928 | | conv | 8,192 | tiled | 1.347836 / 1.319196 / 1.340044 / 1.315078 / 1.344280 / 1.313674 / 1.373761 / 1.321560 / 1.342750 | 0.060087 | | residual | 512 | ordinary | 0.135390 / 0.132429 / 0.098076 / 0.137903 / 0.102707 / 0.133073 / 0.099995 / 0.134678 / 0.100919 | 0.039827 | | residual | 512 | tiled | 0.114185 / 0.092875 / 0.114999 / 0.097107 / 0.115245 / 0.096068 / 0.114594 / 0.095183 / 0.114897 | 0.022370 | | residual | 1,024 | ordinary | 0.264186 / 0.265082 / 0.264674 / 0.265942 / 0.272913 / 0.261593 / 0.263644 / 0.263244 / 0.262502 | 0.011320 | | residual | 1,024 | tiled | 0.228477 / 0.227947 / 0.247047 / 0.228772 / 0.228040 / 0.223431 / 0.225558 / 0.224342 / 0.226524 | 0.023616 | | residual | 1,025 | ordinary | 0.262491 / 0.264941 / 0.262492 / 0.263354 / 0.261608 / 0.262878 / 0.260645 / 0.263853 / 0.259736 | 0.005205 | | residual | 1,025 | tiled | 0.226251 / 0.226731 / 0.227820 / 0.226555 / 0.226164 / 0.225645 / 0.225591 / 0.226975 / 0.225937 | 0.002229 | | residual | 8,192 | ordinary | 2.027561 / 2.025548 / 2.032049 / 2.026818 / 2.022938 / 2.030674 / 2.031405 / 2.030206 / 2.025812 | 0.009111 | | residual | 8,192 | tiled | 1.747816 / 1.749916 / 1.748880 / 1.746666 / 1.752162 / 1.747577 / 1.751658 / 1.746608 / 1.750460 | 0.005554 | </details> ## Validation **Passed:** - Built the unmodified [upstream base](https://github.com/Niko1221/Stra
関連リンク
インストール・モデル・リリースへのサイト内リンク。