Pull requests / #2755

#2755 sycl/A770: commit all GDN layers' conv history + state replay in two launches (STRATA_GDN_COMMIT_BATCH; bitwise)

open · @aslater3 · 0 comments · View on GitHub

BenchmarksModels & quants

Description

## sycl/A770: commit all GDN layers' conv history + state replay in two launches (STRATA_GDN_COMMIT_BATCH; bitwise)

capture_commit replayed the accepted tokens into each GDN layer's state with two launches a layer (gdn_conv_commit and
the split state replay): 36 layers -> 72 launches at the end of every verify window, on the critical path into the
next round's drafting. gdn_commit_replay_layers runs the same two kernels over all layers at once (the layer is the
outermost work-group index, selecting each layer's state, history and inputs), so every thread computes what it did.
It declines (the per-layer loop runs) where the split replay is not the path. Default on on Alchemist, where it is
measured; STRATA_GDN_COMMIT_BATCH=1 forces it on, =0 off. Needs the split replay (#2586).

gdn_bench GDN_CBATCH=1 (new): 36 layers, n_keep 1..6, every float of every layer's state and history compared per
layer vs batched: 0 differ; 1.96 -> 0.83 ms. Engine, A770, Coder IQ1_M, 8K code prompt, 3 pairs: MTP drafting
7.51 / 7.31 / 7.44 -> 6.58 / 6.44 / 6.50 ms a round (-0.91), decode 35.68 / 39.41 / 39.58 -> 35.58 / 40.04 / 41.63 tok/s,
reference tokens (one known low-hit variant).

Stacked on #2586 (the split GDN replay, which this batches); the diff to review is the top commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Related on this site

Quick links to related pages.