Pull requests / #2755
#2755 sycl/A770: commit all GDN layers' conv history + state replay in two launches (STRATA_GDN_COMMIT_BATCH; bitwise)
open · @aslater3 · 0 comments · View on GitHub
Description
## sycl/A770: commit all GDN layers' conv history + state replay in two launches (STRATA_GDN_COMMIT_BATCH; bitwise) capture_commit replayed the accepted tokens into each GDN layer's state with two launches a layer (gdn_conv_commit and the split state replay): 36 layers -> 72 launches at the end of every verify window, on the critical path into the next round's drafting. gdn_commit_replay_layers runs the same two kernels over all layers at once (the layer is the outermost work-group index, selecting each layer's state, history and inputs), so every thread computes what it did. It declines (the per-layer loop runs) where the split replay is not the path. Default on on Alchemist, where it is measured; STRATA_GDN_COMMIT_BATCH=1 forces it on, =0 off. Needs the split replay (#2586). gdn_bench GDN_CBATCH=1 (new): 36 layers, n_keep 1..6, every float of every layer's state and history compared per layer vs batched: 0 differ; 1.96 -> 0.83 ms. Engine, A770, Coder IQ1_M, 8K code prompt, 3 pairs: MTP drafting 7.51 / 7.31 / 7.44 -> 6.58 / 6.44 / 6.50 ms a round (-0.91), decode 35.68 / 39.41 / 39.58 -> 35.58 / 40.04 / 41.63 tok/s, reference tokens (one known low-hit variant). Stacked on #2586 (the split GDN replay, which this batches); the diff to review is the top commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Related on this site
Quick links to related pages.