#3153 sycl: preview Qwen3.8-Flash-Next IQ2_XS at ~100 tok/s on one B70

Beschreibung

## Summary

E2E report and discussion: https://github.com/Niko1221/Strata/issues/3150

This issue and the linked draft PR demonstrate **end-to-end performance of the combined optimizations** on the measured single-GPU workloads. The preview branch is a runnable integration snapshot for reproducing and discussing the result. It is not proposed as one patch to merge in full.

**If upstream is interested in these results, the work will be split into individual optimization PRs**, following the same approach as the focused submissions listed below. Each such PR can then be reviewed and validated independently, including its dispatch, fallback, correctness and performance scope.

One Intel Arc Pro B70 (32 GB VRAM), original Qwen3.8-Flash-Next IQ2_XS, 128 GiB container RAM limit. The preview measures **94.11–106.12 client tok/s** across fresh 1K/4K/8K/32K/128K prompts. The 32K result is **98.13 tok/s**, independently repeated at **98.11 tok/s**. “~100 tok/s” describes the measured range; 32K and 128K remain below 100 tok/s.

| Prompt tokens | upstream TTFT s | Preview TTFT s | TTFT reduction | upstream TPOT ms | Preview TPOT ms | TPOT reduction | Preview tok/s |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 1K | 1.171 | 1.170 | 0.08% | 13.723 | 9.665 | 29.57% | 103.46 |
| 4K | 3.482 | 3.038 | 12.73% | 13.493 | 9.424 | 30.16% | 106.12 |
| 8K | 7.072 | 5.948 | 15.89% | 13.846 | 9.920 | 28.35% | 100.80 |
| 32K | 29.940 | 24.598 | 17.84% | 14.219 | 10.191 | 28.33% | 98.13 |
| 128K | 140.048 | 117.176 | 16.33% | 14.939 | 10.625 | 28.88% | 94.11 |

K=1024. Each full request generates 256 tokens. Each configuration/length has three measured requests after one short and three full-length warmups. Thinking is off, seed42, no prompt-prefix reuse. TTFT is request submission to the first nonempty answer-text SSE delta; TPOT is the first-to-last nonempty delta interval divided by 255. Client tok/s is `1000 / mean(TPOT_ms)`. Reduction percentages use unrounded means: `(upstream - preview) / upstream * 100`; positive means lower latency.

The pure upstream comparator is engine 0.1.45 at `477ac13cbe8e582c20484871ce459d20d82df09d`, and the preview is based on the same commit. Both use identical inputs, sampler and fixed MTP parameters. Actual speculative work is allowed to vary and is not presented as equal-work kernel timing. All 35 paired requests in the completed 128 GiB series, including warmups, match text and token IDs. These are warmed E2E characterization results; model load and profiled requests are excluded. Shared model page cache was not flushed, so a container RAM limit is not a minimum installed-RAM claim.

## Model used

- **Base model:** [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), the original full-expert checkpoint. Its sparse MoE has 48 layers and 512 routed experts per layer, selecting 10 routed experts plus one shared expert. The official card lists 125B main-model parameters (6B active), plus 51B n-gram embeddings and 4B MTP parameters.
- **Measured quantization:** [ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), **IQ2_XS**, revision `ed59f92082b1e93c0e96d60a8b11aab089b52f09`. This is the GSQ/RCO mixed-precision GGUF release, with per-tensor quantization types; IQ2_XS is the release name, not a claim that every tensor uses one uniform two-bit format.
- **Exact files:** [IQ2_XS at the pinned revision](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/tree/ed59f92082b1e93c0e96d60a8b11aab089b52f09/IQ2_XS):
  - [Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/blob/ed59f92082b1e93c0e96d60a8b11aab089b52f09/IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf).
  - [Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/blob/ed59f92082b1e93c0e96d60a8b11aab089b52f09/IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf).
- **Runtime:** Strata's native GGUF expert path with its prepared pack. The MTP runtime is converted from the [original Qwen checkpoint at `de4b8e4d43b917e7706784d8bb445c9af86a3540`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/tree/de4b8e4d43b917e7706784d8bb445c9af86a3540); speculation uses spec4 / min-p0.5 / suffix3. The benchmark is text-only, with thinking off and vision disabled.

This measurement uses the original full-expert IQ2_XS release, rather than the pruned Coder model or the Swift/Unsloth alternatives.

## Optimizations in the preview and existing upstream PRs

The first five rows map to our existing focused upstream submissions. The remaining rows describe additional work present in the preview; they do not claim that an earlier PR already contains those extensions. All five linked upstream PRs are currently open. The E2E table measures the combined configuration; individual PR percentages are not added together.

| Optimization | Scope in this preview | Our existing upstream PR |
| --- | --- | --- |
| IQ4_XS MMVQ pair reuse, Q8 split and packed bit-select lookup | Reuse weight/header loads and reduce IQ4 decoding work on eligible shapes | [#2555](https://github.com/Niko1221/Strata/pull/2555) |
| INT8 QSA decode chunk-down and vector key/value loads | Shorter decode attention/reduction paths on eligible layouts | [#2582](https://github.com/Niko1221/Strata/pull/2582) |
| Synchronous adaptive expert-mirror replacement | Exchange resident/mirrored experts while preserving logical expert identity | [#2583](https://github.com/Niko1221/Strata/pull/2583) |
| Paired FP16 expert prefill, bounded IQ2 gate/up decoding and metadata reuse | Reduce prompt-path launches and decoding work | [#2584](https://github.com/Niko1221/Strata/pull/2584) |
| Q2_0 expert-down packed-word expansion | Replace per-byte/codebook expansion on the eligible down path | [#2746](https://github.com/Niko1221/Strata/pull/2746) |
| Packed resident expert layouts with raw host-mirror storage | Integrate the packed decode path with single-device streamed experts | Not separately submitted from this preview |
| Coalesced host-word exchange for packed resident experts | Extend mirror replacement to the packed-device/raw-host layout | Additional work beyond #2583; no separate upstream PR yet |
| Prepared and register-based IQ gate/up/down variants | Prepare/reuse packed inputs and metadata; include format/shape-specific variants | Not separately submitted from this preview |
| IQ3 packed/S4 and IQ4 INT4-DPAS paths | Additional eligible decode projection variants | Not separately submitted from this preview |
| IQ4 SSM weight-layout cache | Retain the selected SSM projection cache | Not separately submitted from this preview |
| GR cached normalization and bounded/vector/register projection variants | Reuse normalized inputs and low-rank results; reduce eligible projection work | Not separately submitted upstream from this preview |
| GR up rows16/Q8-up32 and paired 2D variants | Additional GR projection implementations for eligible cases | Not separately submitted from this preview |
| GDN state-only commit and QK register variants | Avoid unnecessary commit work and use eligible QK paths | Not separately submitted from this preview |
| Reusable decode scheduling pool and MTP/verify integration | Integrate the selected expert/projection paths with speculative verification | Not separately submitted from this preview |
| Exact FP32 TLA prompt attention | Optional B70/g31 prompt-attention path | Not separately submitted from this preview |
| Optional GR SOA, MTP Q8 SOA and IQ4 head SOA caches | Available in the code but **disabled in the measured B70 configuration** | Not separately submitted from this preview |

A control being enabled does not mean every shape-specific branch executes. These rows are an implementation inventory, not independent claims that every variant contributes to the measured gain.

## Related work and submission scope

This is a combined E2E demonstration for one Arc Pro B70, not a proposal to merge the whole preview as one change. If upstream is interested, we would split the remaining changes into focused PRs, following the earlier submissions, and coordinate with existing work first. The combined gain is not the sum of individual PR gains.

A 2026-10-19 review of upstream found related work below. We did not find another report with this same single-B70 IQ2_XS preview and the 1K–128K upstream/optimized TTFT/TPOT comparison. Existing single-B70 reports [#2922](https://github.com/Niko1221/Strata/pull/2922) and [#2867](https://github.com/Niko1221/Strata/pull/2867) use IQ3_S and IQ3_XXS respectively; [#1054](https://github.com/Niko1221/Strata/issues/1054) includes older single-card IQ2_XS results within a two-card bug report. Their hardware, workloads and measurement boundaries differ, so their TPS are not direct comparators.

| Area in this preview | Our focused submission | Related upstream work and overlap |
| --- | --- | --- |
| IQ4_XS MMVQ | [#2555](https://github.com/Niko1221/Strata/pull/2555) | [#2804](https://github.com/Niko1221/Strata/pull/2804) / [issue #2803](https://github.com/Niko1221/Strata/issues/2803) use KREG kernels at the same MMVQ entry points; different activation reuse/reduction. [#2437](https://github.com/Niko1221/Strata/pull/2437) changes an expert-dot lookup left unchanged by this preview. Coordinate these distinct paths. |
| INT8 QSA chunk/vector decode | [#2582](https://github.com/Niko1221/Strata/pull/2582) | [#2806](https://github.com/Niko1221/Strata/pull/2806) / [issue #2805](https://github.com/Niko1221/Strata/issues/2805) partially overlap in attention; also add slot batching, scores/top-k and active-chunk merge. Reconcile shared dispatch hooks. |
| Synchronous mirror adaptation and packed resident/raw host integration | [#2583](https://github.com/Niko1221/Strata/pull/2583) | Packed/raw storage and coalesced exchange extend our existing PR. These preview-specific extensions have not been independently submitted yet. |
| Paired FP16 experts and bounded IQ2 prompt decoding | [#2584](https://github.com/Niko1221/Strata/pull/2584) | [#1215](https://github.com/Niko1221/Strata/pull/1215) explored a different padded 16-expert oneMKL batch and was closed pending upstream E2E verification. |
| Q2_0 packed-word expansion and further IQ decode variants | [#2746](https://github.com/Niko1221/Strata/pull/2746) | [#2802](https://github.com/Niko1221/Strata/pull/2802) / [issue #2801](https://github.com/Niko1221/Strata/issues/2801) have **directly overlapping Q2 shift/mask expansion arithmetic**, plus broader IQ/lane kernels. Reconcile that common subset in the existing PRs; packed/register variants are separate implementation choices. |
| GR normalization/projection reuse and register variants | No separate upstream submission from this preview yet | [#2812](https://github.com/Niko1221/Strata/pull/2812) / [issue #2811](https://github.com/Niko1221/Strata/issues/2811) fuse a BF16 read; [#2810](https://github.com/Niko1221/Strata/pull/2810) implements Q8_0-weight reads; [#2488](https://github.com/Niko1221/Strata/pull/2488) targets A770 GR fusion. Same bottleneck, different paths. This preview's Q8-up32 retains BF16 weights and produces Q8_1 activations. |
| GDN state-only commit and QK registers | No separate submission from this preview yet | [#2755](https://github.com/Niko1221/Strata/pull/2755) / [#2586](https://github.com/Niko1221/Strata/pull/2586) optimize recurrence and batch commits across layers. Coordinate common state replay; the preview currently selects a per-layer state-only path. |
| FP32 prompt attention | No separate submission from this preview yet | [#2808](https://github.com/Niko1221/Strata/pull/2808) / [issue #2807](https://github.com/Niko1221/Strata/issues/2807) use 8-query unions and FP16 joint_matrix products. This preview uses FP32 with separate device/shape gua