#3150 [SYCL][Perf] Qwen3.8-Flash-Next IQ2_XS: ~100 tok/s on B70 (32 GB VRAM, 32 GiB container RAM limit)

Descrição

# Single-GPU IQ2_XS: combined E2E performance demonstration

Complete runnable preview: **[upstream draft PR #3153](https://github.com/Niko1221/Strata/pull/3153)**, targeting `main` at commit `3440db820d77cc7e71b4a30cc82e966290de1f2d`.

This issue and the linked draft PR demonstrate **end-to-end performance of the combined optimizations** on the measured single-GPU workloads. The preview branch is a runnable integration snapshot for reproducing and discussing the result. It is not proposed as one patch to merge in full.

**If upstream is interested in these results, the work will be split into individual optimization PRs**, following the same approach as the focused submissions listed below. Each such PR can then be reviewed and validated independently, including its dispatch, fallback, correctness and performance scope.

One Intel Arc Pro B70 (**32 GB VRAM**) running the original **Qwen3.8-Flash-Next IQ2_XS** measures **94.12–106.12 client tok/s** across 1K–128K prompts with a **32 GiB container RAM limit**. The 32K result is 98.06 tok/s; the 128K result is 94.12 tok/s. “~100 tok/s” describes this range. The configured container limit is not proof that a machine with 32 GB installed RAM can run this workload: shared model page cache was retained and the full physical-memory accounting has not been established.

## E2E performance results

### B70, 32 GiB container RAM limit

| Prompt | Upstream TTFT s | Preview TTFT s | TTFT reduction | Upstream TPOT ms | Preview TPOT ms | TPOT reduction | Preview tok/s |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 1K | 1.167 | 1.169 | -0.18% | 13.711 | 9.647 | 29.64% | 103.66 |
| 4K | 3.493 | 3.039 | 12.99% | 13.468 | 9.423 | 30.04% | 106.12 |
| 8K | 7.093 | 5.952 | 16.08% | 13.829 | 9.917 | 28.29% | 100.84 |
| 32K | 29.982 | 24.577 | 18.03% | 14.200 | 10.198 | 28.18% | 98.06 |
| 128K | 140.328 | 117.213 | 16.47% | 14.928 | 10.625 | 28.83% | 94.12 |

### B70, 64 GiB container RAM limit

| Prompt | Upstream TTFT s | Preview TTFT s | TTFT reduction | Upstream TPOT ms | Preview TPOT ms | TPOT reduction | Preview tok/s |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 1K | 1.168 | 1.169 | -0.11% | 13.714 | 9.655 | 29.60% | 103.57 |
| 4K | 3.496 | 3.038 | 13.10% | 13.480 | 9.420 | 30.12% | 106.15 |
| 8K | 7.090 | 5.945 | 16.14% | 13.818 | 9.912 | 28.27% | 100.88 |
| 32K | 30.007 | 24.559 | 18.16% | 14.202 | 10.192 | 28.24% | 98.12 |
| 128K | 140.399 | 117.113 | 16.59% | 14.932 | 10.626 | 28.84% | 94.11 |

### B70, 128 GiB container RAM limit

| Prompt tokens | upstream TTFT s | Preview TTFT s | TTFT reduction | upstream TPOT ms | Preview TPOT ms | TPOT reduction | Preview tok/s |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 1K | 1.171 | 1.170 | 0.08% | 13.723 | 9.665 | 29.57% | 103.46 |
| 4K | 3.482 | 3.038 | 12.73% | 13.493 | 9.424 | 30.16% | 106.12 |
| 8K | 7.072 | 5.948 | 15.89% | 13.846 | 9.920 | 28.35% | 100.80 |
| 32K | 29.940 | 24.598 | 17.84% | 14.219 | 10.191 | 28.33% | 98.13 |
| 128K | 140.048 | 117.176 | 16.33% | 14.939 | 10.625 | 28.88% | 94.11 |

### B60, 128 GiB container RAM limit

| Prompt | Upstream TTFT s | Preview TTFT s | TTFT reduction | Upstream TPOT ms | Preview TPOT ms | TPOT reduction | Preview tok/s |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 1K | 1.605 | 1.664 | -3.67% | 22.327 | 14.220 | 36.31% | 70.32 |
| 4K | 5.067 | 4.669 | 7.87% | 22.891 | 14.006 | 38.82% | 71.40 |
| 8K | 10.454 | 9.289 | 11.15% | 23.752 | 14.392 | 39.41% | 69.48 |
| 32K | 44.911 | 39.016 | 13.13% | 23.311 | 14.442 | 38.04% | 69.24 |
| 128K | 214.801 | 189.609 | 11.73% | 23.721 | 15.427 | 34.97% | 64.82 |

## Measurement method and scope

Measured preview commit `3440db820d77cc7e71b4a30cc82e966290de1f2d` against pure upstream `477ac13cbe8e582c20484871ce459d20d82df09d`, engine 0.1.45.

Three measured requests per configuration/length, after one short and three full-length warmups; 256 generated tokens, thinking off, seed42 and no prompt-prefix reuse. Negative reduction means a latency regression.

K=1024. Each full request generates 256 tokens. Each configuration/length has three measured requests after one short and three full-length warmups. Thinking is off, seed42, no prompt-prefix reuse. TTFT is request submission to the first nonempty answer-text SSE delta; TPOT is the first-to-last nonempty delta interval divided by 255. Client tok/s is `1000 / mean(TPOT_ms)`. Reduction percentages use unrounded means: `(upstream - preview) / upstream * 100`; positive means lower latency.

The pure upstream comparator is engine 0.1.45 at `477ac13cbe8e582c20484871ce459d20d82df09d`, and the preview is based on the same commit. Both use identical inputs, sampler and fixed MTP parameters. Actual speculative work is allowed to vary and is not presented as equal-work kernel timing. All 35 paired requests in the completed 128 GiB series, including warmups, match text and token IDs. These are warmed E2E characterization results; model load and profiled requests are excluded. Shared model page cache was not flushed, so a container RAM limit is not a minimum installed-RAM claim.

105 paired requests, including warmups, have matching text and token IDs in the completed comparisons. Actual MTP work is not normalized. The tables show whole-engine behavior, including adaptation and actual cache capacity.

B70 retains the selected fixed expert-cache request16000 and mirror budget16384 MiB. B60 uses a native bmg-g21 build, TLA OFF, auto expert-cache sizing and mirror budget32768 MiB on both executables. These platform differences are part of the reported configurations, not isolated causal GPU comparisons.

RAM values are container limits in GiB (N×2³⁰ bytes). Shared model page cache was retained. These measurements do not establish minimum installed system RAM.

Maximum observed swap sample across completed cells: 0 bytes. New cgroup OOM / OOM-kill events across completed cells: 0 / 0.

## Model used

- **Base model:** [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), the original full-expert checkpoint. Its sparse MoE has 48 layers and 512 routed experts per layer, selecting 10 routed experts plus one shared expert. The official card lists 125B main-model parameters (6B active), plus 51B n-gram embeddings and 4B MTP parameters.
- **Measured quantization:** [ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), **IQ2_XS**, revision `ed59f92082b1e93c0e96d60a8b11aab089b52f09`. This is the GSQ/RCO mixed-precision GGUF release, with per-tensor quantization types; IQ2_XS is the release name, not a claim that every tensor uses one uniform two-bit format.
- **Exact files:** [IQ2_XS at the pinned revision](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/tree/ed59f92082b1e93c0e96d60a8b11aab089b52f09/IQ2_XS):
  - [Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/blob/ed59f92082b1e93c0e96d60a8b11aab089b52f09/IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf).
  - [Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/blob/ed59f92082b1e93c0e96d60a8b11aab089b52f09/IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf).
- **Runtime:** Strata's native GGUF expert path with its prepared pack. The MTP runtime is converted from the [original Qwen checkpoint at `de4b8e4d43b917e7706784d8bb445c9af86a3540`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/tree/de4b8e4d43b917e7706784d8bb445c9af86a3540); speculation uses spec4 / min-p0.5 / suffix3. The benchmark is text-only, with thinking off and vision disabled.

This measurement uses the original full-expert IQ2_XS release, rather than the pruned Coder model or the Swift/Unsloth alternatives.

## Related work and submission scope

This is a combined E2E demonstration for one Arc Pro B70, not a proposal to merge the whole preview as one change. If upstream is interested, we would split the remaining changes into focused PRs, following the earlier submissions, and coordinate with existing work first. The combined gain is not the sum of individual PR gains.

A 2026-10-19 review of upstream found related work below. We did not find another report with this same single-B70 IQ2_XS preview and the 1K–128K upstream/optimized TTFT/TPOT comparison. Existing single-B70 reports [#2922](https://github.com/Niko1221/Strata/pull/2922) and [#2867](https://github.com/Niko1221/Strata/pull/2867) use IQ3_S and IQ3_XXS respectively; [#1054](https://github.com/Niko1221/Strata/issues/1054) includes older single-card IQ2_XS results within a two-card bug report. Their hardware, workloads and measurement boundaries differ, so their TPS are not direct comparators.

| Area in this preview | Our focused submission | Related upstream work and overlap |
| --- | --- | --- |
| IQ4_XS MMVQ | [#2555](https://github.com/Niko1221/Strata/pull/2555) | [#2804](https://github.com/Niko1221/Strata/pull/2804) / [issue #2803](https://github.com/Niko1221/Strata/issues/2803) use KREG kernels at the same MMVQ entry points; different activation reuse/reduction. [#2437](https://github.com/Niko1221/Strata/pull/2437) changes an expert-dot lookup left unchanged by this preview. Coordinate these distinct paths. |
| INT8 QSA chunk/vector decode | [#2582](https://github.com/Niko1221/Strata/pull/2582) | [#2806](https://github.com/Niko1221/Strata/pull/2806) / [issue #2805](https://github.com/Niko1221/Strata/issues/2805) partially overlap in attention; also add slot batching, scores/top-k and active-chunk merge. Reconcile shared dispatch hooks. |
| Synchronous mirror adaptation and packed resident/raw host integration | [#2583](https://github.com/Niko1221/Strata/pull/2583) | Packed/raw storage and coalesced exchange extend our existing PR. These preview-specific extensions have not been independently submitted yet. |
| Paired FP16 experts and bounded IQ2 prompt decoding | [#2584](https://github.com/Niko1221/Strata/pull/2584) | [#1215](https://github.com/Niko1221/Strata/pull/1215) explored a different padded 16-expert oneMKL batch and was closed pending upstream E2E verification. |
| Q2_0 packed-word expansion and further IQ decode variants | [#2746](https://github.com/Niko1221/Strata/pull/2746) | [#2802](https://github.com/Niko1221/Strata/pull/2802) / [issue #2801](https://github.com/Niko1221/Strata/issues/2801) have **directly overlapping Q2 shift/mask expansion arithmetic**, plus broader IQ/lane kernels. Reconcile that common subset in the existing PRs; packed/register variants are separate implementation choices. |
| GR normalization/projection reuse and register variants | No separate upstream submission from this preview yet | [#2812](https://github.com/Niko1221/Strata/pull/2812) / [issue #2811](https://github.com/Niko1221/Strata/issues/2811) fuse a BF16 read; [#2810](https://github.com/Niko1221/Strata/pull/2810) implements Q8_0-weight reads; [#2488](https://github.com/Niko1221/Strata/pull/2488) targets A770 GR fusion. Same bottleneck, different paths. This preview's Q8-up32 retains BF16 weights and produces Q8_1 activations. |
| GDN state-only commit and QK registers | No separate submission from this preview yet | [#2755](https://github.com/Niko1221/Strata/pull/2755) / [#2586](https://github.com/Niko1221/Strata/pull/2586) optimize recurrence and batch commits across layers. Coordinate common state replay; the preview currently selects a per-layer state-only path. |
| FP32 prompt attention | No separate submission from this preview yet | [#2808](https://github.com/Niko1221/Strata/pull/2808) / [issue #2807](https://github.com/Niko1221/Strata/issues/2807) use 8-query unions and FP16 joint_matrix products. This preview uses FP32 with separate device/shape guards; numerical contracts differ. |

Our five linked upstream PRs are open at this snapshot. “No separate submission” refers on