#3248 [Engine] SYCL: +10-35% prompt speed on Arc Pro B70/B65, copy-engine hang fix, int8 crossover cache, failing tests fixed

Description

Sorry it took so long i have been working on nextsycl i had forgotten to bring in the enhancements here, hopefully this will get approved, Improvements all around!

Branch: maxfridbe/Strata_B70 intel-arc-0.2.0 (head d1192104), based on main f57c0adc; merges cleanly into main e88ee39d. Replaces #1111 (closed in the 0.2.0 tracker reset), cut down to what main does not have yet. Only sycl/ and docs/INTEL*.md change.
Problem and result

On Arc Pro B70 and B65 (Battlemage, 32 GB), main’s SYCL engine has one hang, two failing tests and a slow first prompt, and its long prompts are slower than they need to be:

    The copy engine hangs (the card ends up wedged). A queue copy with no device end (pinned host <-> pageable host) goes to the Arc’s copy engine, which hangs on it (Engine reset: engine_class=bcs; repeated runs wedge the device until a reboot). conversation_snapshot_test does this on every run, and so would saving or restoring a --kv-resident conversation. After: host-only copies run on the CPU; the test passes with 0 resets.
    The int8 crossover is re-measured at every start. strata-sycl.sh runs the engine in a --rm container, so STRATA_PREFILL_MMQ=auto’s cache in ~/.cache/strata is lost. The first 2,185-token prompt (B70, Coder IQ1_M) runs at 761 tok/s instead of 1,450. After: the cache is kept under the data root (STRATA_CACHE_DIR).
    Two tests fail on SYCL. s2_expert_grouped_parity (4 failures: the transposed grouped kernels get FMA-contracted under -fp-model=precise, the reference kernels do not) and gr_parity (its QFUSE check expects a q8_1 image the SYCL fused read never writes; the engine is unaffected). After: both pass, and wrong bytes still fail.
    Long prompts are slower than they could be. Two prompt-path kernels change, each with an off switch:
        prompt attention scores one work-item per cell, 2.0-2.2x the warp kernel (STRATA_ATTN_PERCELL=0 restores it);
        from 8K cells, QSA block selection scores with oneMKL GEMM tiles, so selection at 40K goes from 2.1 s to 0.7 s (STRATA_SELECT_GEMM=0 restores it).

    Prompt speed, interleaved A/B on the same B70 binary: +10% at 2K, +21% at 40K and +35% at 128K. Generation speed is unchanged.

Also included, as small tooling fixes for multi-card and Podman hosts:

    STRATA_SYCL_RUNNER=docker|podman|native|distrobox:<name> for strata-sycl.sh.
    gpustat.py reads two cards correctly. Before, it showed 163% busy and both cards’ VRAM summed as one.
    setup_intel.py --check sizes a model across all cards.
    setup_intel.py knows the B65’s PCI id (e222).

Related discussion

#1111 (closed by the 0.2.0 reset; this is its rebase), #423, #809. The hang fix, the cache and the test fixes need no scope agreement. The two prompt kernels change a SYCL-only prompt-path default; both have off switches. Happy to split this into three PRs (hang and test fixes / prompt kernels / tooling) if you prefer.
Approach

    Everything stays in sycl/. The new kernels are declared locally in prefill.cpp, not in the shared headers.
    The GEMM selection only turns on from 8K cells, because oneMKL’s launch cost loses below that (B70: -2% at 2K, even at 8K, 3x at 40K). It reuses the attention scratch, so no VRAM is added.
    Measured and left out, because none of them won on the B70 (identical tokens in each case):
        STRATA_EXP_MODE=8: 82.5 vs 82.5 tok/s decode.
        Queues without profiling: +0.25% decode.
        A 66-wide register top-k for 262K contexts: 2.6x faster in isolation, but it spills registers, so decode was -1.5% and the 128K prompt -6.6%.

Validation

Environment: Arc Pro B70 (bmg-g31) + Arc Pro B65, 32 GB each, xe driver, PCIe Gen5 x8, Ryzen 9 9950X, 61 GiB RAM, Fedora Silverblue 44 (kernel 7.1.5), strata-sycl-dev image under rootless Podman (oneAPI 2026.1), AOT=bmg-g31. Revision: intel-arc-0.2.0 c44fb63b (d1192104 only adds the B65 name), against main f57c0adc built the same way.

Passed:

    Build: AOT=bmg-g31 JOBS=8 bash sycl/tools/build.sh gives 0 errors on both main and the branch.
    ctest on the B70, and on the B65 (ONEAPI_DEVICE_SELECTOR=level_zero:1): 34 of 35 pass on each. That includes conversation_snapshot_test, s2_expert_grouped_parity, gr_parity and the newly registered qsa_prompt_attn_parity, which passes on int8, FP16 and Q4_0 pools. For comparison, main passes 30 of 33, with the snapshot test excluded because it wedges the card. No Engine reset or wedged appeared in the kernel log after any run.
    Token parity. Settings: --pcie-frac 0 --adapt-every 0 STRATA_PREFILL_MMQ=1, IQ2_XS, 2,185-token prompt, 256 greedy tokens. Two runs of main are identical, and the branch with both new kernels off is identical to main.
        With the new kernels on, the first-token KL against the old kernels is 3.9e-10 nats at 8K and 1.3e-11 at 40K, with the same top-1. That is the same size as main’s own int8-vs-FP16 switch (8.4e-10 and 5.1e-12).
        The 256 greedy tokens after a 128K prompt are identical.
        Note: with the default STRATA_PREFILL_MMQ=auto, two runs of main differ, because the per-group choice comes from a timing.
    setup_intel.py --check on the two-card host. The output is in commit 7b70671a.
    Benchmark. Qwen3.8-Flash-Next Coder IQ1_M and IQ2_XS with the served flags (--prefill 4096 --spec 4 --spec-min-p 0.5 --mtp --kv int8 --stream-experts --vram-reserve-mib 2048 --kv-resident 32768, MMQ auto with a warm crossover cache for both builds). Cold page cache, 256 greedy tokens, prefill / decode in tok/s:

card	model	prompt	main f57c0adc	intel-arc-0.2.0
B70	Coder IQ1_M	2,185	1,312 / 87.6	1,449 / 88.1
B70	Coder IQ1_M	8,000	1,620 / 84.9	1,826 / 88.5
B70	Coder IQ1_M	40,000	1,580 / 77.6	1,909 / 78.7
B70	Coder IQ1_M	128,000	1,351 / 78.5	1,821 / 76.4
B70	IQ2_XS	2,185	1,053 / 80.1	1,355 / 77.4
B70	IQ2_XS	40,000	1,376 / 74.1	1,603 / 77.3
B65	Coder IQ1_M	2,185	948 / 60.1	1,018 / 62.0
B65	Coder IQ1_M	40,000	1,039 / 54.7	1,315 / 54.8
B65	Coder IQ1_M	128,000	861 / 53.6	1,246 / 54.4
B65	IQ2_XS	40,000	942 / 53.2	1,162 / 50.1

Decode moves with the text: the prompt kernels change the greedy output after a few dozen tokens, and MTP acceptance changes with it. With identical tokens, decode is unchanged.

Interleaved A/B on the B70 (same binary, --pcie-frac 0 --adapt-every 0 STRATA_PREFILL_MMQ=1, 2 rounds, prefill tok/s, old kernels → new):

    Coder at 2,185 tokens: 1,329 → 1,468 (+10%).
    Coder at 40,000: 1,562 → 1,885 (+21%).
    Coder at 128,000: 1,336 → 1,798 (+35%), with identical tokens.
    IQ2_XS at 40,000: 1,373 → 1,613 (+18%).

Failed:

    ple_parity: the Q2_0 shard-2 fixture is not on this machine. It fails the same way on main.

Not run:

    CUDA, HIP and Metal builds: nothing outside sycl/ and docs changed.
    Arc A-series, B580 and B60: no such cards here. The new kernels are plain SYCL (sub-group 32) plus oneMKL, so they build for every target, but their speed there is unmeasured. Both have off switches.
    A layer split across the B70 + B65 end to end: the engine runs above used one card each.
    Sampled (non-greedy) quality, beyond the first-token KL above.

Limits

    The two prompt kernels are not bit-identical to the warp kernels (FP32 summation order). Their switches restore the old kernels.
    The 8K threshold for the GEMM selection was measured on a B70 only.
    STRATA_SYCL_RUNNER=podman was exercised through strata-sycl.sh itself. native and distrobox: were not re-run on this branch.