Pull requests / #2486

#2486 sycl: Alchemist expert cache in segments, so borrowed prompt GEMM operands are not staged (A770 prompt +8-12%, output identical)

open · @aslater3 · 0 comentários · No GitHub

BenchmarksModels & quantsWindows

Descrição

## Title
sycl: Alchemist expert cache in segments, so borrowed prompt GEMM operands are not staged (A770 prompt +8-12%, output identical)

## Summary
The prompt path borrows the top expert-cache slots, 5.4-7.6 GiB into the ~8.6 GiB arena. oneMKL misreads a GEMM operand that lies more than 4 GiB into **one** allocation, so `staged_gemm` copied nearly every expert GEMM operand first. In an 8K prompt that is 24,576 staged GEMMs and about 200 GB of device copies.

On Alchemist, the arena is now allocated as equal physical segments behind one reserved address range (`sycl_ext_oneapi_virtual_mem`). The rest of the engine still sees one contiguous range. The arena is not registered for staging, because the same GEMMs read right in a segmented range at any offset.

## What changed
- `sycl/include/strata/sycl_queue.hpp`: `malloc_device_segmented` / `free_device_segmented` and `arena_segmented_wanted`. Segments are 512 MiB (whole 2 MiB pages, all equal: Level Zero refused a smaller last segment).
- `sycl/src/core/expert_cache.cpp`: `ExpertCache::open` takes that path on Alchemist, and `close()` frees it.
- `sycl/src/prefill/gemm.dp.cpp`: a comment.
- `sycl/probe/a770/gemm_far_probe.cpp`: the probe that places the misread.
- Switches: `STRATA_ARENA_SEGMENTED=0` gives one allocation as before; `=1` opts in other devices. `STRATA_ARENA_SEGMENT_MIB` sets the segment size.
- No virtual-memory support or any failure falls back to the previous allocation.

## Extra Notes
**Probe** (`gemm_far_probe`, A770, xe, compute runtime 26.05, oneAPI 2026.1):
- One 8 GiB `malloc_device`, T = 7987: X past 4 GiB gives a wrong last partial tile; W and X past 4 GiB gives nearly every element wrong.
- Physical segments behind one range: every case reads right. This held for 0.25, 0.5 and 2 GiB segments, fp16 and bf16, with W, X or Y anywhere, and across segment boundaries.

**Engine** (Coder IQ1_M, `--resident-experts`, learned24, `--prefill-borrow`, spec 4, greedy, `STRATA_PF_XMX=2`), same-day A/B against itself with `STRATA_ARENA_SEGMENTED=0`:

| prompt | segmented | one allocation | output |
|---|---|---|---|
| 8K | 678.8 tok/s prefill, 27.17 decode | 627.4, 27.31 | identical ids |
| 32K | 720.7 tok/s prefill, 22.90 decode | 649.1, 22.98 | identical ids |

**Default path elsewhere.** Battlemage and every other device are byte-identical. The gate is `intel_gpu_gen(dev) == IntelGpuGen::Alchemist`.

Not tested: A750 / A580 (i915), Windows, `--vram-elastic`, a layer split, prompts without borrowing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

No site

Links install, modelos, releases.