Pull requests / #2485

#2485 sycl: pin the RAM-resident expert copy in 2 GiB chunks when one pinned allocation is refused (Arc A770; prompt DMA for every expert, host staging 3.3 s -> 0.16 s)

open · @aslater3 · 0 comentários · No GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Descrição

## Title
Issue: #1545, #1606 (Arc A770)

## Summary
With `--resident-experts`, the RAM copy of the experts the VRAM cache does not hold is requested as **one** pinned host allocation. On an Arc A770 (DG2, `xe`) every single pinned allocation of 4 GiB or more is refused, even with `UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1`, while 2 GiB pieces are not. So the 11-18 GiB copy fell back to `malloc` + `mlock`. The CUDA engine registers that fallback (`cudaHostRegister`); the SYCL port could not (DPCT1027), so the GPU's copies from it ran at the pageable rate and the prompt path streamed every expert through the host stager instead of DMA.

This pins the copy in 2 GiB chunks when the single allocation is refused, as `GgufExpertSource::mirror` already does for the `--stream-experts` mirror: the blobs are laid out so none crosses a 2 GiB boundary and `complement_at()` maps an offset to its chunk. Cards whose single pinned allocation succeeds take exactly today's path.

## What changed
- `sycl/src/core/expert_source.cpp` (`FileExpertSource::pin_cache_complement`): after the single `malloc_host` is refused, re-lay the offsets and allocate `ceil(bytes / 2 GiB)` pinned chunks; on any refusal, free them and keep the pageable fallback. `resident_blob`, `device_alias`, the copy-in loop and the adaptive tier's writes go through `complement_at()`. The exchange-buffer rotation (one contiguous arena) is not offered with chunks. `close()` frees the chunks.
- `sycl/include/strata/core/expert_source.hpp`: `complement_chunks_`, `complement_at()`.
- `STRATA_RESIDENT_CHUNKED=0` keeps the pageable fallback.

## How it was checked
Arc A770 16 GB, `xe`, Linux 7.0, oneAPI 2026.1 (JIT), Ryzen 7 3800X, 30 GB RAM. Coder IQ1_M, `--resident-experts --expert-cache auto --max-context 131072 --kv int8 --kv-resident 20480 --spec 4 --mtp`, `STRATA_VERIFY_STEPPED=1`, `UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1`, an 18 GiB resident copy, same day, with vs without (`STRATA_RESIDENT_CHUNKED=0`):

| | pageable (today) | chunked pinned |
|---|---|---|
| resident copy | 18.03 GiB, pinned 0 | 18.04 GiB, pinned 18.04 GiB in 10 chunks |
| experts the prompt streams by DMA (8K) | 0 of 9,352 | 9,352 of 9,352 |
| host staging during the prompt, 8K / 32K | 3,307 / 13,466 ms | 163 / 626 ms |
| lent slots refilled after the prompt | 583-640 ms | 266 ms |
| adaptive tier swaps (decode) | 3.8-4.0 ms/round | 0.86-0.90 ms/round |
| prompt 8K / 32K | 669.8 / 668.2 tok/s | 680.4 / 677.5 tok/s |
| decode 8K / 32K (with #2207) | 30.88 / 26.87 tok/s | 31.04 / 27.22 tok/s |

Output ids identical in every pair (8K `72b762d5b2a3`, 32K `eb3432fea981`). The end-to-end gain is small because on this card the prompt is GPU-bound (the expert GEMMs), but the stager's ~3 s of host copying per 8K prompt and most of the tier's copy time are gone. This branch alone (`a0116b27`-era `main` = v0.1.44.1): 678.2 tok/s at 8K, all 9,352 by DMA, same ids.

The probe for the allocation limit: `pinned_probe` in the A770 report (#2216) / our fork's `sycl/probe/a770/` (one `malloc_host` of 4.00 GiB fails, 7 x 2 GiB succeed, 17.3 GB/s from pinned against 4.2 GB/s from pageable).

Not tested: i915 (the A750's arena path does not use this function's pinned branch the same way), Battlemage (its single allocation succeeds, so this path does not run), Windows. CUDA/HIP: SYCL-only files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

No site

Links install, modelos, releases.