#6458 [Engine] STRATA_FILL_STAGED: copy mapped blobs to the GPU via pinned buffers

open · @kk1987 · 0 commentaires · Sur GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Description

## Problem and result

On an RTX Spark N1X (HP OmniBook X 14, sm_121, 120 GB unified memory, driver 620.70.06, CUDA 13.4) with the NVFP4-QAD pack and `--mmap-experts`, a pageable `cudaMemcpy` whose source is a file mapping gets slower with every new file page it touches. Outside the engine, 8,000 random 2.7 MB copies from the mapped `experts.bin` into a `cudaMalloc` run at ~500 MB/s for the first 500 and ~40 MB/s at the end; the same copies from anonymous memory, or repeated from one mapped range, stay at ~50 GB/s.

That gives two symptoms. A start takes ~42 min: the profile fill ran at 31 MB/s (2,184 s) and the 300 native dense matrices took 345 s. And long prompts stall: the batched path refills its lent slots from the mapping at the end of the prompt, the refill outlasts the 60 s no-progress watchdog, and the engine is ended a few tokens before the end of the prompt. Three cores of those aborts all have the main thread in `ExpertCache::fill_slot_queued` → `cudaMemcpyAsync` → `cuMemcpyHtoDAsync_v2`. This is most likely what #5678 reports on another N1X laptop.

With `STRATA_FILL_STAGED=1` the fill takes 44-48 s, the dense upload 1.4 s, a start 58-73 s, and a full probe (short prompts, 16K-34K agent turns cold and warm, a 29K prompt, 2K/16K/64K/128K depth) ran without a stall or restart.

## Related discussion

#5678

## Approach

`STRATA_FILL_STAGED=1` copies those sources through pinned buffers: `fill_slot_blocking` through the existing blocking staging buffer (until now HIP only), `fill_slot_queued` / `fill_slot_queued_parts` through a ring of eight pinned buffers whose events say when a buffer can be reused (the copies stay queued on the legacy stream, so `sync_queued` is unchanged), and `NativeDense`'s two uploads through a new `copy_to_device_staged` in 64 MiB pieces. It is opt-in and compiled only into the Linux CUDA engine (not HIP, the gfx906 compat build, Metal or SYCL); without the variable every copy is the pageable one it was. The register and SETTINGS.md have the new entry, and `docs/DGX_SPARK_LINUX.md` a short section. I did not make it the default for integrated GPUs because I could not measure a DGX Spark.

## Validation

**Passed:**
- `BUILD_VISION=0 engine/spark/build_engine.sh` (CUDA 13.4.92, aarch64) on this commit and on the same patch over v0.2.17 (12d491da4).
- Hardware, the patch over v0.2.17: `STRATA_FILL_STAGED=1`, NVFP4-QAD pack converted from nvidia/Qwen3.8-Flash-Next-FP4-Local, setup's GB10 args plus `--max-context 262144 --short-read 160`. `pre-filled 24576 of 24576 slots from the profile in 44.0 s (1543 MB/s); slot 0 verified`, `300 native projection matrices, 2948.44 MiB of weights, in 1.4 s`. A probe of short zh/code prompts, agent turns at 16K/25K/34K cold and warm, one 29K prompt and depth 2K/16K/64K/128K: no `no progress` stop, no restart. Cold prompt 1245 / 1432 / 1260 tok/s at 16K / 25K / 34K, 1572 tok/s at 128K; decode 41-55 tok/s.
- Hardware, this commit: start in 73 s with the same fill and dense lines, a 60-token and an 18K-token request answered.
- Before, same machine and pack, the released 0.1.63 engine: start ~42 min, the 29K prompt stalled at 29,180 of 29,185 tokens and the watchdog ended the engine. The self-built engine with only the profile fill staged stalled three times in one probe (tokens 74, 16254, 25423), all in the refill.
- `tools/settings_register.py --check`, `tools/check_links.py`.

**Failed:** none observed.

**Not run:** HIP, Metal, SYCL and Windows builds (the new code is behind `__linux__` and excluded for those, but I have not compiled them); a DGX Spark; the default path on this machine after the change (it is the unchanged code).

## Limits

Why the driver's pageable path decays is not known; this works around it. Prompts of 80-100 tokens on NVFP4 still cost ~1.5 s on the batched path, ~79% of it dequantization on the FP16 path; `--short-read 160` helps there, but that is a run setting and not part of this PR.

Sur le site

Liens install, modèles, releases.