贡献 / #1669

#1669 sycl: pin the PCIe probe's source buffer (fixes multi gpu layer split)

open · @maxious · 0 评论 · 去 GitHub 看

Server & APIMulti-GPUAMD / HIP

说明

Relates to #485 (the probe's reading sets `pcie_frac`). Not a fix for #1054: that startup wedge on a 2-card split has a different mechanism (the host-mirror fill).

## Summary
`probe_pcie_h2d_gbps` copies from a pageable 256 MiB source. On two Arc Pro B60s with `--layer-split auto` the copy engine faults at a fixed VA in the driver's pool region (`0xffffc001ffa10000`): hundreds of `bcs VM_NOT_FOUND` faults at FaultLevel 4 (the top-level page-table entry is absent, so the VA range was never mapped in that VM), the load wedges, and no window ever starts. Measured 755 and 747 faults in two runs; intermittent on fresh boots, and observed on a layer split (a single-card start was not exercised).

## What changed
`sycl/src/program/generate.cpp`, `probe_pcie_h2d_gbps`: allocate the source with `sycl::malloc_host` and free it with the matching `sycl::free`. The function's own comment already says it times "copies from pinned host memory, as the expert arena's reads are" - a pageable source contradicted that, and is what the driver stages through its bounce path.

This moves a number the default path uses: the probe feeds `pcie_frac_for_gbps`, so the share of missed experts the GPU reads over PCIe follows the reading. Same build otherwise, one variable changed:

| source | probe | `pcie_frac` | runs |
|---|---|---|---|
| pageable (before) | 13.8 GB/s, best of 13.5-13.8 over 4 bursts | 0.38 | nine |
| pinned (after) | 14.4 GB/s, best of 14.4 over 4 bursts | 0.40 | two |

Both cards sit in x8 slots; above 20 GB/s the share is unchanged, so no x16 machine's default moves.

The buffer is allocated, used and freed inside the probe, so the page-locked footprint is transient (256 MiB for ~0.1 s). On a device without host USM the allocation throws, `DPCT_CHECK_ERROR` catches it, the probe returns `-1.0` and the caller keeps the default share - no new hard failure.

## Extra Notes
Two consecutive loads reached the window cleanly with the pin (2/2); the run log shows the 256 MiB host USM allocation the pageable version never made. Not tested: the AMD SYCL path, a single-card start, an x16 slot. Also removes the now-stale DPCT1124 note above the timed memcpy in the same function (that call waits explicitly).

All the split-fault instrumentation (guard watch, stage stamps and the `STRATA_STAGE_WAIT` serial gate, terminate dumps, coredump grabbers) lives on https://github.com/maxious/Strata_SYCL/tree/scratch/split-fault-instruments

本站相关内容

相关页面的快捷入口。