Issues / #2014

#2014 SYCL resident RAM mode: resident_stage_swaps stages the exchange without copying the evicted expert back

open · @kuhlsnu · 0 comentarios · En GitHub

Server & APINVIDIA / CUDA

Descripción

**The SYCL port of `resident_stage_swaps` has no device-to-host copy where the CUDA version has one.** I found this reading the code while porting an exchange check; I have no Intel GPU, so nothing here ran.

CUDA, `src/program/generate.cpp:292-299` (7695859a):

```cpp
const SwapHome home = locate(s.layer);
const strata::core::OnDevice on(home.dev);
if (const cudaError_t e = cudaMemcpyAsync(src.exchange_buffer(q), home.cache->device_slot(slot),
                    (size_t) strata::kernels::cpu::expert_layout().blob_bytes(s.layer), cudaMemcpyDeviceToHost,
                    home.stream); e != cudaSuccess) {
```

SYCL, `sycl/src/program/generate.cpp` (`resident_stage_swaps` at :318), same spot:

```cpp
const SwapHome home = locate(s.layer);
const strata::core::OnDevice on(home.dev);
/*
DPCT1124: cudaMemcpyAsync is migrated to asynchronous memcpy API.
...
*/

bool seen = false;
```

The migration comment is there but the memcpy itself is not, and the swap is still staged (`staged.push_back`, then `stage_exchange`). If this path runs, the RAM copy would take whatever is already in exchange buffer `q` as the evicted expert, so a later request routed to it on the CPU reads wrong weights, with no error.

**What I could not check:**
- whether SYCL ever reaches this function (resident RAM mode with adaptive swaps on a SYCL build)
- whether anything else fills the exchange buffer first

If it is reachable, this could be related to the silent wrong answers in #1484, though that one points at the expert kernels. A quick check: run resident RAM mode with adaptive swaps on SYCL and compare a swapped expert's RAM bytes with the model file.

En el sitio

Enlaces a install, modelos, releases.