#6586 [Engine] fix strata-inkling --gpus 2 null deref and VRAM buffer overrun in ep_dual_gpu

open · @nmqx · 0 commentaires · Sur GitHub

BenchmarksAMD / HIPNVIDIA / CUDAWindowsLinux

Description

## Problem and result
Running `strata-inkling --bench --native Inkling-2-Mini-SQ-20.strata --gpus 2` on `v0.2.19` crashed immediately during prefill with `0xC0000005` (`STATUS_ACCESS_VIOLATION`) on both `--parallel 1 --prompt-len 512` and `--parallel 8 --prompt-len 512` (2× RTX 3090, `sm_86`, non-P2P pinned mode).

Two underlying bugs in `engine/src/inkling2/ep_dual_gpu.cu` caused this:
1. **Null-pointer dereference in `dispatch_moe_ep`**: Experts placed in GPU VRAM (`3072/3072` resident) have no host copy (`host_descs_[key].body == nullptr`), yet `if (de.resident && jb.m <= kMaxParallelSlots)` sent any resident expert receiving `> 64` routed tokens during prefill (`chunk = 512`) into `host_jobs`, where `eval_inkling2_expert_batch(hd.body, ...)` dereferenced `nullptr`.
2. **VRAM buffer overrun on `d_dense_x_`**: `d_dense_x_` was sized for `kMaxParallelSlots * c_.hidden` (`64 * 2304`), whereas `attn.wo_ud.weight` passes `K = n_head * head_dim = 3072`, overflowing `d_dense_x_` by up to `196 KiB` for `M > 48`.
3. **Teardown device safety**: `cleanup()` freed GPU0 dense buffers without setting `cudaSetDevice(gpu_ids_[0])` first.

After this fix, `--selfcheck --gpus 2` and `--bench --gpus 2` complete cleanly with `rc=0` on both `--parallel 1` and `--parallel 8`.

## Related discussion
none

## Approach
Minimal, self-contained fix in `engine/src/inkling2/ep_dual_gpu.cu` (`1 file, +49/-20`):
- Route all `de.resident` experts to `gpu_jobs[de.owner_gpu]` regardless of `jb.m`, and chunk GPU expert launches into slices of `<= kMaxParallelSlots` (`64`) tokens concurrently across GPU0/GPU1 (with a defensive `hd.body == nullptr` guard in `host_jobs`).
- Allocate `d_dense_x_` for `max_tok * max(hidden, n_head * head_dim)` and `d_dense_y_` for `max_tok * max(vocab_unpadded, n_head * head_dim, hidden, n_experts)`, and chunk `gpu_dense_proj` in `kMaxParallelSlots` slices on GPU0.
- Call `cudaSetDevice(gpu_ids_[0])` in `cleanup()` before freeing GPU0 buffers.

Zero behavior or ABI changes for single-GPU `ink_cuda_backend` or non-CUDA backends.

## Validation

**Passed:**
- Build (`v0.2.19` + `6f7d61eb6`, Windows 11 x64, MSVC 19.50, CUDA 13.3, `CMAKE_CUDA_ARCHITECTURES=86`, Ninja): `strata-inkling.exe` clean `[27/27]` link, zero warnings in `ep_dual_gpu.cu`.
- `CUDA_VISIBLE_DEVICES=0,1 strata-inkling.exe --selfcheck --gpus 2` (2× RTX 3090, `sm_86`, non-P2P pinned): `[selfcheck] PASS` (`rc=0`, `0.81s`).
- `CUDA_VISIBLE_DEVICES=0,1 strata-inkling.exe --bench --native Inkling-2-Mini-SQ-20.strata --gpus 2 --vram-reserve-mib 1400 --parallel 1 --prompt-len 512 --gen 256` (exact crashing command): `rc=0`, `wall=273.95s`, `prefill=5.8 ppts (88.39s) | decode=1.5 tps agg (1.5 tps/slot, 169.95s)` (previously `rc=3221225477` / `0xC0000005`).
- `CUDA_VISIBLE_DEVICES=0,1 strata-inkling.exe --bench --native Inkling-2-Mini-SQ-20.strata --gpus 2 --vram-reserve-mib 1400 --parallel 8 --prompt-len 128 --gen 8` (`B=1024` prefill stressing `jb.m > 64` across 8 slots): `rc=0`, `wall=104.44s`, `prefill=12.2 ppts (83.69s) | decode=6.2 tps agg (0.8 tps/slot, 10.30s)`.

**Failed:**
none observed

**Not run:**
- Linux CUDA, NVLink P2P (`p2p=on`), HIP, SYCL, and Metal builds/runs (tested on Windows 11 + 2× RTX 3090 PCIe pinned fallback only).
- Hybrid CPU-doorbell offload (`--cpu-threads > 0` with partial VRAM residency) end-to-end bench (covered only by `--selfcheck` doorbell roundtrip).

## Limits
- This PR strictly fixes the crash, buffer overrun, and teardown device selection in `ep_dual_gpu.cu`. It does not move `ep_dual_gpu.cu`'s CPU attention/ShortConv/AttnRes loops onto GPU, so single-GPU `ink_cuda_backend` remains faster when the model fits on one GPU.

Sur le site

Liens install, modèles, releases.