Issues / #2535

#2535 [Bug]: Windows: a second NVIDIA GPU halves the largest pinned allocation, so the RAM-budget complement silently falls back to pageable memory (CUDA counterpart of #2485)

open · @KLOP09 · 2 comentários · No GitHub

BenchmarksNVIDIA / CUDAModels & quantsWindowsLinux

Descrição

### What happened

On Windows, adding a second NVIDIA card (a GTX 1650 that drives no display and is hidden from CUDA with `CUDA_VISIBLE_DEVICES=<RTX 5070 UUID>`) made the low-RAM resident mode stop page-locking its RAM copy. Same config, same engine, before and after:

```
# GTX 1650 enabled
FileExpertSource: allocating 14.00 GiB page-locked cache complement
FileExpertSource: locked resident cache complement ready: resident 14.00 GiB, pinned 0.00 GiB; page-locking refused (out of memory); locked 14335 MiB via working-set minimum + VirtualLock

# GTX 1650 disabled in Device Manager (nothing else changed)
FileExpertSource: mapped pinned cache complement ready: resident 14.00 GiB, pinned 14.00 GiB; page-locked and mapped
```

**Cause: the largest single pinned allocation is halved, the total is not.** Measured with the CUDA driver API (`cuMemHostAlloc`, `PORTABLE | DEVICEMAP`, the same flags as the complement), only the RTX 5070 visible, ~26 GB RAM available:

| | GTX 1650 enabled | GTX 1650 disabled |
|---|---|---|
| largest single allocation | **7.344 GiB** (7.375 fails at once) | **14.719 GiB** |
| total held in 0.5 GiB pieces | 14.0 GiB | 14.5 GiB |
| 4 + 4 + 4 GiB held at once | the third one fails | — |

It looks like two ~7.3 GiB regions instead of one ~14.7 GiB region. Logging out does not change it (7.344 / 14.0 GiB again). With both GPUs visible to CUDA, and with the `default` flags, 6 GiB is granted and 8 GiB refused, as with only the 5070 visible. Windows reports 15,945 MB of shared GPU memory per adapter (half of 31.1 GB). Before the 1650 went in, a 14 GiB complement was pinned on this PC and a 19.8 GiB one was refused.

**Why the engine is hit:** on Windows, `FileExpertSource::pin_cache_complement` (v0.1.44.1, the same file on main today — read, not run) still asks for the whole complement with one `cudaHostAlloc(Mapped | Portable)`. The stepped registration of #1250 is only switched on inside `#if defined(__linux__)`, and `STRATA_PARTIAL_PIN` registers a single prefix. So with the default settings the whole 14 GiB stays pageable, the prompt path streams it through the stager, and the decode's PCIe share is off. `STRATA_PIN_BY_NEED` is in `ArenaExpertSource::open` only.

**Suggested fix:** #2485 does this for the SYCL port (2 GiB chunks, `complement_at()`), as `GgufExpertSource::mirror` already does. A CUDA counterpart would fix it here, with two changes for this kind of limit:
- 2 GiB chunks would reach only about 12 GiB here (three per 7.34 GiB region), so chunks of 1 GiB or less (or one per layer, as the arena does).
- keep the chunks that were granted, hottest experts first, and lock only the rest, rather than all-or-nothing.

### What it costs (0.1.31, Qwen3.8-Flash-Next IQ2_XS, `--resident-budget-gib 14`, 64K, MTP, the same agentic tasks on two runs, logged in both times)

| medians, same task stage | complement pageable (1650 on) | complement pinned (1650 off) |
|---|---|---|
| decode, short-context tasks (replies ≥ 1,000 tokens) | 46.5 tok/s | 48.0 tok/s |
| decode, long-context repo tasks | 40.2 tok/s | 42.2 tok/s |
| new-session prompt read (8–10K tokens), short-context tasks | 229 tok/s | 233 tok/s |
| new-session prompt read (8–10K tokens), repo tasks | 230 tok/s | 254 tok/s |

So pinning is worth 3–5 % in decode and 2–10 % in prompt reading here: modest. The bigger reason is the next part.

**Possibly related to #879/#606 (not proven):** the all-NaN signature hit twice in the pageable run, 2 of 30 new sessions. Replies were `!` only, with the expert-cache hit at 99.1–99.9 % and decode at 74–127 tok/s. In the first event 14 requests in a row each ran to 32,000 tokens, until we stopped the server; the second was cut by a watcher after 61 s. Both times it struck the first request of a new session right after a ~31K-token session ended. That request read its 8.5–9.4K-token prompt from token 0. In the first event a dozen tokens came out normal and then the run collapsed; the fresh prompts that followed (also read from token 0) did the same, so the poisoned state was not only the conversation's K/V. With the complement pinned it was 0 of 233 new sessions before the 1650 went in. A replicate the next morning with the 1650 disabled gave 0 of 28 new sessions, 8 of them right after a ≥ 29K-token session. 2 of 30 against 0 of 28 alone is not significant (one-sided Fisher p ≈ 0.26). Against all pinned runs it is 0 of 261 (p ≈ 0.01, mixed conditions). The second GPU and a Windows update arrived on the same day, but pinned vs. pageable is the one engine-side difference we found.

**Workarounds (not yet measured):** disable the second card while Strata runs; `--resident-budget-gib 7` (fits one region); `STRATA_PARTIAL_PIN=1 STRATA_PARTIAL_PIN_GIB=7` (half pinned).

<details><summary>Probe script (Python, ctypes, CUDA driver API)</summary>

```python
import ctypes, os
cuda = ctypes.WinDLL('nvcuda.dll'); cuda.cuInit(0)
dev = ctypes.c_int(); cuda.cuDeviceGet(ctypes.byref(dev), 0)
ctx = ctypes.c_void_p(); cuda.cuDevicePrimaryCtxRetain(ctypes.byref(ctx), dev); cuda.cuCtxSetCurrent(ctx)
def alloc(n):
    p = ctypes.c_void_p()
    return p if cuda.cuMemHostAlloc(ctypes.byref(p), ctypes.c_size_t(n), ctypes.c_uint(3)) == 0 else None  # PORTABLE|DEVICEMAP
lo, hi = 1, 16 * 32                      # 1/32 GiB steps
while hi - lo > 1:
    mid = (lo + hi) // 2
    p = alloc(mid * 2 ** 25)
    if p: cuda.cuMemFreeHost(p); lo = mid
    else: hi = mid
held = []
while len(held) < 64:
    p = alloc(2 ** 29)
    if not p: break
    held.append(p)
print(f'largest single allocation {lo/32:.3f} GiB | total in 0.5 GiB pieces {len(held)*0.5:.1f} GiB')
for p in held: cuda.cuMemFreeHost(p)
```
</details>

### Strata version, GPU, OS

0.1.31 (the code path above is the same in v0.1.44.1 / main) · RTX 5070 12 GB (PCIe 5.0 x16) + GTX 1650 4 GB (PCIe 3.0 x4, no display, hidden from CUDA) · Windows 11 Pro 26H2 build 26300.9711 · NVIDIA 617.14 · HAGS on · Ryzen 7 9800X3D · 32 GB

No site

Links install, modelos, releases.