Pull requests / #1003

#1003 resident RAM mode: release each layer's mapped experts as it is copied (Windows)

closed · @xdbxdbx · 0 comments · View on GitHub

BenchmarksNVIDIA / CUDAModels & quantsWindowsLinux

Description

## The problem

On Windows, starting Strata in resident RAM mode (`--resident-experts`) briefly needs about twice the RAM
it ends up using. On a 64 GB PC this can take available RAM down to almost nothing during the load, and a
start can fail.

Why: at startup the engine copies the experts the GPU does not hold into RAM. It reads them through a
file mapping, and every page it reads that way stays counted against the process until it is released.
Today they are only released once, after all 48 layers are copied. So for a while the PC holds the whole
complement twice: once in the copy, and once in the mapped pages it was copied from.

Linux does not have this problem: it already releases each layer's pages as soon as that layer is copied.

## The fix

Do the same on Windows: after each layer is copied, release that layer's mapped pages with the same
`VirtualUnlock` call the function already uses at the end. The pages are not thrown away - Windows keeps
them cached and counts them as available - they just stop being held by Strata.

It covers both expert layouts: the GGUF read in place, and experts.bin.

## Measured

RTX 4090 24 GB, 64 GB RAM, Windows 11, IQ2_XS, `--resident-experts`, 2 slots x 256K context, model on a
SATA SSD. Memory sampled every half second through the load:

| | before | after |
|---|---|---|
| lowest available RAM during startup | 0.2 GB | 16.9 GB |
| Strata's peak memory use | 41.9 GB | 25.8 GB |
| load time | 114 s | 112 s |

Speed after startup is unchanged (2 slots: 37-38 tok/s each after, 39-40 before).

Before the fix, one start of this config failed during the copy ("page-locking refused (out of memory)",
then "verify: mapped staging allocation failed").

## Related

These touch the same code but solve different problems: #818 (trims memory after uploads, opt-in), #833
(closes the experts.bin mapping after the copy), #848 (resident mode across several GPUs, also edits
`pin_cache_complement`).

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.