Issues / #403

#403 `--resident-budget-gib`: a budget clamped to available RAM then fails the safety check (0.1.31)

closed · @brenoperucchi · 4 comentários · No GitHub

BenchmarksNVIDIA / CUDAModels & quantsDocumentationWindows

Descrição


**Version:** 0.1.31 (tag `v0.1.31`), Windows 11 26200, RTX 5090 32 GB, Ryzen 9 5950X, 64 GB DDR4 at the time.
**Model:** Unsloth UD-Q4_K_XL, imported by hand following `docs/UNSLOTH_Q4.md`, with `"--resident-budget-gib", "40"` in the config.

## What happens

The engine clamps the budget, fills it, and then refuses to start with the same number:

```
FileExpertSource: --resident-budget-gib 40.00 is more than the RAM has room for (31.37 GiB available minus 4 GiB headroom): 27.37 GiB
FileExpertSource: RAM budget 27.37 GiB: 9384 of the 48.69 GiB of experts the GPU cache does not hold, by profile rank; the rest are read from the files
strata generate: CPU expert residency: FileExpertSource: resident complement 27.37 GiB exceeds available RAM (31.37 GiB) minus the 4 GiB safety headroom
```

The docs say "On 64 GB, 40. The engine clamps a budget larger than the RAM it finds free", but with that setting the
engine did not start here. An explicit smaller budget (`26`) works around it.

## Likely cause

I read `src/core/expert_source.cpp` but did not step through it in a debugger.

The budget branch (around line 1029) calls `available_memory_bytes(physical)` once and clamps `budget_bytes` to
`room = physical - headroom`. It then packs the ranked experts while `at <= budget_bytes`, so `bytes` can end up equal
to `room` or a few bytes under it. Right after that (around line 1069) `available_memory_bytes` is called **again**,
and `bytes > physical - headroom` is checked against the new reading. If available RAM drops by a few MB between the
two calls (the engine's own allocations, the OS file cache, another process), a budget clamped to the limit fails.
Both log lines show 31.37 because the drop is smaller than the two-decimal rounding.

Some ways to fix it, whichever you prefer: reuse the first reading for the second check when a budget was given; clamp
the budget to `room` minus a small margin such as 256 MiB; or use the clamped value as the hard check's limit.

## Repro

We saw it once. It should happen whenever the budget gets clamped and available RAM shrinks even slightly before the
second check, so it may be intermittent.

<details><summary>Start of <code>strata-ud-q4_k_xl.log</code> up to the failure</summary>

```
strata generate: PCIe probe: 28.2 GB/s host->device -> pcie_frac 0.55 (default 0.55)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: E:\strata-data\packs\ud-q4_k_xl experts (largest blob 3.99 MB), token embedding type 8 in mapped host memory (644 MiB)
strata generate: 1416 MiB of weights loaded from E:\strata-data\packs\ud-q4_k_xl (303 canonical tensors skipped: served natively)
strata generate: 301 native projection matrices, 2975.00 MiB of weights
strata generate: profile E:\strata-src\data\expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: PLE on, table 320001536 rows of E:\strata-data\models-unsloth\UD-Q4_K_XL\Qwen3.8-Flash-Next-UD-Q4_K_XL-00002-of-00004.gguf
strata mtp: draft layer loaded, 814 MiB of VRAM (experts 675, dense 111), files read in 0.47 s (1655 MiB/s)
strata generate: experts via mmap (--mmap-experts; the GGUF shards in place, no experts.bin)
strata generate: experimental native Q5_K head, 675430400 bytes
strata generate: expert cache auto: 24.00 GiB free, 700 MiB reserved (+281 MiB for the draft head) -> 6195 slots
strata generate: expert cache 7893 slots, 23.04 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
                 so a reply can differ slightly from a run without the cache (same quality:
                 bench/results/2026-09-27-cache-parity).
                 PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 7893 of 7893 slots from the profile; slot 0 verified
strata generate: routing-aware prefetch of the file tier on (the next layer's router)
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 15 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.31)
strata generate: token graph hit path: 7893 resident experts, decided on the device
FileExpertSource: --resident-budget-gib 40.00 is more than the RAM has room for (31.37 GiB available minus 4 GiB headroom): 27.37 GiB
FileExpertSource: RAM budget 27.37 GiB: 9384 of the 48.69 GiB of experts the GPU cache does not hold, by profile rank; the rest are read from the files
strata generate: CPU expert residency: FileExpertSource: resident complement 27.37 GiB exceeds available RAM (31.37 GiB) minus the 4 GiB safety headroom
```

</details>

No site

Links install, modelos, releases.