#6662 [Other] Setup picks model tier, size and image encoder from total VRAM, ignoring memory already held

open · @xvcezrqpbxtpzblrt · 0 コメント · GitHub で見る

Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows

本文

Thanks for the report. Please read the contributing guide first and search open and closed issues before filing.

**Write only what you saw.** Where a claim below is from reading code rather than a run, it is marked as static analysis in Reproduction status.

> This report is a static source review of the official tree. It was not reproduced on a running build; Reproduction status below says so explicitly.

### Area


[Other]  (anything else, including setup and installation)

### Desktop app, Terminal or source?


Terminal (`setup.py`); the desktop app runs the same setup code during install and first launch.

### What happened
Hardware detection and auto-configuration in `setup.py` size the model tier, the recommended model size and the image-encoder path from the GPU's **nominal total memory** (`nvidia-smi memory.total`). It never reads the memory that is actually free (`memory.free`). A card that reports 24 GB but is holding, say, 8 GB for the Windows desktop, a browser or another process is therefore treated as a full 24 GB card.

The same repository already treats this as a real failure mode on another path, and says so in its own source:

| Place | What it does |
|---|---|
| `setup.py:1868-1892` (`gpus()`) | Queries `index,name,memory.total,compute_cap,driver_version` and stores only `vram_gb = mem_gb` (total). No `memory.free`, no free-memory field. |
| `strata_control/split_engine.py:346` (`probe_nvidia_cards()`) | Adds `memory.free` to the *same* nvidia-smi call: `"--query-gpu=index,name,memory.total,compute_cap,memory.free"`. |
| `strata_control/split_engine.py:98-103` (`Card.room_mib`) | Decides with free memory, not size: `return max(0, min(self.vram_mib, self.free_mib) - FREE_RESERVE_MIB)`; falls back to size only when free is unknown. |
| `strata_control/split_engine.py:74-76` | Records the production case: a card must have memory *FREE*, "srv118, 2 x RTX PRO 6000 held to 8 GB free: the plan still said 'all-cards split'". |
| `tests/terminal/test_fix3_0212.py:225-235` | Tests that a 96 GB card held to 8 GB free is judged not to fit the experts, and that one busy card is enough to refuse the split. |

So the split path has free-memory protection and a regression test; the single-card auto-configuration path has neither. Four concrete decisions in `setup.py` still use the total-memory figure:

1. Model tier / recommended size — `setup.py:11348` `tier = tier_pick(ram, low_ram_vram(gpu))` and `setup.py:11424` `recommended_size(family, names, ram, low_ram_vram(gpu))`. For a discrete card `low_ram_vram()` returns the total: `setup.py:4582-4585` `return g.get("dedicated_gb", 0.0) if g.get("uma") else g["vram_gb"]`. `tier_pick()` (`setup.py:4621-4628`) adds `ram + vram - TIER_SYSTEM_RESERVE_GB`, where the only buffer is a fixed 5 GB (`setup.py:4618`, `TIER_SYSTEM_RESERVE_GB = 5.0`).
2. Image encoder — `setup.py:4064` `IMAGES_GPU_ENCODER_MIN_VRAM_GB = 23.5`, then `setup.py:4073-4080` `vram = float((gpu or {}).get("vram_gb") or 0.0)` / `small = 0 < vram < IMAGES_GPU_ENCODER_MIN_VRAM_GB` and `"yes"` selects the GPU encoder for a "24 GB" card even when little of it is free.
3. Card usability — `setup.py:2110-2120` `gpu_problem()` checks the card's total size, not its free memory.
4. Vision / plan helpers such as `setup.py:8695-8713` (`unsloth_vram_slack_mib`, `unsloth_small_card_plan`) are likewise driven by the total-memory figure.

This is not a hypothetical concern inside this codebase. `setup.py:1896-1897` already says of a display-attached card: "its desktop needs VRAM too, and a full expert cache beside it has crashed laptops" (#779). At runtime the project checks free memory again and warns about the same class of failure (`serve/server_parts/http_api.py:125-142`, `desktop_vram_note()`, #560/#516). Setup knows the desktop takes VRAM, and the split engine measures what is free — but the install-time auto-configuration bridges neither fact into its model/encoder choice.

### Smallest steps or command that reproduce it
Static reproduction from the reviewed source (no machine-specific state required to see the decision inputs):

1. Read `setup.py:1868-1892`: `gpus()` builds `vram_gb` from `memory.total` only; the card dict has no free-memory key.
2. On a Windows PC with a discrete NVIDIA card, put a real load on the desktop/GPU so the card is held mostly busy (multiple monitors, a hardware-accelerated browser, or another GPU process). Confirm with `nvidia-smi --query-gpu=index,memory.total,memory.free --format=csv,noheader,nounits` that free memory is far below total (the srv118 case was 96 GB total, ~8 GB free).
3. Run setup's hardware step (`python setup.py` / the app's install step) and watch the printed card line: it reports the total size, and the chosen tier / recommended size / encoder decision is unchanged by the low free value.
4. Compare with the split path on the same card: `strata_control.split_engine.probe_nvidia_cards()` reads `memory.free` and `Card.room_mib` refuses to count the busy card (covered by `test_fix3_0212.py:225-235`).
5. To confirm the downstream failure, start with the auto-selected config; the engine either runs out of VRAM for the expert/vision allocation or falls back to reading experts from the SSD tier.

### Reproduction status
Unconfirmed static analysis. The code paths, the fixed reserve, the free-memory split logic and its regression test were read directly from the reviewed upstream source tree. I did not run setup on a GPU whose memory was heavily held by a desktop, so the end-user symptom (failed start vs. SSD-tier slow path) is an expected consequence of the code, not a locally measured result. The fix is low-risk to verify because the source already contains the free-memory probe and a test harness.

### Strata version

0.2.19 · engine 0.1.65 (source tree under review, static analysis; not a running-build query)

### Graphics card, memory and system
Not measured in this static review — no claim about a specific PC is made. The affected class is any Windows setup with a discrete NVIDIA (or AMD) GPU whose desktop or other processes hold a meaningful part of the card's memory; the source itself documents such cases in `setup.py:1896-1897` (#779, #4641) and `strata_control/split_engine.py:74-76` (srv118). Multi-monitor desktops and shared machines are the common cases; a fixed 5 GB reserve cannot cover a 96 GB card held to 8 GB free.

### Model, size and settings
Auto-selected by setup, not hand-picked: the tier from `tier_pick()` / `recommended_size()` (using `low_ram_vram(gpu)`), and the image encoder from `flash_vision_encoder()` (GPU encoder at >= 23.5 GB nominal VRAM). No engine arguments or credentials were collected for this report.

### Logs
None collected — this is a static source review. The evidence is the source and its tests, quoted verbatim above with file and line numbers.

### Why this is worth fixing for everyone
The Windows desktop alone continuously consumes VRAM, which setup itself acknowledges in `gpu_problem`-adjacent code (#779, #4641: "its desktop needs VRAM too, and a full expert cache beside it has crashed laptops"). On the great majority of installations — Windows desktops with a discrete card — the nominal VRAM therefore overstates what the model can actually use. The only compensation in the tier rule is a fixed 5 GB (`TIER_SYSTEM_RESERVE_GB`), tuned for an ordinary desktop; it cannot represent a card held to a few GB free by multiple monitors, a browser, a game or a shared workload, as the project's own srv118 case shows. The result is an auto-configuration that recommends a model size and an encoder path the card cannot serve: the engine fails to start, or quietly serves experts from the SSD tier at a large speed cost, and the person sees no reason connected to the free memory they actually had.

This also closes a consistency gap inside the project: today the split engine and the runtime desktop note measure free memory, while the first configuration a user receives does not. Adding the free-memory column costs one extra field in an nvidia-smi call that setup already makes, and the same repository already has the parsing, the fallback-to-size behaviour and a regression test to copy.

### Root cause
`setup.py:1868-1892` (`gpus()`) treats `memory.total` as the card's usable memory and exposes no free-memory value. Every single-card auto-configuration decision consumes that number: `low_ram_vram()` (`:4582-4585`) returns it unchanged for a discrete card, `tier_pick()` / `recommended_size()` (`:4621-4653`, called at `:11348` and `:11424`) add it to RAM behind a fixed 5 GB reserve, and `flash_vision_encoder()` (`:4064`, `:4073-4080`) compares it with a 23.5 GB threshold. Meanwhile the correct measurement and decision logic already exist in `strata_control/split_engine.py:98-103, 346-365` and are exercised by `tests/terminal/test_fix3_0212.py:225-235, 263-275`; they were simply never wired into the setup path.

### Proposed fix
1. Add `memory.free` to the existing query in `gpus()` (`setup.py:1870-1871`) and store a `free_mib` field per card, using `None` for `[N/A]`/missing values — matching `strata_control/split_engine.py:346-364` (same call, one extra column, graceful fallback).
2. Introduce one "usable VRAM" helper used by the single-card decisions: `min(total, free) - a reserve`, falling back to the total when free is unknown, reusing the `Card.room_mib` idea and `FREE_RESERVE_MIB`. Feed it to `tier_pick()`, `recommended_size()`, `gpu_problem()` and the `flash_vision_encoder()` threshold, so the recommendation reflects memory the card can actually give the engine.
3. Keep the existing user-set reserve and the fixed `TIER_SYSTEM_RESERVE_GB` as a floor, so a momentarily idle card cannot cause an over-optimistic recommendation; treat free memory as an upper bound, not a replacement for the reserve.
4. When free memory is materially below the card's size, print one plain, non-blocking line naming the free value and saying the recommendation was based on it (recommend, never force — consistent with `setup.py` conventions), so the person can see why a smaller model or the CPU encoder was chosen.

Suggested acceptance check: on a card held to a small fraction of its nominal VRAM (e.g. 8 GB free on a 96 GB card, the srv118 case), setup reports the free value, its tier / size / encoder choice matches a card of that usable size, and the same choices are made when free memory is unknown.

### Duplicate search

Searched open+closed issues for `setup GPU memory total free`, `memory.free`, `vram free setup`, `model size recommendation`. Closest, none a duplicate: #779/#4641 (display-attached card needs VRAM), #560/#516 (runtime desktop VRAM note), #60 (Windows page-file limit). All concern related but distinct protections; the split path's free-memory probe (`split_engine.py:346`, with regression test) exists but is not wired into the single-card setup choices described here.

関連リンク

インストール・モデル・リリースへのサイト内リンク。