#6122 [Other] Auto drafting: a PC with several cards reports no card memory, so no default tier can apply
open · @marksmeltzer · 2 comentarios · En GitHub
AMD / HIPNVIDIA / CUDAModels & quantsWindows
Descripción
### The problem A PC with more than one graphics card cannot be described by either of the two signals that Auto drafting selection reads, so on that PC no default drafting tier can ever apply, whatever the cards are actually doing with the experts. Our machine is exactly that case: two RTX 5090, every expert resident, and Auto behaves for us as if nothing were known. Result we want: a way for the code to read "every expert is resident" on a machine with several cards, so a box like ours gets the same default a single card with the same behaviour gets. ### What we measured Hardware and build: two NVIDIA GeForce RTX 5090 (32 GiB each), driver 591.86, Windows 11 Pro 24H2 (build 26100), Strata 0.2.15 with engine 0.1.60-windows-x86_64-cuda-cuda13, model `uds-pri/Unsloth-Qwen3.8-Flash-Next-GGUF` `UD-Q4_K_XL` at 131072 tokens with a 7168 token vision projector, `--expert-cache auto`. From the engine's own start log: ``` expert cache 7953 slots (23.23 GiB) on card 0 expert cache 7287 slots (21.27 GiB) on card 1 ``` So both cards are holding the experts they were given; nothing is coming over PCIe for the layer the engine reads. The Control nevertheless picks the no-knowledge default for the drafter, the pack and the confidence. ### What we read in the code (static analysis, not tested by us) `drafting.py`: - `card_vram_mib(cfg, backend, archs)` returns `None` when the run names more than one card, beside the two cases it is designed for (a shared-memory chip, and a driver that does not answer). We read this as deliberate: there is no single card to report. - `slots_of()` therefore has no card memory to estimate from and, with `--expert-cache auto` and no explicit `--expert-cache N`, returns `None`. It never acts on what it cannot read, which we agree with. - Both default routes then decline: `tuned_applies` returns false for a `where: "resident"` entry, because a card whose memory is not known is not treated as resident; and `streaming_class` returns false, because `None` is never below a slot threshold. - A config-only override is not possible either, and that too is right: the person's `user_args` are merged and then the drafting layer rebuilds what its tables chose. We have not tested any change, so we mark all of the above as unconfirmed static analysis. The numbers in the section above are ours, produced on the machine named. ### An approach we can imagine, not a design we are asking for The engine already prints the answer it knows, and the server already reads the engine's start output when it builds the memory plan. A signal meaning "every expert is resident on every card in use", taken from what the engine reports rather than from one card's memory size, would let both routes see a multi-card box without the code having to guess what a list of cards means. Whether that is the right place for it, and what an entry in the selection table should say, is your call. We are not asking for a default for our exact quant on a multi-card machine. We are asking that our machine be visible to a mechanism that already exists. #5427 is the same area seen from the other side (a GPU lookup that found no card is made again, so a service started at boot does get DFlash2); this is what happens when no retry helps, because the answer genuinely cannot come from one card. ### Validation What passed: nothing, we changed no code. What failed: nothing. What we did not test: the behaviour of the three functions above (a reading of the source only); whether an explicit `--expert-cache N` on a multi-card box makes the routes see it - we did not try it; the behaviour of a single-card machine. ### Your task list (maintainers only) - [ ] No PR - report from reading only, with measured evidence from our box
En el sitio
Enlaces a install, modelos, releases.