#6122 [Other] Auto drafting: a PC with several cards reports no card memory, so no default tier can apply

open · @marksmeltzer · 2 comments · View on GitHub

AMD / HIPNVIDIA / CUDAModels & quantsWindows

Description

### The problem

A PC with more than one graphics card cannot be described by either of the two signals that Auto drafting
selection reads, so on that PC no default drafting tier can ever apply, whatever the cards are actually doing
with the experts. Our machine is exactly that case: two RTX 5090, every expert resident, and Auto behaves for
us as if nothing were known.

Result we want: a way for the code to read "every expert is resident" on a machine with several cards, so a
box like ours gets the same default a single card with the same behaviour gets.

### What we measured

Hardware and build: two NVIDIA GeForce RTX 5090 (32 GiB each), driver 591.86, Windows 11 Pro 24H2 (build
26100), Strata 0.2.15 with engine 0.1.60-windows-x86_64-cuda-cuda13, model
`uds-pri/Unsloth-Qwen3.8-Flash-Next-GGUF` `UD-Q4_K_XL` at 131072 tokens with a 7168 token vision projector,
`--expert-cache auto`.

From the engine's own start log:

```
expert cache 7953 slots (23.23 GiB) on card 0
expert cache 7287 slots (21.27 GiB) on card 1
```

So both cards are holding the experts they were given; nothing is coming over PCIe for the layer the engine
reads. The Control nevertheless picks the no-knowledge default for the drafter, the pack and the confidence.

### What we read in the code (static analysis, not tested by us)

`drafting.py`:

- `card_vram_mib(cfg, backend, archs)` returns `None` when the run names more than one card, beside the two
  cases it is designed for (a shared-memory chip, and a driver that does not answer). We read this as
  deliberate: there is no single card to report.
- `slots_of()` therefore has no card memory to estimate from and, with `--expert-cache auto` and no explicit
  `--expert-cache N`, returns `None`. It never acts on what it cannot read, which we agree with.
- Both default routes then decline: `tuned_applies` returns false for a `where: "resident"` entry, because a
  card whose memory is not known is not treated as resident; and `streaming_class` returns false, because
  `None` is never below a slot threshold.
- A config-only override is not possible either, and that too is right: the person's `user_args` are merged
  and then the drafting layer rebuilds what its tables chose.

We have not tested any change, so we mark all of the above as unconfirmed static analysis. The numbers in the
section above are ours, produced on the machine named.

### An approach we can imagine, not a design we are asking for

The engine already prints the answer it knows, and the server already reads the engine's start output when it
builds the memory plan. A signal meaning "every expert is resident on every card in use", taken from what the
engine reports rather than from one card's memory size, would let both routes see a multi-card box without the
code having to guess what a list of cards means. Whether that is the right place for it, and what an entry in
the selection table should say, is your call.

We are not asking for a default for our exact quant on a multi-card machine. We are asking that our machine be
visible to a mechanism that already exists. #5427 is the same area seen from the other side (a GPU lookup that
found no card is made again, so a service started at boot does get DFlash2); this is what happens when no
retry helps, because the answer genuinely cannot come from one card.

### Validation

What passed: nothing, we changed no code.

What failed: nothing.

What we did not test: the behaviour of the three functions above (a reading of the source only); whether an
explicit `--expert-cache N` on a multi-card box makes the routes see it - we did not try it; the behaviour of
a single-card machine.

### Your task list (maintainers only)

- [ ] No PR - report from reading only, with measured evidence from our box

Related on this site

Quick links to related pages.