#5546 [Engine] One card or two: print the prompt-length crossover at start-up

open · @Norhiro · 9 Kommentare · Auf GitHub

Server & APIMulti-GPUNVIDIA / CUDASecurityDocumentationLinux

Beschreibung

Split out of #5265 at @udqe's suggestion. #5265 stays about choosing K inside a fixed two-card placement; this issue is about the decision before that: whether a request should use two cards at all.

**The problem.** On a two-card machine, the pair loses to one card on short prompts and wins on long ones. Where it crosses differs a lot from box to box, and Strata does not tell the user. Today the search goes to one card only when a card is dropped, so the pair is used for every request, including the short ones where it is slower.

**Two crossovers, not one** (udqe's framing on #5265):
1. **Cards into the request:** a fixed per-request cost of the pair against its streaming gain. This is decided before any K matters. This issue is about this one.
2. **Layers onto which card:** min-max stage fill against steady-state balance, inside a fixed card count. That stays in #5265.

Pricing the first does not need the byte term from #5265, so the two can be reviewed separately.

**What is measured so far.**

| Box | Links | Short prompt, pair vs one card | Where the pair starts to win |
|---|---|---|---|
| This rig: RTX 3060 (x16) + RTX 5060 Ti (x8), Linux, engine 0.1.57, NVFP4-GPTQ, K=18 (#5265) | unequal, 12.3 / 6.9 GB/s H2D | TTFT 2.61 s vs 1.42–1.51 s (+1.1 s) | before 4K (4K: 8.30 s vs 9.75–10.25 s) |
| #5376: 2x RX 9070, UD-Q4_K_XL | equal | TTFT 0.96 s vs 0.60 s at ~80 tokens (+60%) | between 80 and 4K tokens (4K: −7.7%) |
| #5187: split engine vs stock layer split | — | — | about 730 generated tokens per request (one run, per its author; in `docs/SPLIT_ENGINE.md`) |

On this rig the short-prompt cost also moves with K (80-token read: K=12 2,677 ms → K=30 1,994 ms, 5060 Ti alone 1,353 ms), so on an unequal pair part of it belongs to crossover 2. What is left at the far end of that line is small here, but that is an extension of a line, not a measurement.

**Proposal (print, do not choose).**
- When the engine starts on two or more cards, print one line with the predicted crossover: "one card is faster below about N prompt tokens on this machine (pair TTFT X s, one card Y s at 80 tokens)". The numbers could come from the link probe the engine already runs at start-up plus the per-stage prompt model from #5265; I have not checked how much of that is available at that point.
- No automatic switching. The user (or the serve layer) can choose with what is already there.
- Routing short requests to one card at run time would be a later, separate step, and only if the printed numbers turn out to be reliable across boxes.

**Measurement protocol, so boxes can be compared.** Fresh (never-reused) prompts at about 80 / 256 / 1K / 2K / 4K / 8K / 16K tokens, n=5 per point, T=0, TTFT taken as the client-clock time to the first non-empty delta, one card versus the pair on the same engine build. Report medians with min–max and the link speeds. @udqe offered to run this on 2x RTX 4090 (equal links, 26.4 GB/s each); I will run it on this pair as well, so we have one equal-link and one unequal-link curve.

Questions for maintainers: is a start-up line like this acceptable in scope? Is there a better place than the engine for it (for example `strata doctor` or the serve layer)?

(ai-assisted; the numbers from this rig are from my own runs, the others are quoted from the linked threads)

Mehr auf der Site

Links zu Install, Modellen, Releases.