#6426 [Engine] --layer-split auto with DFlash2 on two cards: the noisy PCIe ratio around 4.0 picks a placement with no expert room, and the drafter is dropped

open · @VocanicZ · 1 Kommentare · Auf GitHub

Setup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows

Beschreibung

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

On a two-card layer split with DFlash2, `--layer-split auto` sometimes picks a placement whose last card has no room
for a single expert slot. The engine then exits with `layer split, CUDA1 expert cache: no room`, and the server
restarts without a drafter:

```
[strata] filling the GPU's expert cache (400 experts, 0.51 GiB of VRAM) ...
[strata] Drafting: off (the cards were short of memory): starting again without a drafter (layer split, CUDA1 expert cache: no room)
```

The same machine, config and free memory give a different result from start to start. It happened on 3 of 7 layer-split
starts, on both 0.2.16 and 0.2.17 (engine 0.1.62); the other starts kept DFlash2. The free memory was identical every
time (`CUDA1 has 7.40 GiB free`). What changed was the PCIe probe:

| CUDA0 link GB/s | CUDA1 link GB/s | ratio | auto picked | CUDA1 room for experts at the cache step | result |
|---|---|---|---|---|---|
| 11.4 | 2.9 | 3.93 | layers 45-47 on CUDA1 | 0.00 GiB | no room, drafter off |
| 11.7 | 2.9 | 4.03 | layer 47 on CUDA1 | 0.04 GiB (28 slots) | DFlash2 on |
| 12.6 | 2.9 | 4.29 | layer 47 | 0.04 GiB | on |
| 10.7 | 2.9 | 3.69 | layers 45-47 | 0.00 GiB | no room, drafter off |
| 12.6 | 2.9 | 4.33 | layer 47 | 0.04 GiB | on |
| 10.6 | 2.9 | 3.66 | layers 45-47 | 0.00 GiB | no room, drafter off |
| 12.6 | 3.0 | 4.25 | layer 47 | 0.04 GiB | on |

Every start logged `no placement can start the prompt path on the figures here; choosing by decode speed alone`.

The ratio lands on both sides of `kLinkScaleMin` (4.0, `split_rules.hpp`). That decides whether CUDA1's miss cost is
weighted, which moves the pick between K=45 and K=47. Two things then let the K=45 pick fail at the cache step:
1. **The split search plans the last stage as larger than the cache step finds it.** The cache step subtracts DFlash2's
   after-cache bind (`df2_after`, about 160 MiB in serve), and the search does not book it. The search reported 143 MiB
   (K=45) and 294 MiB (K=47) left on CUDA1, while the cache step found 0.00 and 0.04 GiB.
2. **With the start gate off, the decode-speed fallback never checks that a later stage can hold an expert slot.**

Expected: auto picks a placement that loads, every time.

User impact: ~1.1 GB per card goes unused and decode is slower without the drafter. Which one you get depends on a
timing probe.

### Smallest steps or command that reproduce it

Two 8 GB cards (CUDA0 at x16, CUDA1 at a slower link). Use `--draft dflash2 --layer-split auto` with a tight
`--vram-reserve-later-mib`: ours is `--expert-cache 400 --max-context 16384 --kv int8 --vram-reserve-mib 2000
--vram-reserve-later-mib 5800`, Q2_0 pack. Start the server several times: some starts keep DFlash2 and some fall back.
Once the engine has `STRATA_SPLIT_LINK_MIN` (0.1.63+), setting it to 3.5 and then to 4.5 should force each side.
Workaround: `--layer-split 47`.

### Reproduction status

Reproduced locally

### Strata version

0.2.16-4 (58ee32277) and 0.2.17-3 (3142b66b9), engine 0.1.62 in both

### Graphics card, memory and system

RTX 3060 Ti 8 GB (CUDA0) + RTX 3070 Ti 8 GB (CUDA1), i9-10900, 128 GB RAM, Ubuntu 24.04.5, driver 595.99.02. Other GPU
processes start after Strata, which is why reserve-later is set.

---
A patch with CPU regression tests is in a draft PR linked below. It has not been compiled for CUDA or run on GPUs, and
the PR lists what still needs checking.

Mehr auf der Site

Links zu Install, Modellen, Releases.