#6426 [Engine] --layer-split auto with DFlash2 on two cards: the noisy PCIe ratio around 4.0 picks a placement with no expert room, and the drafter is dropped

open · @VocanicZ · 1 commentaires · Sur GitHub

Setup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows

Description

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

On a two-card layer split with DFlash2, `--layer-split auto` sometimes picks a placement whose last card has no room
for a single expert slot. The engine then exits with `layer split, CUDA1 expert cache: no room`, and the server
restarts without a drafter:

```
[strata] filling the GPU's expert cache (400 experts, 0.51 GiB of VRAM) ...
[strata] Drafting: off (the cards were short of memory): starting again without a drafter (layer split, CUDA1 expert cache: no room)
```

The same machine, config and free memory give a different result from start to start. It happened on 3 of 7 layer-split
starts, on both 0.2.16 and 0.2.17 (engine 0.1.62); the other starts kept DFlash2. The free memory was identical every
time (`CUDA1 has 7.40 GiB free`). What changed was the PCIe probe:

| CUDA0 link GB/s | CUDA1 link GB/s | ratio | auto picked | CUDA1 room for experts at the cache step | result |
|---|---|---|---|---|---|
| 11.4 | 2.9 | 3.93 | layers 45-47 on CUDA1 | 0.00 GiB | no room, drafter off |
| 11.7 | 2.9 | 4.03 | layer 47 on CUDA1 | 0.04 GiB (28 slots) | DFlash2 on |
| 12.6 | 2.9 | 4.29 | layer 47 | 0.04 GiB | on |
| 10.7 | 2.9 | 3.69 | layers 45-47 | 0.00 GiB | no room, drafter off |
| 12.6 | 2.9 | 4.33 | layer 47 | 0.04 GiB | on |
| 10.6 | 2.9 | 3.66 | layers 45-47 | 0.00 GiB | no room, drafter off |
| 12.6 | 3.0 | 4.25 | layer 47 | 0.04 GiB | on |

Every start logged `no placement can start the prompt path on the figures here; choosing by decode speed alone`.

The ratio lands on both sides of `kLinkScaleMin` (4.0, `split_rules.hpp`). That decides whether CUDA1's miss cost is
weighted, which moves the pick between K=45 and K=47. Two things then let the K=45 pick fail at the cache step:
1. **The split search plans the last stage as larger than the cache step finds it.** The cache step subtracts DFlash2's
   after-cache bind (`df2_after`, about 160 MiB in serve), and the search does not book it. The search reported 143 MiB
   (K=45) and 294 MiB (K=47) left on CUDA1, while the cache step found 0.00 and 0.04 GiB.
2. **With the start gate off, the decode-speed fallback never checks that a later stage can hold an expert slot.**

Expected: auto picks a placement that loads, every time.

User impact: ~1.1 GB per card goes unused and decode is slower without the drafter. Which one you get depends on a
timing probe.

### Smallest steps or command that reproduce it

Two 8 GB cards (CUDA0 at x16, CUDA1 at a slower link). Use `--draft dflash2 --layer-split auto` with a tight
`--vram-reserve-later-mib`: ours is `--expert-cache 400 --max-context 16384 --kv int8 --vram-reserve-mib 2000
--vram-reserve-later-mib 5800`, Q2_0 pack. Start the server several times: some starts keep DFlash2 and some fall back.
Once the engine has `STRATA_SPLIT_LINK_MIN` (0.1.63+), setting it to 3.5 and then to 4.5 should force each side.
Workaround: `--layer-split 47`.

### Reproduction status

Reproduced locally

### Strata version

0.2.16-4 (58ee32277) and 0.2.17-3 (3142b66b9), engine 0.1.62 in both

### Graphics card, memory and system

RTX 3060 Ti 8 GB (CUDA0) + RTX 3070 Ti 8 GB (CUDA1), i9-10900, 128 GB RAM, Ubuntu 24.04.5, driver 595.99.02. Other GPU
processes start after Strata, which is why reserve-later is set.

---
A patch with CPU regression tests is in a draft PR linked below. It has not been compiled for CUDA or run on GPUs, and
the PR lists what still needs checking.

Sur le site

Liens install, modèles, releases.