#6427 [Engine] layer split auto with DFlash2: plan DFlash2's drafter and after-cache bind on the last stage, and keep the gate-off fallback loadable

open · @VocanicZ · 0 コメント · GitHub で見る

Setup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

本文

Fixes #6426

## Problem and result

Rig: 2x 8 GB, RTX 3060 Ti (CUDA0, PCIe x16) + RTX 3070 Ti (CUDA1, slower link), 48-layer model (Q2_0 pack), serve with DFlash2 on the last stage and `--vram-reserve-later-mib 5800`. No placement can start the prompt path here, so `--layer-split auto` chooses by decode speed alone (`no placement can start the prompt path on the figures here; choosing by decode speed alone`). The PCIe probe puts the CUDA0/CUDA1 link gap on either side of `kLinkScaleMin` (4.0), and that moved the pick between K=45 and K=47. K=45 stops at `layer split, CUDA1 expert cache: no room` and the server starts the engine again without a drafter.

The pick should never land on a placement whose last stage cannot hold an expert slot. Two things allowed it:
1. **The search planned the last stage larger than the cache step found it.** With DFlash2 on the last stage, two terms were missing from the search:
   - what DFlash2 binds after the cache (`df2_after`: draft logits 48 MiB, and 112 MiB of prompt staging in serve);
   - DFlash2's drafter and the head, ~1100 MiB on this rig against `kDrafterMib`'s 1000 (the MTP figure). Measured on CUDA1 from `holds its weights` to the cache step: 7.30 -> 6.21 GiB at K=47 and 7.23 -> 6.13 GiB at K=46, less the session.
2. **With the startability gate off, nothing checked that a later stage's cache could hold anything.**

## Hardware run (Strata alone on both cards, this branch and its parent built here)

The same engine flags every time, through `terminal/serve/server.py` (the server's own drafter fallback in place):

```
strata --serve --pack <Q2_0 pack> --native <Q2_0 gguf> --expert-profile <profile> --expert-cache 400 --prefill auto \
  --max-context 16384 --kv int8 --ple-io ram --vram-reserve-mib 2000 --vram-reserve-later-mib 5800 \
  --draft dflash2 --draft-model <dflash2 q4_0 pack> --draft-fallback none --mtp-draft-vocab <draft_vocab.bin> \
  --spec 6 --draft-conf 0.8 --layer-split auto
```

With nothing else on the cards, the probe read CUDA0 12.3-12.6 GB/s and CUDA1 2.9-3.0 (gap 4.1-4.3, above the step), so `STRATA_SPLIT_LINK_MIN` forces each side: `=4.5` leaves CUDA1's miss cost unscaled (x1.00, the side that failed), `=3.5` scales it (~x4.2).

| engine | `STRATA_SPLIT_LINK_MIN` | starts | auto pick | CUDA1 at the cache step | DFlash2 |
|---|---|---|---|---|---|
| parent (3142b66b9) | 4.5 | 3 | K=45 | `6.06 GiB free of 7.66, room for experts 0.00 GiB` | 0/3: `CUDA1 expert cache: no room`, started again without a drafter |
| parent | 3.5 | 3 | K=47 | `6.21 GiB free of 7.66, room for experts 0.04 GiB` | 3/3 on |
| first commit only (b1b697376) | 4.5 | 5 | K=46 | `6.13 GiB free of 7.66, room for experts 0.00 GiB` | 0/5: no room, started again without a drafter |
| first commit only | 3.5 | 5 | K=47 | 0.04 GiB | 5/5 on |
| first commit only | unset | 3 | K=47 | 0.04 GiB | 3/3 on |
| this PR | 4.5 | 5 | K=47 | `6.21 GiB free of 7.66, room for experts 0.04 GiB` | 5/5 on |
| this PR | 3.5 | 5 | K=47 | 0.04 GiB | 5/5 on |
| this PR | unset | 3 | K=47 | 0.04 GiB | 3/3 on |

(The released 0.1.62 engine has no `STRATA_SPLIT_LINK_MIN`; six starts of it on the same setup read a gap above 4 and kept DFlash2 at K=47.)

The first commit alone was not enough: it moved the pick from K=45 to K=46, which also had no room. The second commit books DFlash2's drafter over `kDrafterMib`, and the search's plan then matches the cache step. On the `=4.5` start with this PR, the search logs `CUDA1: ... 22 MiB is left for them after its session and cache` at K=47, the same 22 MiB that `split_rules_test` computes from the logged figures, and the cache step finds 0.04 GiB (28 slots).

## Approach

- `stage_reserve.hpp`:
  - `df2_after_mib(serve)` (48, plus 112 in serve) is now the one figure for what DFlash2 binds after the cache beside its head. The cache step's `df2_after` uses it, with the same value as before.
  - `kDf2DrafterOverMib` (112): DFlash2's drafter and the head over `kDrafterMib`, measured as above and rounded up.
  - `df2_search_mib(serve)` = the two together. The search books it through a new `Inputs::df2_bind_mib`, on the drafter's stage only. The cache step measures the loaded drafter, so it subtracts only `df2_after`.
  - The search books it only when DFlash2 will run on the last stage: `--draft dflash2` (not `--draft auto`), not `STRATA_DF2_STAGE=first`, no `--pipeline-windows`, a native pack, a draft model, and not `STRATA_DEFAULTS=026`. These are the conditions of the cache-step term, plus the DFlash2 refusals that are known before the search.
- `split_rules.hpp`: `later_stage_empty(held)`. When nothing passes the gate, the fallback first searches with the rule that every stage after the first holds at least one slot in the search's own fill. That fill stops at the first ranked pair that does not fit, like the cache step.
  - The rule covers the 2-, 3- and 4-way searches.
  - If no placement qualifies, the search runs as before.
  - The first stage is left out, because it sizes its cache through the single-card path.
  - Both parts are needed. Without the floor, the decode-speed pick can still be a placement with no slots on CUDA1. Without the larger reserve, K=45 and K=46 look like they hold slots.
- Unchanged:
  - one GPU, a manual `--layer-split`, a split without DFlash2 (the term is 0), DFlash2 on the first stage, and every start in which some placement passes the gate;
  - the MTP drafter's pricing (`kDrafterMib`).
- Not mirrored to the SYCL engine, which has no DFlash2. The fallback rule could be mirrored there separately.

## Related discussion

Related: #5571 and #5691 (later-stage reserve constants against measurement: `kDf2DrafterOverMib` is one more measured constant of that kind), #1428 (auto K sensitivity), #2418 / #4041 (the 4x link step, not changed here).

## Validation

**Passed:**
- CUDA build: `cmake -G Ninja -S engine` with `tools/packaging/build_cuda_linux.sh`'s flags and `CMAKE_CUDA_ARCHITECTURES=86-real;86-virtual`, CUDA 13.0.88, GCC 13.3, Ubuntu 24.04, target `strata`. It was built for this branch and for its parent.
- Hardware: the table above, 32 starts on RTX 3060 Ti + RTX 3070 Ti, driver 595.
- CPU: the `cpu-check` preset's variables (Ninja), `ctest` on this branch: `100% tests passed out of 141`. These four skipped themselves: io_priority_test, expert_multi_test, expert_parity, pool_test.
- `split_rules_test`: `DFlash2 on CUDA1, gate off: K=45 holds 0 slots, K=46 0, K=47 16; pick K=47 at gap 3.66, K=47 at 4.34`.
- `stage_reserve_test`: `864 cases, PASS`. The 864-case grid against the old expression still holds, because the new field defaults to 0.

**Failed (before the fix, as expected):** the parent's starts at `=4.5` in the table, and the first commit's K=46 starts.

**Not run:**
- HIP build and an AMD run. The DFlash2 terms are under `STRATA_HAVE_DFLASH2`, and the fallback rule is backend-neutral. A HIP build check would be welcome.
- `python tools/check` as a whole.

## Limits

- `kDf2DrafterOverMib` is measured on one model and one rig. A model whose DFlash2 drafter or head is larger would need it raised, like `kDrafterMib`. Sizing the term from the draft pack and the head before the search would remove the constant, but it touches more of the load order than this fix needs.
- The draft head DFlash2 binds after the cache (178.4 MiB here) is priced at only a few MiB in `df2_after` on a split, because `df2_head_bytes()` reads the row size of CUDA0's head, which a split does not load. It comes out of `--vram-reserve-later-mib`, and this PR does not change it.
- This setup stays tight: K=47 keeps 28 slots on CUDA1. `--layer-split 47` remains the user-side workaround on released engines.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

関連リンク

インストール・モデル・リリースへのサイト内リンク。