Issues / #2463
#2463 Layer-split auto (0.1.44.1): a 2x link gap is no longer priced, so a 4090 x16 + 3090 x8 pair picks K=45 and runs ~25% slower; STRATA_SPLIT_LINK_ABS=1 restores 0.1.43.1
open · @alphastorm · 0 commentaires · Sur GitHub
BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows
Description
**Strata version, GPU, OS:** 0.1.44.1 (`296b59d2`, release engine) · RTX 4090 24 GB (CUDA0, PCIe Gen3 x16, probe 12.4 GB/s) + RTX 3090 24 GB (CUDA1, Gen3 x8, probe 6.2 GB/s) · Windows 11, driver 617.42 · i9-7900X (X299, so Gen3 is the ceiling), 128 GB DDR4-3200 Qwen3.8-Flash-Next IQ3_S with setup's plan for this pair: `--max-context 262144 --kv int8 --kv-resident 32768 --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --conversation-cache-mib 8192 --pipeline-windows 2`, layer split auto (CUDA0 then CUDA1; #1576 keeps that order). ### What happened On this pair 0.1.44.1's auto split runs layers 0-44 on the 4090 and only 45-47 on the 3090, where 0.1.43.1 ran 0-31 and 32-47. Decode drops by about a quarter and long prompts read at about half speed. `STRATA_SPLIT_LINK_ABS=1` on 0.1.44.1 brings back K=32 and 0.1.43.1's numbers. Measured back to back on the same host: same flags and pack, each arm from a cold start on idle GPUs, 512 output tokens per depth through the API, one request per depth, prompt and decode rates as the server reports them. | | 0.1.43.1 | 0.1.44.1 | 0.1.44.1 + `STRATA_SPLIT_LINK_ABS=1` | |---|---|---|---| | miss cost CUDA0 / CUDA1 | x1.62 / x3.24 | x1.00 / x1.00 | x1.62 / x3.23 | | auto K (layers on CUDA0 / CUDA1) | 32 (0-31 / 32-47) | 45 (0-44 / 45-47) | 32 (0-31 / 32-47) | | CUDA1 expert cache | 8,185 slots, 16.50 GiB | 1,536 slots, 3.42 GiB | 8,183 slots, 16.50 GiB | | profiled pairs held (routed mass) | 17,512 (98.9%) | 10,580 (97.4%) | 17,510 (98.9%) | | predicted per decode window | 18.6 ms | 16.6 ms | 18.6 ms | | decode tok/s at ~60 / 8K / 32K / 64K / 100K | 173 / 206 / 197 / 194 / 184 | 113 / 151 / 154 / 148 / 148 | 164 / 196 / 196 / 189 / 187 | | prompt tok/s at 8K / 32K / 64K / 100K | 1,583 / 3,120 / 3,506 / 3,599 | 1,222 / 1,784 / 1,916 / 1,868 | 1,605 / 3,123 / 3,519 / 3,604 | | TTFT at 100K | 28.6 s | 55.0 s | 28.6 s | 0.1.44.1's default also prints: ``` strata generate: layer split auto: K=47 would give a stage fewer than 3 layers (a tiny expert cache there clamps the prompt chunk of every card, #1616); chose K=45 instead (16.6 ms per decode window, not 16.5) strata generate: layer split auto: NEAR TIE - K=42 is predicted 16.7 ms per decode window, within 0.7% of K=45 (16.6 ms); the cards differ, so the prediction may not rank them (it prices decode, not the prompt). To try the other: --layer-split 42 strata serve: WARNING: prompt chunk 5888 tokens, not 8192: CUDA1 (NVIDIA GeForce RTX 3090) has 1536 expert-cache slots, and a 8192-token chunk borrows 1536 of them (it must keep 128, and lend at most 85%) - prompts read slower than on CUDA0 alone (#448) ``` ### Why #2007's `link_miss_scale` ([split_rules.hpp#L77-L98](https://github.com/Niko1221/Strata/blob/296b59d2f73c8f04a805df29c2fa97655adace78/include/strata/program/split_rules.hpp#L77-L98)) scales a stage's miss term only from a 4x gap against the fastest link. At 12.4 against 6.2 GB/s both stages get x1.00 ([generate.cpp#L3846-L3855](https://github.com/Niko1221/Strata/blob/296b59d2f73c8f04a805df29c2fa97655adace78/src/program/generate.cpp#L3846-L3855)), and the search then holds 6,932 fewer profiled pairs (97.4% of the routed mass instead of 98.9%). The comment there expects a gap below 4x to matter less than the probe's noise and the formula's fit. On this rig it is the largest effect, perhaps because the slower link is also on the slower card (82 SMs at 1.75 GHz against 128 at 2.52). So the 0.1.41 rule is right on this pair, the 2x Tesla T10 rig in that comment needed it gone, and #2418 (a 4.2x gap, x16 + x4) asks for a uniform cost. The link ratio alone does not seem to separate the three rigs.
Sur le site
Liens install, modèles, releases.