#6308 [Engine] RTX 4090 + RTX 3090 layer split: 0.1.62 reads prompts 6-12% slower than 0.1.59 (prompt experts moved from MMQ to the fused int8 kernels)
open · @alphastorm · 0 评论 · 去 GitHub 看
BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows
说明
### What I measured The same Windows 11 machine, model files and settings, measured with two stock releases side by side: - **A**: Strata v0.2.13 (engine 0.1.59). **B**: Strata v0.2.16 (engine 0.1.62, Windows CUDA 13 archive, build `1d295787b365+cuda13.0`). - GPUs: RTX 4090 24 GB (Main, drives the display) and RTX 3090 24 GB (Helper). Stock setup choices `--gpu-roles 0=main,1=helper --layer-split 24`, stage weights trimmed. Both engines log the same stage order (GPU 1, then GPU 0) and the same split: layers 0-23, then 24-47. - Model: Qwen3.8-Flash-Next GSQ-RCO IQ3_S. Server arguments are identical apart from paths: `--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --max-context 262144 --kv int8 --kv-resident 32768 --draft-conf 0.5 --ple-io direct --trim-stage-weights`, DFlash2 drafting. - Engine environment: each side uses its own release's stock sm_86 table, plus `STRATA_GDN_CONVL2=2 STRATA_GDN_TC=1 STRATA_SELECT_TWO_PASS=1`. The only difference is `STRATA_ADAPT_NOWAIT=1`, which is in 0.1.59's table and not in 0.1.62's. - Method: six cold starts in the order A1 B1 B2 A2 A3 B3. Each start sends one fresh unique prompt per depth through the OpenAI-compatible endpoint, with 512 output tokens. Prompt speed is the server's own `prompt_n / prompt_ms`. ### Results Prompt tokens/s (server timings), three cold starts each: | Depth | A (0.1.59) | B (0.1.62) | B / A | |---:|---|---|---:| | 8K | 1817, 1835, 1793 | 1702, 1711, 1702 | 0.94 | | 32K | 3380, 3370, 3377 | 3033, 3018, 3020 | 0.90 | | 100K | 3942, 3913, 3916 | 3465, 3472, 3480 | 0.88 | | 200K | 3955, 3964, 3962 | 3471, 3465, 3472 | 0.88 | Starts on the same release agree within about 2%. Decode at the same depths is equal or better on 0.1.62 (217-227 against 202-220 tok/s at 8K), and a repeated 32K prompt gets its first token about 7x sooner. This report is only about cold prompt reads. ### What differs in the logs - 0.1.62 prints `strata: prompt experts on the fused int8 kernels (STRATA_PF_FUSED=1, #136)` and `strata: prompt kernels: chunks of 1024 tokens or more run 48 of 48 layers on the fused kernels; shorter chunks run 48 on the staged fused entry.` 0.1.59 prints neither. - The prompt path borrows fewer cache slots on 0.1.62: 1627 on CUDA0 and 1460 on CUDA1 (2.94 GiB on each card), against 2021 and 1822 (3.66 GiB) on 0.1.59. Expert-cache sizes and free VRAM after loading are the same on both releases (10779/10767 and 8930/8919 slots; 667 and 562 MiB free). ### Context [`fused_tile_rule.hpp` lines 29-33](https://github.com/Niko1221/Strata/blob/64524fb9fc82222ddce554c0643b64f056ec8366/engine/include/strata/prefill/fused_tile_rule.hpp#L29-L33) notes that compute capability 8.9 joined the sm_86 class without a card of that class having run these kernels. The 0.2.14 release notes then report a single RTX 4090 reading IQ3_S prompts of 4K to 32K tokens in the same or less time than 0.2.13 (for example 5,656 against 6,110 ms at 20,000 tokens). On this 4090 + 3090 layer split, with 80% of the experts resident, prompt reads are slower from 8K to 200K. ### Question Is the slower prompt read expected for a layer-split Ampere + Ada pair, now that the Windows archive has no MMQ? If it helps, I can run the same six-start A/B with `STRATA_PF_FUSED_SM86=0`, or with any other switch you name. Reproduction status: Measured locally.
本站相关内容
相关页面的快捷入口。