Issues / #3313
#3313 [Engine] the sm_86 table is skipped when a card of another architecture is visible (2x RTX 3090 + RTX 5060 Ti)
open · @txiki739 · 0 commentaires · Sur GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDALinux
Description
### Area Engine ### Desktop app, Terminal or source? Source: the engine built from v0.2.0-beta (3d1fb480, engine 0.1.45.3) for `86;120`, started with `serve/server.py --engine strata`. ### What happened 0.2.0-beta turns the sm_86 table on (`apply_nv_defaults`, src/program/generate.cpp) only when every visible card has the same compute capability. On a PC with two RTX 3090 (sm_86) and an RTX 5060 Ti (sm_120) - the layers split over the two 3090s, the 5060 Ti a helper expert cache (`--layer-split auto --split-device 1 --expert-cache-device1 auto --remote-expert-opt`) - no table is applied, and the start says nothing about it. With `CUDA_VISIBLE_DEVICES=0,1` (the two 3090s only) the same engine prints `sm_86 (Ampere): 9 exact speed switches on by default`; with `0,1,2` it prints no such line. With the table's nine switches set by hand in the config's `env` (UD-IQ4_XS): | | default | sm_86 table by hand | |---|---:|---:| | `strata_bench`, short chat decode (median of 9) | 122.5 | 126.3 (+3.1%) | | `strata_bench`, agent turns decode (median of 15) | 118.0 | 123.9 (+5.0%) | | `strata_bench`, 32K prompt tok/s | 1,917 | 1,913 | | six fixed prompts, greedy, decode geomean (2 runs each) | 135.2 | 150.8 (+11.5%) | The `strata_bench` runs are in the benchmark PR #3314. The six-prompt numbers are from my own harness (Spanish chat and thinking, an 18K document, code, a 3K-token edit, a 5K prompt; 256-2,600 new tokens each, longer answers than `strata_bench`'s). Of its +11.5%, the four kernel switches (`CPU_IQ_ALL_EXACT1`, `ATTN_DAILY`, `GR_UP75`, `GU_SWIGLU`) gave +5.3% on their own and `ADAPT_NOWAIT` + `ADAPT_LAG` +1.7%. On UD-Q4_K_XL the four kernel switches gave +1.7%. Suggestion: the switches are bitwise and process-wide, so with them set the helper card computes the same values (here they ran on all three cards and were faster). Either pick the table from the cards that run the dense layers (a split's stages, or the main card) and leave helper-only cards out of the "same compute capability" check, or at least print one line when the table is skipped because the visible cards differ and let `STRATA_SM86_PERF=1` force it (today only `=0` is read). ### Reproduction status Every start: the table line appears with the two 3090s visible and never with the 5060 Ti visible too. ### Strata version 0.2.0-beta (engine 0.1.45.3, 3d1fb480) ### Graphics card, memory and system 2x RTX 3090 24 GB + RTX 5060 Ti 16 GB (PCIe 4.0 x8 each, one 3090 behind the chipset), Ryzen 9 5950X, 128 GB DDR4, Linux (CachyOS, kernel 7.2), driver 615.78.08, CUDA 13.4, GCC 16.2.1.
Sur le site
Liens install, modèles, releases.