Issues / #1238
#1238 0.1.40: --layer-split auto with STRATA_STAGE_TRIM=1 still picks K=2 on a P40 + RTX 3070; 572b607 fixes it but is not in the release
open · @paulhothersall · 2 comments · View on GitHub
BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quants
Description
**On v0.1.40, `--layer-split auto` with `STRATA_STAGE_TRIM=1` and the P40 first still picks K=2 and prefills at 107 / 112 tok/s (4K / 32K), in two launches; @gopinath87607's 572b607 (the footprint fix posted in #880) picks K=16 and gives 343 / 654.** Limit: one rig, IQ3_XXS, three runs per cell; 572b607 was built on the #880 branch (ce1e63f), not cherry-picked onto the 0.1.40 release. Setup: P40 24 GB + RTX 3070 8 GB, 64 GB RAM, IQ3_XXS, ctx 36,864, `--layer-split auto`, same-day runs. Prefill = prompt tokens / TTFT (runs 2-3). Decode is not compared (31-45 tok/s inside one launch). | build | order | trim | K | prefill 4K / 32K | |---|---|---|---|---| | 0.1.40 | P40 first | 1 | 2 | 107 / 112 (two launches) | | 572b607 | P40 first | 1 | 16 | 343 / 654 (two launches) | | 0.1.40 | P40 first | 0 | 16 | 192 / 220 | | 572b607 | P40 first | 0 | 16 | 192 / 220 | | 0.1.40 | 3070 first | 0 or 1 | 43 | 460 / 617 | | 572b607 | 3070 first | 0 or 1 | 43 | 459 / 617 | My reading: the 2 MiB-granule pricing in 572b607 closes the gap, as its commit message says. Trim-off and 3070-first match 0.1.40, so the fix changes only the arm that was wrong. Ask: could 572b607 go into the next release, or is there a reason it stays out? Not tested: other quants or cards, a single-card case, and 0.1.40 with the commit cherry-picked.
Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.