Issues / #1910

#1910 [Regression] Prefill 35-55% slower on v0.1.42 vs v0.1.41 on dual 2080 Ti (cold long prefix)

open · @wszhoho · 7 comments · View on GitHub

BenchmarksMulti-GPUNVIDIA / CUDAModels & quantsLinux

Description

## What do you want to do?

**Report a prefill throughput regression between v0.1.41 and v0.1.42** on dual-GPU CUDA hardware, in the hope that it is a fixable regression rather than an intended trade-off.

### Environment
- Hardware: 2 × RTX 2080 Ti (11 GB each, 22 GB total VRAM), 62 GB RAM, Linux
- Model: IQ3_S (GSQ-RCO, Qwen3.8-Flash-Next family), 256K context
- Always measured with a cold 40K-token prompt (27,318 prompt tokens), same prompt across all runs

### Measured prefill (cold 40K prefix)
| Build / config | Prefill (tok/s) |
|---|---|
| v0.1.41 | ~1150–1280 |
| v0.1.42, defaults | ~790 |
| v0.1.42, `--no-prefill-borrow` | ~565 |
| v0.1.42, `STRATA_SPLIT_OWN=1` | ~530 |
| v0.1.42, `STRATA_ROUTE_TAIL_SKIP=0` | ~800 (still far below v0.1.41) |

So on this hardware v0.1.42 is 35–55% slower than v0.1.41 for a cold long prefix, and none of the exposed knobs (`--no-prefill-borrow`, `STRATA_SPLIT_OWN=1`, `STRATA_ROUTE_TAIL_SKIP=0`) restores v0.1.41 speed. This suggests a lower-level regression in the prefill path (e.g. the BF16 prefill change, chunk scheduling, or the per-card weight/session split) rather than a single settable option.

### Question for the maintainers
Was v0.1.42's prefill path changed in a way that could explain this on small-VRAM dual-GPU layer-split setups (22 GB total)? If so, is there an intended opt-in/opt-out equivalent to v0.1.41's behavior that we are missing?

(Not a criticism — just flagging the regression so it can be triaged. Happy to provide more measurements or a reproducible script.)

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.