Issues / #1760

#1760 Regression: automatic GPU ordering in be8db787 reduces decode throughput ~15% on dual RTX 5070 Ti

closed · @L-proger · 3 comentarios · En GitHub

BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows

Descripción

Possibly related to #1352.

The automatic GPU ordering introduced in `be8db787c233acdde3e5887b652a87df23f0445a` causes a reproducible decode performance regression on my dual RTX 5070 Ti setup.

## Hardware

- Windows 11
- 2 × NVIDIA GeForce RTX 5070 Ti 16 GB
- Same model, config and prompt used for all tests
- IQ3_S model

## Performance

| Version / setting | Decode speed |
|---|---:|
| `v0.1.40.4` | ~91 tok/s |
| `v0.1.41` / latest with automatic GPU ordering | ~77 tok/s |
| latest + `"gpu_order": "as_given"` | ~91 tok/s |

So the automatic ordering causes roughly a 15% decode throughput regression on this machine, while opting out restores the previous performance completely.

## Git bisect

I bisected between:

- good: `v0.1.40.4`
- bad: `v0.1.41`

Result:

```text
be8db787c233acdde3e5887b652a87df23f0445a is the first 'bad' commit

serve: card order for an auto layer split, faster card last
(SMs x clock), gpu_order as_given opts out (#1352)
```

## GPU scores and resulting order

Using Strata's own `gpu_speed_scores()` / `ordered_gpus()`:

```text
scores: {0: 173740000.0, 1: 171640000.0}

[strata] layer split: card order 1,0
(the faster card last; "gpu_order": "as_given" keeps 0,1, #1352)

auto order: [1, 0]
as_given: [0, 1]
```

For comparison, `nvidia-smi` reports the same maximum SM clock for both cards:

```text
index, name, clocks.max.sm [MHz], pci.bus_id
0, NVIDIA GeForce RTX 5070 Ti, 3090 MHz, 00000000:01:00.0
1, NVIDIA GeForce RTX 5070 Ti, 3090 MHz, 00000000:03:00.0
```

The CUDA-derived scores differ only slightly, but that is enough for the new heuristic to reorder the cards from `[0,1]` to `[1,0]`.

On this system, that reorder reduces decode throughput from ~91 tok/s to ~77 tok/s.

## Workaround

Adding this to the model config restores the previous performance on the latest version:

```json
"gpu_order": "as_given"
```

With that setting enabled, decode throughput returns to ~91 tok/s.

## Notes

Because both GPUs are effectively the same model and the score difference is very small, it may be worth avoiding automatic reordering when scores are very close.

More generally, `SM count × clock` alone may not be enough to choose the optimal order, since actual performance can also depend on PCIe topology, display usage, device placement and the work assigned to the final pipeline stage.

En el sitio

Enlaces a install, modelos, releases.