Pull requests / #450

#450 calibrate: a gpu list is not a layer split (#447)

closed · @EdderTalmor · 0 Kommentare · Auf GitHub

Setup & installServer & APIMulti-GPUNVIDIA / CUDADocumentation

Beschreibung

Fixes #447.

`tools/calibrate.py` read "the config has a `gpu` list" as "this model splits layers across several cards" and appended `--layer-split auto`. A `gpu` list does not imply a split: the helper-GPU expert tier (`--expert-cache-deviceN`, docs/SECOND_GPU.md) runs one stage on one card with a second card visible for the tier, and the engine rejects `--expert-cache-remote` together with a layer split — so `--calibrate` exited before the sweep started and left the machine uncalibrated with only a warning line.

The fix mirrors `serve.server.engine_args()`: append the split only when several cards actually run stages (`len(gpu_list(cfg)) > 1`). Extracted as `split_args(cfg)` so it is testable without a GPU.

Tests added (`tools/test_calibrate.py`): the helper-tier config gets no split, a real multi-card split still gets `auto", an explicit `--layer-split` in the config is left as it is. Full suite passes: 12/12.

Verified on the reporter's setup shape: `gpu: [0]` + `CUDA_VISIBLE_DEVICES: "0,1"` + `--expert-cache-device1` — the sweep now starts and writes the calibration record.

Mehr auf der Site

Links zu Install, Modellen, Releases.