Issues / #644
#644 Specifying explicit GPU split crashes Strata
closed · @tonydiep · 1 コメント · GitHub で見る
Server & APIMulti-GPUModels & quantsDocumentation
本文
# ISSUE
Specifying explicit GPU split crashes Strata.
[Multi-GPU Documentation](https://github.com/Niko1221/Strata/blob/main/docs/MULTI_GPU.md) says Strata supports specifying split in strata-<model>.json but Strata crashes on any value except "auto"
```
#strata-<model>.json
"layer_split": "auto" <-------works
"layer_split": "24,16,12,12" <-------fails
"layer_split": [24,16,12,12] <-------fails
```
# Specifying Split Fails
[strata] layer split across GPUs [2, 0, 1, 3] (24,16,12,12)
[strata] starting the engine: reading the model's weights ...
Traceback (most recent call last):
File "/home/tonydiep/Strata/serve/server.py", line 2710, in <module>
sys.exit(main())
~~~~^^
File "/home/tonydiep/Strata/serve/server.py", line 2587, in main
engine = StrataEngine(exe, engine_args(cfg), cwd=cfg.get("cwd"), log=cfg.get("log"), env=env, lazy=lazy)
File "/home/tonydiep/Strata/serve/server.py", line 291, in __init__
raise RuntimeError("the engine exited before it was ready" + (f" (see {log})" if log else "") +
start_failure_hint(log, log_start))
RuntimeError: the engine exited before it was ready (see /home/tonydiep/Strata/strata-iq3_xxs.log)
tonydiep@tiny:~/Projects/optimise-llms$ tail /home/tonydiep/Strata/strata-iq3_xxs.log)
bash: syntax error near unexpected token `)'
# "auto" Layer Works
[strata] layer split across GPUs [2, 0, 1, 3] (auto)
関連リンク
インストール・モデル・リリースへの站内リンク。