Issues / #644

#644 Specifying explicit GPU split crashes Strata

closed · @tonydiep · 1 comentarios · En GitHub

Server & APIMulti-GPUModels & quantsDocumentation

Descripción

# ISSUE

Specifying explicit GPU split crashes Strata. 

[Multi-GPU Documentation](https://github.com/Niko1221/Strata/blob/main/docs/MULTI_GPU.md) says Strata supports specifying split in strata-<model>.json but Strata crashes on any value except "auto"

```
#strata-<model>.json

 "layer_split": "auto"  <-------works
 "layer_split": "24,16,12,12"  <-------fails
 "layer_split": [24,16,12,12]  <-------fails
```

# Specifying Split Fails

[strata] layer split across GPUs [2, 0, 1, 3] (24,16,12,12)
[strata] starting the engine: reading the model's weights ...
Traceback (most recent call last):
  File "/home/tonydiep/Strata/serve/server.py", line 2710, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/tonydiep/Strata/serve/server.py", line 2587, in main
    engine = StrataEngine(exe, engine_args(cfg), cwd=cfg.get("cwd"), log=cfg.get("log"), env=env, lazy=lazy)
  File "/home/tonydiep/Strata/serve/server.py", line 291, in __init__
    raise RuntimeError("the engine exited before it was ready" + (f" (see {log})" if log else "") +
                       start_failure_hint(log, log_start))
RuntimeError: the engine exited before it was ready (see /home/tonydiep/Strata/strata-iq3_xxs.log)
tonydiep@tiny:~/Projects/optimise-llms$ tail /home/tonydiep/Strata/strata-iq3_xxs.log)
bash: syntax error near unexpected token `)'


# "auto" Layer Works

[strata] layer split across GPUs [2, 0, 1, 3] (auto)


En el sitio

Enlaces a install, modelos, releases.