#4175 [Engine] hot-swap the running model at runtime
open · @jdobson1 · 0 Kommentare · Auf GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quants
Beschreibung
## Problem
Strata serves one model per server process. To change the model you stop the
server and start it again on the new one — a full restart that drops the API
and makes every client reload. On a machine that runs several models on one
card (this one switches between Qwen3.8-Flash-Next and DeepSeek V4.1), that
restart is minutes of downtime every time the model is changed.
## Result
The server can now swap its engine in place while it keeps running. A new
`POST /v1/models/<name>/load` unloads the current engine, builds the target's
from its `strata-<name>.json` run config, and adopts it atomically. The
dashboard picker lists the switchable models from `serve/strata-*.json`
(`GET /v1/models/available`). Requests answer 503 only for the length of the
swap. The swap works for a different engine binary too (DeepSeek's dsv41), not
just variants of the one already running.
## Approach
- `serve/server.py` — `start_swap` runs the work in a background thread and
`POST .../load` returns 202 at once; requests answer 503 while it runs.
`_do_swap` frees VRAM/RAM first, builds the other model with the same
helpers `main()` uses — including its own `exe` from the config, so a config
can name a different engine binary — then adopts it under the status lock.
The swap rebuilds every tokenizer-derived cache (`stop_ids` and the
`PromptEncoder` in `self.prompts`), not just `self.tok`/`self.template`, so
a model with a different vocab is tokenized with its own tokenizer (#537).
- `tools/strata_tokenizer.py` — a literal-token char from a non-byte vocab
(DeepSeek's |/▁) is the token's own UTF-8 text, not a byte-alphabet lookup;
`token_bytes` falls back to the char's UTF-8 bytes instead of raising
`KeyError`, so the model's special tokens detokenize.
## Validation
- `serve/test_lifecycle.py`: 9/9 pass (model discovery, on-demand load, safe
unload, streaming reload).
- `serve/test_detok.py`: pass (the five pack-vocab cases skip without a local
pack on the test path; the rest pass).
- Live hot-swap on the running server, both directions, on the AMD Radeon GPU
(24560 MiB) with the server's own 60.9 GiB RAM:
- `iq3_s` (Qwen3.8-Flash-Next, engine 0.1.46.2) → `deepseek-v4.1-flash` →
`iq3_s`. Each side served real responses afterwards.
- iq3_s measured ~48 tok/s (predicted), deepseek ~8.3 tok/s — its 132 GB of
routed experts read mostly from the CPU lane, 5.86 GB weights + 17.59 GB of
the hottest experts on the card (2000 of 15360), the rest in place on CPU.
- DeepSeek's `<|end▁of▁sentence|>` rendered correctly in the reply,
confirming the `token_bytes` fix live.
## Limits
- A swap is a full engine reload: the API answers 503 for the load time
(seconds when the files are warm on disk, longer for a cold model).
- Tensor/layer-split (multi-GPU `replicas`) configs are not hot-swappable yet;
the swap refuses one and says to restart the server with that model.
- One swap at a time: a swap refuses to start while another is in progress or
a request is in flight.
Mehr auf der Site
Links zu Install, Modellen, Releases.