#4175 [Engine] hot-swap the running model at runtime

open · @jdobson1 · 0 commentaires · Sur GitHub

BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quants

Description

## Problem

Strata serves one model per server process. To change the model you stop the
server and start it again on the new one — a full restart that drops the API
and makes every client reload. On a machine that runs several models on one
card (this one switches between Qwen3.8-Flash-Next and DeepSeek V4.1), that
restart is minutes of downtime every time the model is changed.

## Result

The server can now swap its engine in place while it keeps running. A new
`POST /v1/models/<name>/load` unloads the current engine, builds the target's
from its `strata-<name>.json` run config, and adopts it atomically. The
dashboard picker lists the switchable models from `serve/strata-*.json`
(`GET /v1/models/available`). Requests answer 503 only for the length of the
swap. The swap works for a different engine binary too (DeepSeek's dsv41), not
just variants of the one already running.

## Approach

- `serve/server.py` — `start_swap` runs the work in a background thread and
  `POST .../load` returns 202 at once; requests answer 503 while it runs.
  `_do_swap` frees VRAM/RAM first, builds the other model with the same
  helpers `main()` uses — including its own `exe` from the config, so a config
  can name a different engine binary — then adopts it under the status lock.
  The swap rebuilds every tokenizer-derived cache (`stop_ids` and the
  `PromptEncoder` in `self.prompts`), not just `self.tok`/`self.template`, so
  a model with a different vocab is tokenized with its own tokenizer (#537).
- `tools/strata_tokenizer.py` — a literal-token char from a non-byte vocab
  (DeepSeek's |/▁) is the token's own UTF-8 text, not a byte-alphabet lookup;
  `token_bytes` falls back to the char's UTF-8 bytes instead of raising
  `KeyError`, so the model's special tokens detokenize.

## Validation

- `serve/test_lifecycle.py`: 9/9 pass (model discovery, on-demand load, safe
  unload, streaming reload).
- `serve/test_detok.py`: pass (the five pack-vocab cases skip without a local
  pack on the test path; the rest pass).
- Live hot-swap on the running server, both directions, on the AMD Radeon GPU
  (24560 MiB) with the server's own 60.9 GiB RAM:
  - `iq3_s` (Qwen3.8-Flash-Next, engine 0.1.46.2) → `deepseek-v4.1-flash` →
    `iq3_s`. Each side served real responses afterwards.
  - iq3_s measured ~48 tok/s (predicted), deepseek ~8.3 tok/s — its 132 GB of
    routed experts read mostly from the CPU lane, 5.86 GB weights + 17.59 GB of
    the hottest experts on the card (2000 of 15360), the rest in place on CPU.
  - DeepSeek's `<|end▁of▁sentence|>` rendered correctly in the reply,
    confirming the `token_bytes` fix live.

## Limits

- A swap is a full engine reload: the API answers 503 for the load time
  (seconds when the files are warm on disk, longer for a cold model).
- Tensor/layer-split (multi-GPU `replicas`) configs are not hot-swappable yet;
  the swap refuses one and says to restart the server with that model.
- One swap at a time: a swap refuses to start while another is in progress or
  a request is in flight.

Sur le site

Liens install, modèles, releases.