#6080 Two cards of different makes behind one server

closed · @sebastiendarocha · 0 comments · View on GitHub

needs-format

Description

Two cards of different makes behind one server

## The problem

A PC with one NVIDIA card and one AMD card could not serve both behind one
server. The replica router started every replica from the same engine program,
and every card list was numbered in one backend's order (nvidia-smi's, or
HIP's), so the two families of cards could not be addressed separately. The
only way was two separate servers, with two ports, two keys and two front
ends.

## The change

An entry of `"replicas"` may now carry `"backend"` (`cuda` or `hip`) and
`"exe"` (the engine program of that replica). A CUDA engine and a HIP engine
then serve behind the same front, each on its own card:

```json
"replicas": [
  {"backend": "cuda", "gpus": [0], "exe": "/opt/strata/components/engine/<pkg-cuda>/strata",
   "env": {"LD_LIBRARY_PATH": "/opt/strata/components/engine/<pkg-cuda>"}},
  {"backend": "hip",  "gpus": [0], "exe": "/opt/strata/components/engine/<pkg-hip>/strata",
   "env": {"LD_LIBRARY_PATH": "/opt/strata/components/engine/<pkg-hip>"}}
]
```

Each replica's `"gpus"` are numbered as **its own** backend numbers the cards,
so card 0 of a CUDA replica and card 0 of a HIP replica are two different
cards. The check that refuses a card in two replicas is therefore per backend:
two replicas of the same backend may not take the same card, two replicas of
different backends may both name card 0.

The two engines never exchange a session: each keeps its own conversations,
and the router sends a conversation back to the replica that holds it. Nothing
of the session file crosses the backends, so the session fingerprint (which
names the backend) is not in the way. This is the arrangement the split
serving path (`docs/SPLIT_SERVE.md`, "NVIDIA with AMD") needs for its prefill
and decode roles, without a rebuild of the engine.

`"replica_shared_arena": true` makes the experts sit in RAM once for the two
engines (one shared memory object, the engine's `--shared-expert-arena`);
without it each engine holds its own copy of the experts.

## What it is not

This is the replica router, not the split: the router hands whole requests to
replicas, it does not hand a prompt from one card to another. A request still
runs entirely on one card. What the change buys is one server (one model, one
key, one address) for a PC that has cards of two makes.

## Files

- `serve/replicas.py` — the config, the per-backend card numbering
- `serve/server.py` — `start_replicas`: each replica starts its own engine
- `serve/test_replicas.py` — three tests (two cards of different makes, the
  per-replica `backend`/`exe`, the per-backend check of a card in two replicas)
- `docs/MULTI_GPU.md` — "Two makes of card behind one server"

## Tested on

A PC with an RTX 2070 (sm_75) and an RX 9060 XT (gfx1200), Debian 13, 31 GiB
of RAM, Coder IQ1_M. The two engines served together behind one server with
the shared arena, each on its own card, and both answered. `serve/` passes in
full (1569 passed, 24 skipped).

Related on this site

Quick links to related pages.