#2232 Throughput with 4 slots on 2x RX 7900 XTX + an old Xeon: where to go from here?

Description

Thanks again for 0.1.43.1 (#1996). Now that the engine holds under our load, I would like your advice on speed. Below is our setup, what we measured, and where we would like to get to. Some of it may be a wrong assumption on our side; please correct anything that is.

## Setup

- **GPUs:** 2x RX 7900 XTX 24 GB (gfx1100), power-capped at 272 W. We cannot raise the cap: the same PSU feeds the CPUs and the RAM.
- **Host:** 2x Xeon E5-2630 v2 (Ivy Bridge, AVX only), 128 GB DDR3 (192 GB from next week), each XTX on PCIe 3.0 x8.
- **Other cards in the box:** 2x MI50 32 GB (gfx906). They run other services; the Strata vision encoder runs on one of them (`cuda_device`).
- **Engine:** 0.1.41 with our patch C, moving to 0.1.43.1. ROCm 7.2.4, `STRATA_ISA_FLOOR=avx`, `STRATA_HIP_WMMA=1 STRATA_SELECT_WMMA=1`.
- **Config:** `parallel 4`, `--max-context 262144 --kv int8 --kv-resident 32768 --spec 4 --mtp --no-prefill-borrow --batch-groups auto --trim-stage-weights --layer-split 27 --conversation-cache-mib 6144 --conversation-cache-slots 4 --vram-reserve-mib 700`, `--expert-cache auto` with a profile.

## The workload

The engine serves a small company (in-house, no data leaves the box):

- **3 people using agent sessions** (Hermes desktop). The conversations grow long: 100K-170K tokens is common, because the session context is 262K and compression only starts near the top.
- **Background agent cards** (Hermes kanban, 1 at a time). These are also long conversations, but they do not need to be fast.
- **Delivery-note reading:** a vision request, then a structured text request, a few times an hour.
- **Short calls** from other tools (OCR fallback, guards).

94% of the tokens written in production are written while 2 or more slots are busy.

## What we measured

**One request alone:**
- 52-62 tok/s at 83% of the experts in VRAM (CUDA0 11,833 of 13,824 slots, CUDA1 8,631 of 10,752).
- 71.3 tok/s in queue mode (no `--batch`), where 100% of the experts fit. This was measured at 65K context.
- MTP acceptance in production is 69-73%.

**Each slot costs** 0.52 GiB (CUDA0) and 0.50 GiB (CUDA1) of VRAM, plus 3.09 GiB of pinned RAM at 262K. With `--expert-cache auto`, one slot fewer means about 360 + 265 more resident expert slots (83% → ~86%).

**4 slots, production log, 350 minutes of real work:**

| | Total decode | Per request (4 busy) |
|---|---|---|
| Only decode windows (nobody reading a prompt) | ~72 rows/s | ~18 tok/s |
| Whole batch time, including prompt reads | ~32 rows/s | **~8-9 tok/s** (what our users see) |

While one slot reads a long prompt, the other slots get very little decode time. In that stretch, 101 of the 182 minutes in batch were such stalls.

**Prompt reads:** long prompts read at ~1,850-2,000 tok/s. A conversation that has to be read again from its 16-21K checkpoint costs 73 s for ~150K tokens. In 2 hours of production, 46 re-reads over 20K tokens took 39 of 123 minutes.

**Conversation cache (6 GiB):**
- One parked 168K conversation takes 3-3.7 GB, so the cache holds 1-2 long ones.
- 62 of 67 evictions came from the 4-conversation limit, not from the byte budget.
- Restores themselves are fast (0.2-2 s).

**2 slots instead of 4** (same 6-client stress, 75 min, `--conversation-cache-slots 4`): stable, 22 GiB RAM free instead of ~10. The total written did not go up (~2,500 tokens/min against ~2,900 with 4 slots on 0.1.43), and each delivery note took 190-210 s instead of ~110 s because the requests waited for a slot.

**`--batch-mtp`:** from generate.cpp (0.1.43.1) we understand that on a layer split it needs serial windows (`--batch-groups 1` or `STRATA_BATCH_MTP_SERIAL=1`) and about 0.9 GB per slot for the drafters. That is 3.6 GB fewer experts for us, and the pipelined groups are 25-30% faster anyway. So we have not tried it. Is that right for our case?

## Where we would like to get to

- **4 concurrent agent sessions at 20-25 tok/s each, as users see it**, instead of 8-9 today. A turn on a long conversation should not stall everyone else for a minute.
- **Long conversations should not be read again** when their owner comes back after a few minutes.
- **The background cards can be slow.** If it helps, they could run on a separate, slower instance.

## Questions

1. **Decode during long prompt reads.** Is `STRATA_BATCH_DECODE_SHARE` the right knob? What value would you start from for 4 slots and 100K+ prompts? Is there anything else that keeps the decoding slots moving while one slot reads, such as the prefill chunk size inside the batch?
2. **Conversation cache.** With 192 GB of RAM we plan to raise `--conversation-cache-mib` to 16-24 GiB. Is `--conversation-cache-slots` larger than `--batch` fine, for example 6-8 parked conversations with 4 slots? Would more frequent checkpoints make a partial restore cheaper than a full re-read?
3. **Queue mode.** Would `parallel 1` (100% experts, MTP on every request) with 6-8 parked conversations be a sensible alternative for this mix, or does the queueing always lose with several agent sessions?
4. **VRAM for experts with 4 slots.** Is there a safe way to get closer to 100% resident? For example a lower `--kv-resident`, a smaller `--vram-reserve-mib`, or a shorter `--max-context` for some slots.
5. **The MI50s (gfx906).** We know a gfx906 build cannot share a process with gfx1100 today. Is a mixed build, with a gfx906 card as helper expert cache next to two gfx1100 cards, something you would consider? We can test on real hardware. Otherwise, any advice for a separate gfx906 instance on the 2 MI50s, for the slow background jobs?
6. **Hardware.** We are thinking of moving the two XTX to a modern desktop: Zen 5, PCIe 4.0 x8/x8, 128 GB DDR5. How much of our numbers is bound by the old CPU and PCIe 3.0 (the 17% of experts computed on an AVX-only CPU, KV streaming from RAM), and how much by the GPUs? We would rather not buy the wrong thing.

We can run any test or build you suggest and report back with the same harness.