#6361 [Benchmark] Unequal NVIDIA pair, both card orders, `--layer-split auto`, and the helper mode: RTX 3060 12 GB + RTX 5060 Ti 16 GB, 0.2.16, NVFP4-GPTQ

open · @Norhiro · 0 comments · View on GitHub

BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsDocumentation

Description

This answers the TEST_REQUESTS.md item "Two NVIDIA cards of different sizes (a large and a small one), in both card orders, with `--layer-split auto`" (#880, #1238). One machine, one model, `tools/strata_bench.py --long`, 3 runs per test. Each card order was started twice, alternating (A B A B), and then once more with the order left to the server. Because the engine suggested it for one order, I also ran the 3060 as a helper expert cache instead of a split stage (twice).

**Summary**
- `--layer-split auto` picked a sensible split in both orders and did not fall into the K=2 case of #1238. With the 3060 first it chose K=16 (3060: layers 0-15, 5060 Ti: 16-47). With the 5060 Ti first it chose K=29 (5060 Ti: 0-28, 3060: 29-47). Both times it logged `K=… would give a stage fewer than 3 layers (#1616) … chose K=… instead`. The caches held 95.8% and 96.0% of the routed mass.
- Both orders loaded and ran every test with no OOM or abort. VRAM free with everything loaded was 610-711 MiB per card. I did not add `--vram-reserve-later-mib`.
- The two orders are close. With the 5060 Ti first, prompt reading was 3-6% faster at 4K and 3% faster at 32K, and the agent-turn TTFT was 3.4 s against 3.75 s. That held in both launches of each order. 16K was the same. Short-chat decode medians varied between launches of the same order (26.2-30.9 tok/s), so I do not read an order effect into decode.
- With `gpu_order` left to the server (5060 Ti listed first), the server put the faster card last (#1352), which is the 3060-first order, and the numbers match that order. On this pair that is the slightly slower of the two orders, by the margins above.
- With the 5060 Ti first, the 3060 is the last stage and gets only 2,219 expert slots. The engine warns and lowers the prompt chunk from 16,384 to 12,032: `WARNING: prompt chunk 12032 tokens, not 16384: CUDA1 (NVIDIA GeForce RTX 3060) has 2219 expert-cache slots …`, and suggests the 3060 as a helper expert cache (`--expert-cache-device1`, docs/SECOND_GPU.md). Even with the smaller chunk, this order read prompts as fast or faster.
- Helper mode (5060 Ti main, 3060 `--expert-cache-device1 auto --remote-expert-opt`) trades prompt speed for decode. Short-chat decode was 32.8-34.1 tok/s (against 26.2-30.9 for the layer split), and agent-turn decode was 52.9 against 35.7-41.9 tok/s. Prompt reading was 15-25% slower than the layer split (4K 397, 16K 790, 32K 1,007 tok/s), because, as SECOND_GPU.md says, the prompt runs on CUDA0 only. The engine warned once or twice per launch that only 498 MiB of VRAM was free on CUDA0 at the start of a request (below 512 MiB). Nothing failed.
- Through `serve/server.py` I found no config key for helper mode. With two cards in `"gpu"`, `layer_split_of()` is true unless `--peer-device` is among the args, so the server adds `--layer-split` (and setup's `gpu_roles` "helper" also means a split stage). I ran it with `"gpu": 0` and `"env": {"CUDA_VISIBLE_DEVICES": "0,1"}` (the config's env is applied after the server sets it). If there is a supported way, I would like to know it.

**Hardware and software**
- GPUs: RTX 3060 12 GB (GA106) at PCIe Gen3 x16 and RTX 5060 Ti 16 GB (GB206) at PCIe Gen3 x8. The engine's own probe measured 12.3-12.4 GB/s and 6.9 GB/s. Driver 615.71.09, CUDA 13 (pip wheels). The desktop used about 440 MiB on the 5060 Ti and 280 MiB on the 3060 before each run. Nothing else ran on the GPUs.
- CPU: Core i7-7800X (6 cores / 12 threads, AVX-512; the engine chose its AVX2 build). 4x 16 GB DDR4-2666, 4 channels (94 GiB visible). Models on NVMe.
- OS: Fedora 45, kernel 7.2.15.
- Strata 0.2.16 (`64524fb9`), engine 0.1.62. Source build: `cmake -S engine -DSTRATA_ENABLE_CUDA=ON -DSTRATA_PORTABLE=ON -DCMAKE_BUILD_TYPE=Release`, gcc 15 as the CUDA host compiler. Server from `terminal/serve/server.py`, Python 3.14.
- Model: the NVFP4-GPTQ pack (`--native-dense-gguf`, `--embd-gguf`, and `--ple-gguf` from UD-Q4_K_XL part 2; the tool names it `nvfp4-UD-Q4_K_XL`).

**Configuration** (the same for every run except the card order)

```text
strata --pack pack --native qwen-nvfp4-gptq-dense.gguf --max-context 65536 --prefill auto --mtp rt \
  --expert-profile expert-profile.bin --expert-cache auto --spec 4 --spec-min-p 0.5 \
  --native-dense-gguf qwen-nvfp4-gptq-dense.gguf --ple-gguf Qwen3.8-Flash-Next-UD-Q4_K_XL-00002-of-00004.gguf \
  --embd-gguf token-embd-bf16.gguf
```

Server config: `"layer_split": "auto"`, `"drafting": "mtp"`, environment `STRATA_PREFILL_AUTO_MAX=16384`, `STRATA_PREFILL_CPU_SHARE=auto` (from my daily config). A: `"gpu": [1, 0]` (3060, 5060 Ti) with `"gpu_order": "as_given"`. B: `"gpu": [0, 1]` (5060 Ti, 3060) with `"gpu_order": "as_given"`. C: `"gpu": [0, 1]` without `gpu_order`. H (helper): `"gpu": 0`, `"env"` adds `"CUDA_VISIBLE_DEVICES": "0,1"`, and the args add `--expert-cache-device1 auto --remote-expert-opt`. No `--vision`, default KV, no `--vram-reserve-later-mib`.

**What the planner saw and chose** (engine log)

| | A: 3060 → 5060 Ti | B: 5060 Ti → 3060 |
|---|---|---|
| Per-layer estimate, first / last card | 1.46 / 0.75 ms | 0.75 / 1.46 ms |
| Free before the session carve, first / last | 6.11 / 12.92 GiB | 9.81 / 9.22 GiB |
| K chosen | 16 (55.2 ms predicted per decode window) | 29 (57.0 ms) |
| Expert slots, first / last card | 3,124 (8.04 GiB) / 3,030 (7.80 GiB) | 3,935 (10.13 GiB) / 2,219 (5.71 GiB) |
| Routed mass held | 95.8% | 96.0% |
| Prompt chunk | 16,384 | 12,032 (warning above) |
| VRAM free with everything loaded | 711 / 704 MiB | 610 MiB on CUDA0 |

H (helper): CUDA0 (5060 Ti) expert cache 2,266 slots (5.84 GiB), CUDA1 (3060) 4,117 additional experts (10.60 GiB, `results return through pinned host rows`), prompt chunk 13,312, 578 MiB free on CUDA0 with everything loaded, `adaptive overlap: off: not with --remote-expert-opt`. A long request logged `CUDA1: 7565 expert entries … host 27 ms staging+launching, 3 ms waiting for it`.

**Results** (medians of 3 runs, 9 for short chat; each order started twice)

| | A run 1 | A run 2 | B run 1 | B run 2 | C (server's order) | H run 1 | H run 2 |
|---|---:|---:|---:|---:|---:|---:|---:|
| Short chat decode, tok/s | 26.2 | 26.4 | 26.4 | 30.9 | 30.8 | 34.1 | 32.8 |
| 4K prompt, tok/s (TTFT s) | 471.4 (8.57) | 466.1 (8.67) | 488.0 (8.27) | 494.4 (8.16) | 473.1 (8.53) | 396.6 (10.17) | 396.6 (10.17) |
| 16K prompt, tok/s | 1,018.4 | 1,018.4 | 1,013.7 | 1,026.4 | 1,019.6 | 789.1 | 790.0 |
| 32K prompt, tok/s (TTFT s) | 1,309.0 (24.54) | 1,311.8 (24.49) | 1,345.2 (23.88) | 1,344.9 (23.88) | 1,310.9 (24.50) | 1,007.2 (31.87) | 1,006.4 (31.89) |
| Agent: cache hit / turn TTFT / decode | 71.0% / 3.78 s / 35.7 | 71.0% / 3.75 s / 39.7 | 71.0% / 3.39 s / 41.9 | 71.0% / 3.43 s / 40.8 | 71.0% / 3.74 s / 41.3 | 70.9% / 4.02 s / 52.9 | 71.0% / 4.04 s / 52.9 |

Prompt ranges within a launch were within ±1% (for example A run 1, 32K: 1,305.8-1,317.5). The short-chat decode ranges were wide (for example 20.5-28.7 and 24.4-34.5), and the median moved by 4.5 tok/s between two launches of the same order.

**Not tested**
- Helper mode with the cards the other way round (3060 main, 5060 Ti helper), and helper mode without `--remote-expert-opt`.
- A manual `--layer-split`, `--kv int8`, `--vision`, and other models or quants.
- Longer prompts than 32K, concurrency, and a long soak.
- A single-card run with the same tool on the same day. As a rough reference only, the 5060 Ti alone with my daily config (different tool) reads a 31K prompt in about 31.5 s (about 990 tok/s), with 32-34 tok/s decode.

During the A, B and C runs, a low-priority (`ionice -c3`, `nice 15`) rsync of non-model files from the same NVMe to a NAS was running. It was paused during the helper runs. A and B gave the same numbers in their first and second launches, so I do not think it moved the results.

The tool's per-run reports (README.md, results.json, engine log excerpt) are available on request.

Related on this site

Quick links to related pages.