#6559 [Benchmark] 70-minute soak on a two-card layer split (RTX 3060 + RTX 5060 Ti, 0.2.18): no errors, no slowdown, temperature-0 answers vary

open · @Norhiro · 1 commentaires · Sur GitHub

BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsDocumentation

Description

Answers the TEST_REQUESTS item "A long soak (an hour or more of back-to-back requests) on a split or resident setup".

**Build and machine**
- Strata v0.2.18 (`6e38f588a`), engine 0.1.64, built from source with `-DSTRATA_ENABLE_CUDA=ON -DSTRATA_PORTABLE=ON -DCMAKE_CUDA_ARCHITECTURES="86;120"`, nvcc 13.3, g++ 15 (the AVX2 build). Served by the release's `terminal/serve/server.py`.
- RTX 3060 12 GB on PCIe 3.0 x16 and RTX 5060 Ti 16 GB on PCIe 3.0 x8 (X299), driver 615.71.09. Intel Core i7-7800X (6 cores / 12 threads, one NUMA node), 96 GiB DDR4 (speed not checked). Fedora 45, kernel 7.2.15. Both cards also hold a remote-desktop session (~400 MiB).
- Model: Qwen3.8-Flash-Next NVFP4-GPTQ pack (`--pack`, `--native` dense GGUF), MTP head, `--spec 4 --spec-min-p 0.5`, 64K context, no images.
- Config: `"gpu": [1, 0]` (3060 first), `"layer_split": "auto"`, `STRATA_PREFILL_AUTO_MAX=16384`, `STRATA_PREFILL_CPU_SHARE=auto`. The engine picked `layer split auto: K=16`, `prompt chunk auto: 16384 tokens`, expert cache 3118 slots (8.03 GiB) on the first stage.
- Engine arguments: `--pack PACK --native DENSE.gguf --max-context 65536 --prefill auto --mtp MTP --expert-profile terminal/data/expert-profile.bin --expert-cache auto --spec 4 --spec-min-p 0.5 --native-dense-gguf DENSE.gguf --ple-gguf UD-Q4_K_XL-00002-of-00004.gguf --embd-gguf token-embd-bf16.gguf`

**Load**
70.1 minutes, 948 requests back to back with no pause, one at a time, `/v1/chat/completions` streaming, thinking off. A loop over five kinds:
- `short`: the same Japanese question every time, temperature 0, up to 200 tokens (79 requests)
- `chat`: a three-turn conversation, temperature 0.7, 160 tokens (237)
- `long16k` / `long32k`: summarise a fixed 15,436 / 31,565-token text, temperature 0, 200 tokens (79 each)
- `agent`: a six-turn conversation that grows from 7.3K tokens, temperature 0, 200 tokens (474)
nvidia-smi every second (temperature, power, clocks, VRAM, throttle reasons, PCIe width), RAM every 10 s.

**Results**
- **0 failed requests.** No Xid / NVRM / AER lines in the kernel log during the run. The engine log has no errors.
- **No slowdown.** Decode tok/s, mean per kind, and first third → last third of the run:

| Kind | n | Decode mean | First third → last third | Time to first token, median / max |
|---|---|---|---|---|
| short | 79 | 33.0 | 32.8 → 33.3 | 0.69 / 1.03 s |
| chat | 237 | 39.3 | 39.0 → 39.4 | 0.47 / 1.79 s |
| long16k | 79 | 36.3 | 36.1 → 36.3 | 0.52 / 16.2 s |
| long32k | 79 | 40.9 | 41.1 → 40.9 | 0.43 / 25.1 s |
| agent | 474 | 42.8 | 42.9 → 42.7 | 0.54 / 10.6 s |

  The long texts repeat, so after the first read (16.2 s and 25.1 s) the prompt is served from the reused prefix (0.4-0.5 s): this soak exercises decode and prefix reuse more than repeated cold prefill.
- **Cards**: 5060 Ti max 78 °C, mean 66 W, SM clock 2,934 → 2,942 MHz (first / last 10 minutes), VRAM 15,237-15,307 MiB, x8 throughout; 3060 max 76 °C, mean 62 W, 1,790 → 1,782 MHz, VRAM 11,254-11,328 MiB, x16 throughout. No thermal slowdown bits; the 5060 Ti reports `0x400` in `clocks_throttle_reasons.active` the whole time (I did not look up that bit), the 3060 hit `SwPowerCap` for 19 s.
- **RAM**: used memory rose about 3 GB in the first 10 minutes and then stayed between 81.3 and 82.2 GB to the end: no sign of a leak.

**Temperature-0 answers are not identical**
The `short` request (same prompt, temperature 0, 79 times) gave **22 different answers**: the most common one 20 times, the next 11. All share the first two sentences and part at the third (one word choice, then different continuations). The variants are spread over the whole run, not drifting with time. The long summaries gave 56 and 71 different texts out of 79 each. My guess is the decode miss split (CPU lane vs GPU lane rounding a missing expert differently, as the comment at `pcie_share_gfx12_default` says) together with the expert cache changing under load, but I have not tested that. If greedy answers are meant to be repeatable on a split, this may be worth a look; if not, maybe a line in the docs.

**Not tested**: resident mode, one card alone, longer than 70 minutes, concurrent requests, images, other models.

Run with help from Claude Code (Anthropic), which wrote the load script and collected the numbers; I checked the results.

Sur le site

Liens install, modèles, releases.