#6242 [Benchmark] 6x Tesla V100 16GB (sm_70): 0.1.62 parity on concurrent prefill - the +16% single-prompt gain does not transfer to 4x26K concurrency

closed · @Barmallini · 1 コメント · GitHub で見る

BenchmarksServer & APIMulti-GPUNVIDIA / CUDAModels & quantsLinux

本文

## Problem and result

The 0.2.16 notes quote a V100 gain of 17,464 ms -> 15,068 ms on a 26,000-token prompt (+16% over 0.1.58). On our 6x V100 box, reading the same 26K prompt size **four at once**, 0.1.62 lands at parity with 0.1.58 (843.7 vs 856.3 tok/s mean, ~-1.5%, at the edge of run-to-run spread) - no measurable transfer of the single-stream gain. Two useful positives for the sm_70 pile: the 0.2.14 own-kernel prefill regression reported in #6053 (-11% here) is **gone** in 0.1.62, and decode is unchanged (86.6 tok/s, 64% draft acceptance). We are staying on 0.2.16.

## Machine and config

- Proxmox VM, 6x Tesla V100 16GB (sm_70), 2x Xeon E5-2695 v4, 88 GiB RAM, Linux
- 0.1.62: the prebuilt `strata-engine-0.1.62-linux-x86_64-cuda-cuda12.tar.gz` (BUILD.json archs "60;61;70;75;80;86;120;89", CUDA 12.9) - no source build; sm_70 confirmed in the fatbin (54 ELFs)
- 0.1.58: v0.2.12 built from source, CUDA 12.9, `-DCMAKE_CUDA_ARCHITECTURES=70 -DSTRATA_EXPERIMENTAL_SM60=ON`
- Qwen3.8-Flash-Next GSQ-RCO IQ3_S: native pack + the model's own second GGUF shard as `--ple-gguf`, all 24,576 experts resident, layer split 9,18,27,35,43 + `--trim-stage-weights`, `--batch 4`, `--kv int8 --kv-resident 262144`, control-vector projection (layers 4-44), 262,144 context, conversation cache 20,480 MiB / 6 slots, `--spec 4 --spec-min-p 0.30`, `--pool-workers 29`
- Config deltas for the 0.1.62 arm: `--pcie-frac` removed (0.1.62 probes PCIe itself: "CUDA3 PCIe probe 3.2 GB/s -> pcie_frac 0.09"); `--vram-reserve-mib` removed from args (the server re-adds `--vision --vram-reserve-mib 700` on its own because the config has a vision section - the 0.1.62 engine accepts it even though `--help` no longer lists it)
- Engine env unchanged: `STRATA_PREFILL_LAYER_MAJOR=0`, `STRATA_NO_LARGEPAGES=1`, `STRATA_SLOT_CKPT_MOVE=1`, `STRATA_SLOT_COPY_DIRECT=1`, `STRATA_CKPT_POOL_READY=1`
- Start lines, 0.1.62 arm: `session is up (engine 0.1.62)`, `prompt kernels: every chunk runs 48 of 48 layers on the int8 kernels.`

## Method

4 concurrent chat completions of 25,962/25,963 tokens each (fresh nonce prefix per run, engine log confirms `0 reused` = cold read), metric is the engine serve line `prompt N tokens = 0 reused + N read in X ms (Y tok/s)`. 2 runs per arm; every slot reported. The first 0.1.62 slot (826.5) carried residual post-start warmup and is excluded from the mean, same as the 814.9 slot in the 0.1.58 reference. 0.1.62 measured today on a fresh engine start; the 0.1.58 reference is the Nov 16 rerun on this box (same config, same env, llama-side-service-free) from the #6053 follow-up.

## Measurements

| | 0.1.58 (Nov 16) | 0.1.62 (today) |
|---|---|---|
| prompt tok/s, run 1 | 857.5 / 861.5 / 862.3 (+814.9 warmup, excl.) | 826.5 (warmup, excl.) / 854.0 / 850.1 / 835.3 |
| prompt tok/s, run 2 | 847.4 / 853.5 / 855.0 / 856.9 | 845.1 / 853.8 / 847.3 / 837.5 |
| mean / median | 856.3 / 856.9 (7 slots) | 843.7 / 846.2 (7 slots) |
| wall time per request | 124.0 / 124.9 s | 124.8 - 126.0 s |

Decode probe (0.1.62, temp 0): 496-token prompt, 256 generated in 2,956 ms = **86.6 tok/s**, drafts accepted 159 of 249 (64%). Reference: this box historically served ~86 tok/s long answers on the 0.1.44-0.1.58 line with the same projection - parity, and also no visible transfer of the 0.2.14-line decode gains to this shape.

RAM: startup peak 77 GiB RSS, settles at ~60 GiB; after the two stress runs used 62 GiB / 23 GiB available - the same fresh-start profile as 0.1.58.

## Validation

- Passed: every request of every run returned a token; `--ple-gguf` with the model's own second shard boots on 0.1.62 (the #5942 fix is in this engine); the vision tower loads (956 MiB) and the MTP draft layer loads (835 MiB).
- Failed: nothing. One startup stumble was ours, not the engine's: the v0.2.16 checkout removes the tracked top-level `data/` (layout move, 4058aa782), so a config still pointing at `data/expert-profile.bin` dies with `read_expert_profile: cannot open`. Points to `terminal/data/expert-profile.bin` now.
- Not run: quality/KL between 0.1.58 and 0.1.62, single-prompt 26K latency on this rig (the release's +16% shape - we only had a window for the concurrent load that production actually serves), mixed concurrent decode, vision generation, longer soak.

## Limits

- One box, one quant, one shape. The comparison point is a day-before 0.1.58 rerun (same box/config/env), not same-day.
- The decode reference is an era number (0.1.44-0.1.58), not a same-day 0.1.58 measurement; stress generations are 1 token, so decode above rests on the single 256-token probe.
- -1.5% mean is within the spread this box shows between runs of the same engine; we do not claim it as a real regression, only that no gain of the release-notes size appears under 4-way concurrency.

## Related

- #6053 (closed): the 0.2.14 -11% sm_70 report and the community prefill pile. This is the promised-tuning follow-up on the same box.
- #6066: sm_75 arms (own int8 / MMQ) - on our shape the whole kernel question is moot now that 0.1.62 has no llama.cpp kernels and lands on parity by itself.

関連リンク

インストール・モデル・リリースへのサイト内リンク。