#2905 bench: 4x V100-SXM2 + NVLink platform dossier - P2P measured end to end, #2525's data-path question closed, a validated A/B stand (no engine changes)

Beschreibung

A follow-up to #2525 on the same machine (Minisforum MS-01 + Spring V100-SXM2-NVLINK_D carrier, 4x Tesla V100-SXM2-16GB, NVLink 2.0, engine 0.1.44.1 = `296b59d2`, IQ3_S, 262144). #2525 left one question its own counters could not answer - which flow the 62-78 GB per card during a long prefill actually is - and measured P2P at two sizes only. This folder is the rest of that measurement: the full P2P size ladder, the engine's own counters read against them, the SM70 kernel inventory, and an A/B stand whose noise floor has itself been measured. **No engine changes - measurements, probes and scripts only.**

To be explicit about intent, because I would not want this read as a request: this is not a push for V100 support. The support matrix starting at sm_75 and Volta being a community-tested target in [docs/NVIDIA_V100.md](../main/docs/NVIDIA_V100.md) is the right arrangement for an architecture CUDA 13 no longer builds, and this machine runs a community build and will keep doing so. Everything below is the data half of a question I got curious about, written so that anyone can use a piece of it or ignore all of it.

Three reasons it may still be worth a minute:

1. The *Limits* section of `NVIDIA_V100.md` says a layer split over V100s was not measured. This is such a measurement - four cards, `--layer-split auto`, 262144 context, out to 125 011 prompt tokens, with the engine's counters in the frame.
2. Two of the findings are not about Volta at all, and would show up on any layer-split machine: the `auto` split search is degenerate on this build, and a different split is worth +4.5 % / +7.3 % prompt. Neither involves an sm_70 code path.
3. One transport fact is not about Strata either: on this board a kernel reading a peer's VRAM directly reaches 0.94 of a `cudaMemcpyPeer` copy, which changes what a P2P route would have to allocate - or lets it not allocate at all.

## Status labels used throughout

Every claim in the folder is marked **measured** (this machine), **calculated** (arithmetic on measured numbers), **hypothesis** (not tested) or **not measured**. Nothing is presented as a fact on the strength of the code alone. Where a claim rests on the code plus a match of volumes rather than on a counter, it says so.

## The interconnect, measured end to end

`p2p-results.md`, `p2p-matrix.csv`, `p2p-benchmark.csv` (211 rows: 17 sizes 4 KiB..256 MiB x 6 pairs x 2 directions, no gaps), probes in `probes/`.

- NVLink is fully up and healthy: 24 links, 2 per pair at 25.781 GB/s, P2P and native atomics on all 6 pairs, `CudaArrayAccessSupported = 1`, and enabling peer access costs **0 MiB** of VRAM.
- Peer copy: a **38.77 GB/s plateau**, identical on all six pairs in both directions (256 MiB), reproduced from #2525's 38.8 with a 0.08 % difference. Fixed cost **6.7 us**: `T = 6.7 us + size / 38.8 GB/s`; 50 % of the plateau at 256 KiB, 80 % at 1 MiB, >= 99 % at >= 32 MiB.
- **The 38.8 GB/s ceiling is per device, not per pair or per link**: six pairs running at once give 77.50 GB/s total, 12.92 GB/s per pair (a 3.00x fall). What sets the limit is still a hypothesis.
- **Direct kernel access to a peer's VRAM is a usable transport**: reading a peer's memory from a kernel 32.13 -> 36.25 GB/s at 32 -> 256 MiB = **0.838 -> 0.936 of a peer copy**; copy done inside a kernel 36.63; writing into a peer 36.64; two kernels reading at each other on one pair **65.1 GB/s**. 576 rows, 0 correctness errors, all 12 ordered pairs. This is the new result, and it is what decides the shape of the transport note below.

## The open question from #2525 (its §9.1): answered

#2525 asked which flow moves 62-78 GB into each card during a long prefill - expert streaming or activation staging. It is expert streaming, and three independent sources agree:

- the engine's own counter for a 125 011-token request: `for 269454.7 MB of expert reads`, with `the OS read 1.3 MB from storage (0 major faults)` - i.e. 269 GB taken from the page cache and pushed to the devices; a second session logs 269 472 MB;
- #2525's own PCIe counters for the same shape of request: 270.5 GB into the four cards;
- the slot arithmetic: the prompt path borrows 1954/1841/1706/1631 cache slots for its buffers while all four expert caches are filled exactly (5632/5632, 6656/6656, 6144/6144, 5469/6144), so ~1954 experts x ~2.05 MB x 16 chunks is ~64 GB per card, against 61.9-78.4 GB measured.

Activation staging between stages - the flow a P2P route would carry - is 32.2 GB per request by calculation from the code, which matches #2525's measured 35.2 GB TX within 9 %. So the dominant flow is host->card and the inter-card flow is ~10 % of the bytes.

**And on this machine the dominant flow cannot be moved to NVLink at all**, not as a limitation of the patch but of the geometry: for NVLink to carry an expert, another card has to be holding it, and with a layer split each stage holds only its own layers, all four caches are filled exactly, and 4 x 16 GB of VRAM cannot duplicate 50.3 GB of experts. That is a calculation on measured geometry, and it is the reason the notes below do not propose "put expert streaming on NVLink".

## Two findings that are not about this hardware

**(a) The `--layer-split auto` search is degenerate on this build, and the choice is worth several percent.** Thirty search lines in the working log give four different splits, all scored `predicted 28.4 ms` (`K=12,25,37` x20, `11,24,36` x5, `11,24,37` x4, `11,25,37` x1); the choice moves between runs of the same binary and the same config. Pinning `12,25,37` instead of `auto` measures **+4.48 % prompt tok/s at 31 307 and +6.66..+7.33 % at 122-125k, TTFT -4.2..-6.8 %**, in two independent pairs of cells. The mechanism is visible in the counters: +512 cache slots on CUDA0 (5632 -> 6144), coverage on the narrow CUDA3 89.0 % -> 96.6 %, residency 97 % -> 99 %, -13.1 GB of expert stream per request. The search criterion does not see the cost of the LM head on the last stage, the prefill pipeline, per-card residency, or the cost of streaming.

This is also why this machine's profile sits 1.5-5.7 % below #2525's at the **same binary (sha256 `bdfec9f5..`) and byte-identical config**: the cause was found by pinning the split, and it is the `auto` choice (`11,24,36` vs `12,25,37`) - not the version, not the hardware, not background load, not warm-up. Anyone comparing two reports from one machine should know that the split search can be the variable.

**(b) A structural identity between VRAM and host bytes.** Each cache slot given to something else becomes a non-resident expert that is re-read on every 8192-token chunk, so for a prompt of length T: **1 MiB taken from the expert cache = (T / 8192) MiB of extra host traffic per request**. Two consequences that are arithmetic rather than opinion: moving the hand-off buffers into VRAM saves exactly the bytes it costs (net 0 on a machine with a full cache), while reading the peer's VRAM from a kernel, with no receiving buffer, is net -16.1 GB per request. Freeing VRAM the other way - not trimming CUDA0 at `auto` - is where the bytes actually are.

The calibrated price of a host byte here is about 3.9 GB/s (one pair of configurations: -13 967 MB of host traffic against -3.59 s of TTFT), i.e. 1 GiB of traffic ~ 0.27 s of TTFT ~ 0.5 % at 125k. That is one pair of points in a narrow neighbourhood and it overstates larger reductions; it is quoted so that any idea can be priced before it is built, not as a prediction.

## What is not proven, and said so in the folder

- **No NVLink byte counters exist on this machine** (DCGM 4.7.0 without the profiling module; NVML Sampling & Reporting absent in both 580.178.04 and 595.91.07). Every routing statement is indirect: code + the match between the engine's counters and `rxpci`/`txpci`. No tool ever showed a byte on a link.
- In 0.1.44.1 a layer-split machine issues **no GPU->GPU copy at all**: `--peer-device` with `--layer-split` is refused by the parser, and the stage-helper route is hard-wired `p2p = false` (`src/prefill/prefill.cpp:1591`). Whether a P2P route would be faster here is **not measured** - only that the transport is capable and what the byte budget is.
- `STRATA_PREFILL_HELP` / `..._FRAC` change nothing measurable on this machine (two sessions each), and `--peer-device` cannot be turned on for lack of free VRAM: both are 0 % here, and the folder does not pretend otherwise.
- **Decode is not a usable measure on this machine**: up to 7.8 % session-to-session spread with an unchanged config, because draft acceptance itself moves 3-7 percentage points. The folder reports decode only as a "must not get worse" control.
- Changing `--layer-split` changes the output on a long prompt: at 10 012 tokens the `auto` and `12,25,37` answers share a 32-character prefix out of 647; at 1259 tokens they are identical. Reported as an observation (F-8); the mechanism is not established, and it is why a placement change cannot be accepted by bit-identity alone.
- GPU3 is not reachable from an external CUDA context while the service runs (`cudaSetDevice(3)` -> out of memory, ~328 MiB free); with the service stopped it behaves exactly like the other three (<= 0.06 GB/s difference).
- Everything is one concurrent request. Concurrent serving was not studied.

## Volta compatibility, checked rather than assumed

`sm70-compatibility.md`, `correctness-report.md`, `correctness-matrix.csv` (41 rows: 27 PASS, 4 LIMITATION, 3 FINDING, 7 GAP). **2084 kernels from 62 cubin/PTX objects load with no error and no JIT**, resource limits are not exceeded (max 255 registers, 41 104 B shared), the SM70 paths are on by default, and the `>= sm_75` requirements are covered by fallback paths - a 20-row "component -> requirement -> check" table. The SM70 numeric primitives (HMMA, fp32/fp16 GEMM) match a CPU reference within `1e-4 * K`; P2P copy and cross-card synchronisation are correct 37/37. Greedy generation is bit-reproducible within one placement configuration and across service restarts; the **first request after any change to engine state returns different text**, and the expert-cache hit rate warms 98.8 -> 99.8 % over 2-3 requests - which is why the stand repeats a request until two `text_sha` match before timing anything.

## A stand that is already validated, in case a branch ever needs a run

`ab-stand.md` and `test-plan.md`, implementation in `scripts/`. It runs a 2x2 matrix plus a repeat of the base (`auto:base -> auto:patched -> split12:base -> split12:patched -> auto:base`), so the repeat sets the noise threshold for that run; it takes a manifest of sha256 for binary, config and scripts plus `build_info` from `/props`; it waits for three readiness signals rather than a sleep; it requires bit-identical output before benchmarking (and says why that check cannot apply to a placement change); it fixes output length with `logit_bias` on the EOS ids and checks `finish_reason`; and it extracts the counters that matter - `MB of expert reads`, `the OS read ... (major faults)`, cache slot counts, `the prompt path borrows ...`, `layer split auto:`, per-card `rxpci`/`txpci`.

Measured noise, so that a difference can be called a difference: **prompt tok/s 0.06 % at 122k, 0.14 % at 32k, 0.42 % at 4k between sessions**; the reproduction of the pinned-split gain in a second independent pair of cells was +4.48 % and +6.66 %. Significance thresholds used: |delta| > 1 % at >= 32k, > 2 % at 4k.

The machine is available and cheap to use: any branch builds here with a private CUDA 12.8 / gcc-14 in about 20 minutes, rolls back in two, and a full A/B run needs one ~40 minute service window. Happy to run anything you point at - and equally happy to run nothing. Nothing in this folder asks for a decision.

## Contents

`bench/results/2026-10-17-community-4x-v100-nvlink-platform/`

- `README.md` - index and the short engineering report: what is confirmed, what is not, the bottleneck, what is proven vs not proven, the order in which the notes sug