#2525 bench: community reports - RTX 4090 and Minisforum MS-01 + 4x Tesla V100 SXM2 (NVLink), Flash-Next at 204800 / 262144, engine 0.1.44.1

Descrição

Two reports of the same release on two different machines, measured 2026-10-14/15 with **one method, one prompt generator and one case set**, so they can be read side by side. Each is self-contained: the previous version was re-measured in the same session, and a same-binary control gives the noise band that decides what is a change.

- `bench/results/2026-10-14-community-rtx4090-iq3xxs-200k-code/` - RTX 4090, IQ3_XXS, context 204800, code-explanation prompts; the follow-up of the [2026-10-08 report](https://github.com/Niko1221/Strata/tree/main/bench/results/2026-10-08-community-rtx4090-iq3xxs-200k-code) (same PC, same model, same config), so it covers 0.1.41, 0.1.42, 0.1.43/0.1.43.1 and 0.1.44/0.1.44.1 at once
- `bench/results/2026-10-14-community-rtx4090-iq3xxs-200k-ru/` - the same sweep with Russian prose
- `bench/results/2026-10-14-community-4x-tesla-v100-iq3-s-262k-code/` - a **Minisforum MS-01 mini PC driving a 4-slot SXM2 NVLink carrier board** (4x Tesla V100-SXM2-16GB) with `--layer-split auto`, IQ3_S, context 262144, measured out to **260331 prompt tokens (99.3% of the window)**; first report from this machine in this format

The reason for publishing them together is that they are the positive and the negative half of one claim: **the two defaults that turned on in 0.1.41 and 0.1.42 are adaptive.** On the single-card RTX 4090, which streams experts from NVMe, they give +31% short-prompt reading and +13.8% decode, and both effects were isolated to their own feature by turning each one off. On the four-card V100 machine, where 98% of the experts are resident, the engine's own start log turns `STRATA_ROUTE_TAIL_SKIP` off by itself and never arms the CPU prefill share - and neither gain appears. That is the designed behaviour, and now it is measured on both sides.

## Hardware, model, configuration

**Machine 1 - RTX 4090.** RTX 4090 24 GB (23028 MiB), Ryzen 9 7950X, 46464 MiB RAM, NVMe (ADATA LEGEND 960), Ubuntu 26.04.1, kernel 7.0.0-38, driver 610.57.04, CUDA 13.4. Model `qwen3.8-flash-next-iq3_xxs`, context 204800, INT8 KV, `--mmap-experts`, expert cache auto (6986 slots / 11.24 GiB), MTP draft pack, `--spec 4`, `draft_vocab=cyrillic`, experimental speed projection (control vector, layers 4-44), `--vram-reserve-mib 989`, `--pool-workers 10`, and `env: {"STRATA_PF_FUSED": "1"}` in the server config. Engine **compiled from source** for `sm_89` with CUDA 13.4 - the release publishes Windows archives only, so this is the only path on Linux; 0.1.38 … 0.1.40.3 and 0.1.44.1 in these reports were all built the same way on this machine.

Unchanged from the previous reports: the launch command (compared line by line in the server log), the server config files, the learned expert profile, and the engine's auto prompt chunk - `8192 tokens, a 96-slot ring` in 0.1.38 through 0.1.44.1 - so the numbers are comparable with the earlier reports from this PC.

**Machine 2 - a Minisforum MS-01 mini PC plus a four-slot SXM2 carrier board.** Host: MS-01 (DMI: "Micro Computer (HK) Tech Limited" / "Venus Series", board AHWSA, BIOS AHWSA.1.22 of 2024-03-12), i9-13900H (14 cores / 20 threads, Raptor Lake-P), 96 GB DDR5, 2 TB NVMe (SOLIDIGM P44 Pro), 2x 10GbE SFP+ + 2x 2.5GbE, Ubuntu 26.04.1, kernel 7.0.0-38, driver 580.178.04. **RAM is not a constraint: ~16.7 GiB in use with the model loaded, peak 17.5 GiB of the 96 GB installed (under 19%).**

The GPUs are not PCIe cards in a server: they are four SXM2 modules on a **Spring V100 SXM2 4-GPU carrier board (model V100-SXM2-NVLINK_D V2.10)** - NVLink 2.0 in NVIDIA's Hybrid Cube Mesh, host side behind a Broadcom **PEX8748** Gen3 switch with two external SFF-8654 channels (single x16 or 8+8 bifurcation), the modules liquid-cooled on a 360 mm AIO loop as the board's manual requires. The board ships with a `PCIe x16 -> 2 x SFF-8654` transition board; **only one cable is used here, and the link runs as x8, because the MS-01's full-length slot is wired electrically as x8** - measured, not assumed: `/sys/bus/pci/devices/0000:00:01.0` reports `max_link_width = 8`, the negotiated link is 8.0 GT/s x8, the board's x16-capable upstream port negotiates x8 Gen3, and each of the four GPUs behind the switch also negotiates x8 Gen3. So four x8 GPU downlinks share one x8 host uplink (4:1 oversubscription, ~7.9 GB/s shared), while **NVLink is fully up: 6 links per card at 25.781 GB/s, `nvidia-smi topo -m` reports NV2 between every pair** - ~155 GB/s per direction per card. Whether Strata's inter-layer traffic goes over NVLink or the PCIe switch was probed - see below.

Model `qwen3.8-flash-next-iq3_s`, context 262144, INT8 KV, `--mmap-experts`, `--vram-reserve-mib 512`, `--pool-workers 12`, `--spec 4`, `draft_vocab=cyrillic`, the same kind of control vector and online-learned expert profile. Engine **built on the machine with a private CUDA 12.8 toolkit and a private gcc-14** for `sm_70` (`STRATA_EXPERIMENTAL_SM60=ON`, `engine-cuda12/BUILD.json` records `"source": "local"`); 0.1.43 and 0.1.44.1 were both built with that same toolchain, and the two trees' server configs were compared before the swap and are **byte-identical**. Sizing from the start log: `layer split: CUDA3 runs layers 37-47, expert cache 5442 slots (11.37 GiB)`, `token graph hit path: 24088 resident experts`, `layer split: 98% of the experts resident, the prompt path's streamed ring 96 slots`, and the prompt path borrows cache slots from the other three cards. **The prompt chunk is the same `8192 tokens, a 96-slot ring`** as on machine 1, which is what makes the two reports method-comparable.

**An interconnect probe was run on machine 2** (`nvlink-probe.py`, raw data `nvlink-probe-phases.json`). DCGM's NVLink byte counters are not obtainable there — DCGM 4.7.0 has no profiling module on that host, and that does not change with a root host engine — so the probe measures the other side of the same question: any byte not going over NVLink has to cross the card's PCIe link, and those counters work unprivileged. Result: idle is exactly 0 on all four cards; a generation request moves ≤ 0.7 GB per card (matching `docs/MULTI_GPU.md`: a few hundred KB per verify window); **a 125 351-token prompt moves 62–78 GB into and 7–12 GB out of every card, sustained 1.2–1.6 GB/s with peaks of 5–6 GB/s against the ~7.9 GB/s ceiling of the Gen3 x8 link the machine has**. The engine's P2P prompt routes are not enabled in this configuration (`STRATA_PREFILL_HELP` unset, no `--peer-device`, and `src/prefill/prefill.cpp:1591` takes the mapped-host route `whether or not the cards could reach each other (measured there)`). So on this machine the prompt path is PCIe-bound, and that is a plausible mechanism for the small long-prompt regression — stated as a hypothesis, not separated by these runs.

**P2P itself was then measured directly** (`p2p-probe.cu`, results in `p2p-probe-results.json`), because the code comment behind `pp->p2p = false` records that the host route was chosen from measurements on **2x RTX 3090, PCIe 4.0 x16 each, no P2P** (`src/prefill/prefill.cpp:1560-1570`) — i.e. on hardware where the P2P branch could not be turned on at all. On this board it can be: `cudaDeviceCanAccessPeer` is 1/1 on all six pairs with `AccessSupported=1`, `NativeAtomicSupported=1`, `PerformanceRank=0`, NVML reports `NVLINK=OK` for every pair, and **peer copies run at 38.4 GB/s idle and 29.4–29.9 GB/s while the server is doing a 31375-token prefill** — a bandwidth this PCIe fabric cannot produce (7.9 GB/s per card behind one shared x8 uplink), which is also indirect evidence that an enabled P2P route really would use NVLink here. Enabling peer access costs **0 MiB** of device memory, so it is not blocked by the 256–1350 MiB of VRAM free with the model loaded. What the prompt path actually moves is 62–78 GB *into* each card against 7–12 GB *between* cards, so the first thing a P2P route would change is the smaller flow; whether the dominant flow could also be served from a peer's resident copy is the question I cannot answer from the counters. A platform dossier — capability matrix, measured peer bandwidth under load, the byte budget per phase, the code map, and a validation protocol that works without DCGM counters — is in `nvlink-platform-dossier.md`, together with the probes. A branch is enough to test any of this here: the machine builds from source with a private CUDA 12.8 / gcc-14 in ~20 minutes and rolls back in 2.

**A follow-up run (2026-10-15) measured both fabrics and the two prompt-path knobs** — dossier §8, data in `interconnect-experiments/`. With the model unloaded, so that all six pairs can be measured: card↔card **38.8 GB/s on every pair in either direction**, host→card 6.29 GB/s one card at a time and **1.56 GB/s per card when all four stream at once** (6.29 / 4 = 1.57 — the 4:1 oversubscription measured rather than inferred). Summing the four cards per second during one 125351-token prefill: 5.25 GB/s in a median second, 7.67 GB/s at p95, 9.33 GB/s in the worst second, 33 of 75 seconds at or above 6 GB/s — against a nominal Gen3 x8 ceiling of 7.88 GB/s per direction, so a long prefill here runs at about two thirds of its host link. `--peer-device` is refused together with `--layer-split` by the argument parser itself (`strata generate: --peer-device cannot be combined with --layer-split (use one or the other)`), which closes the configuration question: in 0.1.44.1 no flag or environment variable makes a layer-split machine issue a GPU→GPU copy. `STRATA_PREFILL_HELP` and `STRATA_PREFILL_HELP_FRAC` were then run in two sessions each against a control at 120 / 2048 / 3072 / 4096 / 32768 / 131072 tokens: **no change in reading speed anywhere** (every difference inside the control band); the helper raised card→host traffic by 15–20 GB per request in one session and not in the next, and where it engaged it cost no measurable time. The one anomaly seen alongside it — two 14–21 s stalls at 131072 tokens with the reading rate unchanged — did not reproduce.

## Method (identical in both reports)

Full warm-up sweep run first and discarded (on machine 1 its first 4K read ran at 1522.7 tok/s with a 2.62 s TTFT against 2155.7-2166.5 tok/s and 1.85 s in the measured pass - the cold expert cache, not the version), then 4K / 16K / 32K / 64K / 128K and the top of the configured context, plus a generation-only case; 3 runs per configuration, medians reported; a random marker in every prompt so the prefix cache is never reused (the reused column reports 0 in every run of both machines); throughput from the engine's timing fields; TTFT over streaming; `tools/needle_bench.py` at 32K and 128K, depths 10/50/90; peak VRAM sampled every 2 s (per card on the four-card machine). The whole session on each machine - warm-up, measurement, control, previous version, opt-outs - ran as one detached process, so nothing was interleaved by the driver.

## What each folder contains

`runs.json` (0.1.44.1) plus reference runs, so each comparison is self-contained:

- `runs-0.1.40.3-previous.json` (machine 1) / `runs-0.1.43-previous.json` (machine 2) - the previous version, **re-measured in the same session**: its binary / whole tree was swapped in, the server restarted, the warm-up sweep run and discarded, then the same sweep measured, then swapped back;
- `runs-0.1.44.1-control.json` - a full repeat of the sweep on **the identical 0.1.44.1 binary**: the noise band of this method (machine 1: prompt -5.8…+2.1%, decode -6.7…+8.9% depending on the case; machine 2: prompt within ±0.3%, decode -4.4…+1.8%);
- machine 1 only: `runs-0.1.44.1-no-tail-skip.json` and `runs-0.1.44.1-no-cpu-share.json` - the same build with one new default turned off each, which is what turns "consistent with X" into "caused by X";
- machine 1 only: `runs-long-context.json`, `runs-long-context-control.json`, `runs-long-context-0.1.40.3.json` and the matching `needles-long-context*.json` - the same sweep extended to 156291 / 185531 / 200015 actual prompt tokens out of the confi