#6104 [Benchmark] RTX 3070 Ti + RTX 3070 (8 GB each, x16/x4): R-resident prefill, LM cache loans, and 0.1.60 migration
closed · @Burntgogi · 3 コメント · GitHub で見る
BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityWindowsLinux
本文
Report-ID: dual8g-iq3s-field-20261117 Custom field measurements on two 8 GiB Ampere cards with unequal PCIe links. An optional later-stage GPU-resident residual improved long prefill; a combined cache/PCIe configuration improved decode on the older engine. A port to engine 0.1.60 preserved aggregate throughput closely but failed several per-input regression checks, so the older custom engine remains deployed. These are existing measurements from November 14–16, 2026, **not a run of the draft standard benchmark, an official stock-versus-stock comparison, or a request to change upstream defaults**. No new performance measurements were made for this report. Numeric request records, settings and experimental source diffs are included below and in the evidence comments. Exact input/output text and the fixed expert profile are withheld for privacy, limiting independent reproduction. ### Hardware and fixed settings | Item | Configuration | | --- | --- | | CPU / RAM | Intel Core i5-12600K; 64 GB class, approximately 62.6 GiB OS-visible | | Logical stage 0 | RTX 3070, 8 GiB; initially measured PCIe 3.0 x4 | | Logical stage 1 | RTX 3070 Ti, 8 GiB; PCIe 4.0 x16 | | Layer split | Fixed 14 / 34, in the order above | | Software | Ubuntu Linux x86_64; NVIDIA 595.99.02; GCC 14 / CUDA 13, sm_86 source builds | | Model | Qwen3.8-Flash-Next GSQ-RCO IQ3_S; same model artifacts and expert profile within comparisons | | Context / KV | 131,072 maximum; int8 KV; resident window 8,192 | | Prefill / LM | Prefill chunk 4,096; layer-major target 8,192 | | Decode | MTP speculative length 4, minimum probability 0.5; two pipeline windows | | Common environment | Ring 48, I/O threads 4, routed stream 1, stage trim 1, MTP stage 0 enabled | RAM type/speed, storage type, exact model/pack revision and original GGUF filenames were not independently archived in this public export; they are unavailable rather than inferred. The machine retained its graphical desktop/remote session. Available external CPU-load samples are in the numeric exports; this was not an isolated idle-host benchmark. GPU power figures below are **limits, not measured energy or wall power**. ### 1. Later-stage residual residency: long prefill The starting point already used layer-major prefill. The local opt-in keeps a later stage's residual `R` on its own GPU across that stage's layers while retaining the pinned host hand-off between stages. It falls back to the host path if the full target cannot fit. It does not eliminate all inter-stage or expert transfers. Source base: [`712fec6`](https://github.com/Niko1221/Strata/commit/712fec6b4297c9cfd32ab43daaa2979a9aa01edf), engine 0.1.59. Power limits: Ti 230 W / 3070 210 W. Both reserves: 768 MiB. | Fresh prefill | Repeated comparison, median tok/s | Change | Repeats per arm | | --- | ---: | ---: | ---: | | 125,970 tokens | 1,139.26 → 1,345.71 | +18.12% | 3 | | 32,768 tokens | 1,052.44 → 1,220.10 | +15.93% | 5 | **The repeated comparison includes installed-binary versus rebuilt-binary effects.** A separate same-binary OFF/ON pilot had one pair per length: 1,136.26 → 1,342.18 tok/s at 126K (+18.12%), and 1,049.40 → 1,224.29 at 32K (+16.67%). This pilot supports the mechanism but is not a repeated same-binary performance study. In that 126K pilot, counted residual H2D copies fell from 180,580,556,800 to 5,159,444,480 bytes and residual D2H copies from 175,421,112,320 to zero. The counters exclude unrelated transfers. Controlled OFF/ON runs matched generated output and checkpoint checks. Residual dumps cover all FP32 dimensions **at every 64th token**, not every intermediate token. Unrestricted OFF/OFF output initially varied; controlled parity fixed adaptive swapping/PCIe sharing and serialized penalties in both arms. One exploratory local assertion failure is retained in the failure ledger. ### 2. Combined configuration A: decode on 0.1.59 This is a **combined configuration result**, not an isolated estimate for one flag. All arms already have the residual-residency patch. Power limits here and in the migration tests: Ti 260 W / 3070 240 W. | Arm | Independent per-stage PCIe update | Extra LM-workspace cache loan | First/later reserve MiB | Adapt interval / pool tasks | | --- | --- | --- | --- | --- | | Baseline R | Off | Off | 768 / 768 | Original defaults | | A-minus | On | Off | 768 / 768 | 8 / 64 | | A | On | On | 768 / 512 | 8 / 64 | The extra loan makes an LM workspace a disjoint temporary prefix inside the existing expert-cache loan, then restores borrowed expert blobs before decode. Ordinary upstream prefill cache lending predates this experiment. The PCIe change lets later-stage auto-share estimates update even when the first stage has no useful update; it does not replace the fixed layer split with an auto planner. Nine inputs (six fresh 4K, three cached 32K), two samplers, three arms and three rounds produced **162 scored requests plus 27 cache primes**. Output cap: 1,024. Decode rates are `sum(generated tokens) / sum(native decode seconds)`; they exclude prefill/loading time. They are not arithmetic means of request rates. | Sampler | Baseline R tok/s | A-minus tok/s | A tok/s | A vs baseline | | --- | ---: | ---: | ---: | ---: | | OMP-like: temperature 0.7, top-p 0.8, presence penalty 1.5 | 50.843 | 55.962 | 57.178 | +12.46% | | Neutral: temperature 1, top-p 0.95, top-k 20, penalties neutral | 50.011 | 52.253 | 54.238 | +8.45% | All four early-EOS requests were retained. The three arms each had one neutral D5 response of 34 tokens; A also had one neutral D2 response of 688 tokens. No favorable replacement requests were substituted. The original strict neutral tokens-per-window sign-count screen **failed on 8 of 9 inputs for A**. That failure is preserved; deployment followed separate functional/resource checks rather than retroactively relabeling the screen as passed. The loan was approximately 522 MiB. A's additional median process PSS was 274.48 MiB; minimum available system RAM sampled at 1 Hz was 5.621 GiB. Subsequent bounded checks passed 5/5 math cases, five forced tool calls, five OMP-wire-format read/edit/AST-test/final sequences, a 96K restore with 99.9929% reuse and three 126K recall markers. These are narrow checks, not a full OMP application productivity test or broad model-quality evaluation. ### 3. Port to engine 0.1.60: close aggregates, failed local gate Source base: [`09b6038`](https://github.com/Niko1221/Strata/commit/09b603886925994d0d4461857b01321ef3aa431c), Control 0.1.60.1. The evaluated candidate carries the local R/loan/PCIe changes, uses a full **MMQ-enabled build with `STRATA_PF_FUSED=0`**, and retains A's settings. It is not the default own-kernel release path. Four inputs (D1/D5/D6 fresh 4K, L2 cached 32K), two samplers, three rounds, 1,024-token cap: 48 scored requests, plus separately labelled warm-up, primes, 18 fresh prefill requests and post-checks. Order was old/new, new/old, old/new. All 24 paired input payloads matched; **0/24 generated texts matched across versions**. A 370-token early-EOS response was retained. Output/routing and MTP acceptance can therefore affect decode results. | Metric | Old custom A | New custom A, later reserve 512 MiB | Change | | --- | ---: | ---: | ---: | | OMP-like decode tok/s | 56.683 | 55.836 | −1.49% | | Neutral decode tok/s | 55.805 | 57.655 | +3.31% | | Fresh 4K prefill tok/s, median of 3 | 520.30 | 519.70 | −0.12% | | Fresh 32K prefill tok/s, median of 3 | 1,245.38 | 1,243.91 | −0.12% | | Fresh 126K prefill tok/s, median of 3 | 1,358.03 | 1,357.33 | −0.05% | The local decision rule also rejected per-input regressions exceeding 3%. OMP-D5 (−5.77%), OMP-D6 (−4.23%) and neutral-D6 (−4.52%) failed it. Thus the aggregates do not establish equivalence. Three repeats and changed outputs do not identify a kernel regression or a statistically precise 3% boundary. The stock 0.1.60 own-kernel build passed basic 4K/32K API smoke checks, but layer-major declined the fused layout. An experimental own-layout adapter enabled the LM path but could not lend the complete extra prefix at the fixed 4,096/8,192/512 settings. The MMQ candidate passed controlled 16K/32K OFF/ON output/residual checks and bounded math/tool/agent/cache checks. These feature fallbacks were not CUDA crashes. Smoke timings are not presented as a stock performance comparison. ### 4. Memory allocation observation relevant to #6084 `STRATA_MTP_STAGE0=1` was enabled throughout. At the same 768/512 MiB reserve flags, the Ti's expert slots dropped on the new engine while free memory rose: | Configuration | Ti expert slots | Ti startup free MiB | | --- | ---: | ---: | | Old custom A | 1,495 | 613 | | New custom A, later reserve 512 | 1,384 | 837 | | New custom A, later reserve 288 | 1,492 | 613 | The last row is a local **224 MiB reservation workaround using the same new binary**, retaining the 512 MiB physical startup floor. It is consistent with the dead MTP-head reservation discussed in [#6084](https://github.com/Niko1221/Strata/pull/6084), but **the PR's actual predicate change was not applied or tested**. Restored capacity did not ensure faster execution: OMP-like 55.612 tok/s (−1.89% vs old A), neutral 55.168 (−1.14%), with per-input regressions down to −6.75% and prefill medians down 3.34–4.41%. The workaround was rejected. Its new arm has 24 newly scored requests; the 24 old controls are reused from the preceding migration comparison, not independent fresh controls. These results are not evidence that #6084 regresses performance. ### Upstream relationship and scope Credit for layer-major prefill belongs to @vizdatom ([#1670](https://github.com/Niko1221/Strata/issues/1670)); @Norhiro's split-stage work is discussed in [#3976](https://github.com/Niko1221/Strata/issues/3976). This report adds one unequal-link, equal-VRAM CUDA case and optional local extensions. It is related to [#5265](https://github.com/Niko1221/Strata/issues/5265), but does not validate unequal-VRAM auto-planning or both GPU orders. #6084 already addresses the reservation predicate; no competing fix PR is proposed. At report preparation, [v0.2.15](https://github.com/Niko1221/Strata/releases/tag/v0.2.15) ships Control 0.1.61 with the same 0.1.60 engine files as v0.2.14. Control 0.1.61 itself has not been tested here. All engine diffs are optional experimental source evidence, under the [upstream license](https://github.com/Niko1221/Strata/blob/main/LICENSE), not a merge-ready code submission: HIP/SYCL/Metal were not built or tested. No continuous one-hour same-engine soak, 144K context qualification, or new default-kernel performance sweep is claimed. Measurements from different phases have different baselines and power limits; their percentage gains must not be multiplied. No matched claim against the former Radeon configuration is made. The useful upstream questions are whether the later-stage R-residency approach merits broader controlled validation, and whether extra LM-workspace lending should be reviewed separately from per-stage PCIe adaptation. ### Sanitized evidence The collapsed records retain numeric requests, including warm-up/prime/support rows, EOS results and rejected candidates. Missing measurements are empty CSV fields. `scored=1` identifies the original local scoring subset; R's varied support requests must not be pooled as one benchmark. Case identifiers are neutral labels. Timings use native decode milliseconds and tok/s; memory uses GiB/MiB. `samplers.json` records the request overrides; omitted sampler fields retain the same server defaults within each comparison. The historical records and provenance are below. Two evidence comments contain migration records/settings and the source diffs. No prompt/output text, private expert-profile binary/hash, raw logs, session transcripts, credential fields, home paths, hostnames or device serial identifiers are included. Public artifact checksums identify only the sanitized files and softwar
関連リンク
インストール・モデル・リリースへのサイト内リンク。