#6360 [Benchmark] 2x RTX 3090 / Qwen3.8-Flash-Next / IQ3_S / standard-v1 draft protocol (MTP + DFlash2)

open · @michaeltg17 · 0 comentarios · En GitHub

BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quantsSecurityDocumentationWindows

Descripción

## Problem and result

Draft-protocol pilot: no `strata-qwen38-thinking-v1-draft.1` result exists for 2× RTX 3090 (sm_86) + 128 GB RAM on Windows/CUDA-13. This PR adds **two complete standard-suite runs on the same machine the same day, the only difference being the drafting engine** (MTP vs the DFlash2 drafter — the latter is what the 0.2.16 CLI downloads and uses by default today), plus a clearly-labeled loaded-machine quick run.

Headline medians (freshly loaded engines; short-decode / prompt-4K / prompt-16K):

| Drafting | Decode tok/s | Prompt 4K / 16K tok/s | TTFT 4K / 16K | Concurrency (aggregate ×3 rounds) | Sanity |
| --- | --- | --- | --- | --- | --- |
| MTP `--spec 4`, 3 slots, VMs stopped | 81.8 | 983 / 1,960 | 4.20 / 8.41 s | 80.4 / 81.7 / 85.0 (cap 3) | 12/12 |
| DFlash2 `--spec 6`, 1 slot, VMs near-idle | **118.5** | **1,371 / 2,547** | 3.02 / 6.45 s | 115.9 / 101.8 / 105.7 (cap 1) | 12/12 |

Both standard runs: `Status: complete`, 21/21 required requests, 0 failed, 0 skipped, optional concurrency stage included. Long-context recall **6/6 needles** at 32k and 128k (separate instances, labeled in Validation).

## Related discussion

none (results-only submission, per docs/COMMUNITY_BENCHMARKS.md and the Submit and review section of docs/BENCHMARK_PROTOCOL.md).

## Approach

Folders added under `docs/bench-results/`:

- `2026-11-20-standard-rtx-3090-plus-rtx-3090-e2f62039/` — standard run **A** (MTP drafting). Status complete; `results.json`, `outputs.jsonl`, `engine-log.txt` (byte-identical to the runner's `engine.log`, renamed only because the repo's `.gitignore` drops `*.log`; sha256-verifiable via the `artifacts` list), `config.json`, `provenance/`, plus two needle evidence files.
- `2026-11-20-standard-rtx-3090-plus-rtx-3090-e32b1be3/` — standard run **B** (DFlash2 drafter), same protocol, same day, hours later. Status complete; same file set (engine-log.txt sha256-verifiable); its README's "Author's note" carries the A↔B comparison.
- `2026-11-20-community-2x-rtx-3090-qwen3-8-flash-next-iq3-s/` — `tools/strata_bench.py` quick run taken right after setup with the VMs running; its README carries an author addendum marking it a loaded-machine cross-check that the standard runs supersede.

Common setup (both standard runs): custom track — Qwen3.8-Flash-Next GSQ-RCO IQ3_S (ISTA-DASLab, pinned rev `ed59f92082b1e93c0e96d60a8b11aab089b52f09`, SHA-256 verified by the runner at run time), engine **0.1.62-windows-x86_64-cuda-cuda13 release build**, machine calibration (`--prefill-pipe 1536`, `expert-profile.bin`), **experimental speed projection ON** (refusal-direction control vector, `--cvec-mode project --cvec-dir per-layer`, layers 4–44 — run B's engine log shows `41 steered`), vision on (`--vram-reserve-mib 700`), `--kv int8 --kv-resident 32768`, context 204800, layer split auto over both cards.

Per-run differences (declared): run A — MTP `--spec 4 --spec-min-p 0.5`, server capacity 3, engine binary built from Strata commit `55a557bb6d77facc73d291c3f9894289496c770c` (source checkout), **VMs stopped**. Run B — DFlash2 drafter (`--draft dflash2`, q4_0 pack + selector, `--spec 6 --draft-conf 0.35 --draft-fallback none`, serial), server capacity 1, engine from the 0.1.62 component-store release (commit `64524fb9fc82222ddce554c0643b64f056ec8366`), **VMs running near-idle** (~95 GB RAM free). The model files, quantization and hashes are identical across runs.

## Validation

Environment for all commands: Windows 11 10.0.26300, driver 616.92, CUDA 13 (driver supports 13.4), 2× RTX 3090 24 GB (PCIe gen4 x8 negotiated), Ryzen 7 9800X3D, 128 GB RAM, NVMe; otherwise idle desktop. Run A: Hyper-V VMs stopped. Run B: Hyper-V VMs running at near-idle load.

**Passed:**
- Run A — `strata bench run --profile launch-iq3_s-200k --track custom --track-notes "…" --model-revision ed59f92082b1e93c0e96d60a8b11aab089b52f09 --quantization IQ3_S --custom-pack "…" --background "Windows 11 Hyper-V host; Ubuntu Server + Talos VMs stopped for the run; otherwise idle desktop" --optional-concurrency --load-timeout 3600`; full form recorded in results.json `command`. Observed: `complete`, 21/21, sanity **12/12**, 2 min 34 s after model load; concurrency rounds 80.4 / 81.7 / 85.0 tok/s aggregate (capacity 3).
- Run B — same command with DFlash2 track-notes/background (`"… VMs running at near-idle load (not stopped for this run) …"`); the profile (`launch-iq3_s-200k`) carried the DFlash2 args. Observed: `complete`, 21/21, sanity **12/12**, **1 min 39 s** after model load; concurrency rounds 115.9 / 101.8 / 105.7 tok/s aggregate (capacity 1); engine log confirms `control vector mode = project, dir = per-layer, layers 4..44 (41 steered)` and the DFlash2 draft head loading.
- `python terminal/tools/needle_bench.py --url http://127.0.0.1:9931 --lengths 32k,128k --depths 10,50,90 --out needle-2026-11-20-2x3090-engine-0.1.62.json` — production server after the 0.1.62 store update (DFlash2, ESP off at that moment), light operator agent traffic. Observed: **6 of 6 found**, prompts 27,870 and 106,321–106,323 tokens, 12–48 s each.
- The same needle command on engine **0.1.59** (MTP, earlier the same day, before the update): **6 of 6** found, 16–80 s each — kept as `needle-2026-11-20-2x3090.json`.

**Failed:**
- none observed (warm-up and the performance/concurrency requests that hit their output caps are `truncated` per the protocol, not failures; they are itemized in the report READMEs).

**Not run:**
- 32K/128K prompt-throughput stages — the draft.1 protocol suite as run covers short/4K/16K; 128K is covered here only by recall (needles), not by throughput.
- Source revision of the original BF16 weights — not recorded on disk (the MTP draft tensors were fetched by SHA-pinned HTTP range reads) → `source_revision: null` with reason in results.json.
- DFlash2 with the speed projection off, and any run on 0.1.59 as a protocol run — unverified; all headline numbers are for the configurations above.

## Limits

- PCIe negotiates **gen4 x8** for both cards on this board (setup warned; not verified under load).
- **Speed projection (refusal-direction control vector) is ON in both standard runs** — decode/prompt numbers belong to that configuration; stated in the track notes and READMEs.
- Run B ran with the two near-idle Hyper-V VMs up (run A had them stopped); prefill/decode are GPU-dominated, but this is the one uncontrolled variable between A and B.
- The two standard runs also differ in engine build lineage (checkout build vs component-store release, both version 0.1.62, commits in the READMEs).
- The needle passes are separate engine instances (labeled above and in the README) — recall, not throughput, at 128K.
- The community quick run is intentionally kept as a loaded-machine datapoint (VMs running, right after setup); the standard runs supersede it for comparison.

En el sitio

Enlaces a install, modelos, releases.