#5442 [Benchmark] RTX 5080 / Qwen3.8-Flash-Next-GSQ-RCO / IQ3_S / strata-qwen38-thinking-v1-draft.1

closed · @teohhc · 0 评论 · 去 GitHub 看

BenchmarksSetup & installNVIDIA / CUDAModels & quantsSecurityDocumentationLinux

说明

## Problem and result
The benchmark-results protocol (`strata-qwen38-thinking-v1-draft.1`, docs/BENCHMARK_PROTOCOL.md) is in its draft phase and upstream has no protocol-conformant submission yet (all 145 existing `bench/results/` dirs predate it). This PR contributes the first one: a complete standard-track run of `Qwen3.8-Flash-Next-GSQ-RCO` IQ3_S on an RTX 5080, labelled with the draft protocol ID as the protocol requires for pilot data.

Report directory: `bench/results/2026-11-07-standard-rtx-5080-af567e8a/` — `results.json`, `README.md`, sanitized `config.json`, `engine.log`, `outputs.jsonl`, plus small provenance files. No code changes, no weights, no generated packs.

## Related discussion
none

## Approach
Follows the protocol's Submit and review rules exactly: one hardware/model/quant/configuration combination, every scheduled outcome retained (nothing dropped), artifacts inspected for redaction before publishing and hashed afterwards, results-only diff. Track is `custom` because the configuration is the setup-wizard profile `iq3_s-64k`, not stock release defaults; the README's operator-notes section records the profile, model provenance (HF `ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF`, revision `ed59f92082b1e93c0e96d60a8b11aab089b52f09` pinned by setup.py of v0.2.11, shard sha256s matching that revision's catalogue), and software versions (engine 0.1.57, runner v0.2.11 at tag commit `c3b3f5035506f3e966f1b4b7b188aeaabae92f1d`).

## Validation

**Passed:**
- `sha256sum` over every file exactly as committed in this branch: all six `artifacts` entries in results.json match (clone at `54828c7`, this commit `963b79f`, Arch Linux).
- `strata bench report bench/results/2026-11-07-standard-rtx-5080-af567e8a` → "Status: complete. 21 of 21 required requests finished, 0 failed, 0 skipped"; sanity 12/12 (instruction 4/4, retrieval 4/4, tool calls 4/4); per-case speed tables match results.json.
- Redaction check: engine.log shows only `/home/<user>/`-sanitized paths (runner-applied); no weights or generated packs staged; `git diff --stat` = 8 files, +2439/-0.

**Failed:**
- none observed

**Not run:**
- No re-benchmark of the suite (the engine on the bench machine is in active use; the archived run is the submission). Per-request evidence for re-verification is in outputs.jsonl. No build/test checks apply: results-only PR, no code touched.

## Limits
- Draft-phase data: `model.revision` / `tokenizer.revision` are null with reasons in JSON ("not pinned (draft)"); the actual HF pin used is recorded in the README prose instead.
- 7 of 9 performance requests ended `truncated` at the 512-token answer cap (long thinking cases) — retained, listed per request, summarized separately; prompt-only rates are not comparable. 0 failed, 0 skipped.
- The optional concurrency stage was not selected (0 planned); only warmup, performance and sanity ran.
- `required_duration_s` = 75.6 s, below the 2–4 minute range the protocol doc reports descriptively for the author's older machines (the doc states it is a safeguard, not a floor); model hashing before the run (3 min 27 s) is excluded per protocol.

本站相关内容

相关页面的快捷入口。