改动 / #3573
#3573 [Benchmark] Signal 3.8 Flash Next AP-Q4_K_XL and AP-IQ4_XS on an RTX 4090 + 93.5 GiB RAM (2026-10-21, Strata 0.1.46.3)
open · draft · @clothespin · 0 评论 · 去 GitHub 看
BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
说明
## Problem and result
#2041 adds agentionai's Signal 3.8 Flash Next to setup, and its documentation quotes speeds measured on 2026-10-21. CONTRIBUTING.md asks that
the individual runs, failures and variability behind a performance claim are preserved, and that a results-only submission is kept apart from code.
This pull request is that submission: the raw data and a community-guide report for 14 benchmark runs and two comparisons of Signal's two sizes on
one RTX 4090 PC. Result, medians of the runs (short answers / 16K prompt / agent session, tokens per second): setup's AP-Q4_K_XL config 62.7 / 2670 /
75.6, setup's AP-IQ4_XS config 78.4 / 3155 / 94.7, the documentation's hand-built AP-Q4_K_XL config 57.2 / 2372 / 70.0. The runs vary more than
those numbers suggest (ranges in the report), and the kernel log shows memory pressure during four runs and recovered storage read errors during
model loads (see Limits). Nothing outside the new folder changes.
## Related discussion
#2041 (the code and documentation change this data belongs to). `docs/COMMUNITY_BENCHMARKS.md` "Submit a report" and CONTRIBUTING.md "Share a
benchmark" describe this kind of pull request; no scope agreement was needed beyond them. It does not ask for any change to the README's
headline numbers. Opened as a draft until the storage read errors in this PC's kernel log (see the report) have been looked at.
## Approach
One folder, `bench/results/2026-10-21-community-rtx-4090-signal-ap-q4-k-xl-ap-iq4-xs/`, laid out as the community guide describes: `README.md`
from its template, per-run `results.json`, the tool's own report, engine log excerpts, the state before and after every run, a 10 s sampler, the
four configurations, `kernel-log-events.txt`, `scripts/` and `checksums.sha256`. Home folders, the host and user name are replaced by
placeholders; model and data folders are replaced too. The tables in the README are rebuilt from the data by `scripts/make_tables.py` and
`scripts/events_by_run.py`.
## Validation
Measured on Strata 0.1.46.3 (commit `361b1393` of the branch of #2041; `tools/strata_bench.py` and `tools/ab_engine.py` are identical on `main`
`9de04cd4`), Linux 7.2.11, RTX 4090 (driver 615.78.08, CUDA 13.4), Ryzen 9 7950X3D, 93.5 GiB RAM, 2026-10-21 00:10 to 01:46 PDT.
**Passed:**
- `python3 -I scripts/make_tables.py perrun|summary|start|ab` rebuilds the README's tables from `runs/` and `ab/`; the per-run tables equal the ones in
#2041's `docs/SIGNAL_AP_Q4_K_XL.md` (checked cell by cell for the three configurations, and for the `--adapt-*` and `--kv` figures).
- `sha256sum -c checksums.sha256` in the folder: 110 files OK. Every `.json` and `.jsonl` file parses.
- A search of the added files for home folders, user and host names, e-mail addresses and local paths finds none.
- Failed requests: none in any of the 14 runs; skipped tests: none. Every run recorded the same commit and the same engine build.
- Both model files hashed again after the runs (2026-10-21, 02:39 to 02:42): SHA-256 equal to the values `setup.py` pins in #2041.
**Failed:**
- Nothing failed as a request. Conditions that were not clean: 12 page allocation failures or stalls in 4 runs (run 2 of the hand-built config 2, run 2 of
setup's AP-Q4_K_XL config 6, run 3 of setup's AP-IQ4_XS config 1, run 3 of setup's AP-Q4_K_XL config 3), 20 swap write errors on the compressed-RAM
swap device in two of them (run 2 of the hand-built config and run 3 of setup's AP-Q4_K_XL config), and 14 bcachefs
checksum errors on reads of the two model files, each recovered on retry ("successful retry, self healing"). Re-reading both files with `sha256sum`
after the runs produced 4 more. The report lists them per run (`kernel-log-events.txt`, table in the README).
**Not run:**
- The draft standard protocol (`strata bench run`): it has no profile for a fine-tune of Qwen3.8-Flash-Next. These are the community guide's quick check.
- AMD, Windows, Mac, other cards or RAM sizes; pictures; reasoning on or temperature above 0; output quality; the draft head's acceptance.
- A re-run on a quiet PC, and any health check of the SSD or the memory (SMART, a bcachefs scrub, a memory test): they need root or a reboot.
- No code was built or changed, so no backend was rebuilt.
## Limits
- One PC, one engine build, short prompts (16K tokens at most), a desktop session open, the `powersave` CPU governor: the speeds are that PC's.
- Memory pressure and storage read errors were present in some runs; the report says which, and the figures stay in every table. The cause of neither is
established, and the read errors reproduce with a plain `sha256sum` without Strata, so the PC's storage path is not known to be healthy.
- The hand-built and setup configurations differ in six settings and the image encoder, so their comparison does not say which setting matters.
- The tool's `README.md` in each run folder says "background workloads and power limits: not recorded"; the top-level report records them.
本站相关内容
相关页面的快捷入口。