Pull requests / #2076

#2076 bench: 2x RTX 5060 Ti + i7-12700, IQ3_XXS high reasoning on 0.1.42

open · @inlanger · 0 Kommentare · Auf GitHub

BenchmarksMulti-GPUNVIDIA / CUDAModels & quants

Beschreibung

Community measurements from 10–11 October on **2x RTX 5060 Ti 16 GB + i7-12700 + 64 GB DDR4-3200**, running **Strata 0.1.42** in Docker with GSQ-RCO IQ3_XXS, Huihui projection and the Cyrillic MTP vocabulary.

- Full high-reasoning responses: **150–162 tok/s** at 2.3–2.5K input; **123–139 tok/s** at 122K input, with about 47 s prefill.
- **Peak single-response decode: 178.2 tok/s**, with tail skip 7, aux CPUs auto and split MTP batching off (2,509 input / 2,225 output tokens).
- Same-binary A/B/B/A: tail skip 7 with batched split MTP gives **149.5 → 161.9 pooled tok/s (+8.3%)**. This is a settings comparison, not a version-upgrade claim.
- P-core-only pool: no consistent advantage; the 11-worker pool was retained.

Adds the report and index entries, all **81 requests**, synthetic fixtures, answers/reasoning, sanitized configurations, timing logs, artifact hashes, and scripts to recalculate or replay the measurements. No engine changes.

Validation: recalculated summaries match the saved measurements; paired request hashes match; token counts and validation flags match the source data; Python syntax, local links and `git diff --check` pass. No new inference runs were performed for this submission.

Mehr auf der Site

Links zu Install, Modellen, Releases.