Pull requests / #2184

#2184 bench: a 15-question checkable quality suite for head-to-head comparisons

open · @dvasdekis · 0 comentarios · En GitHub

BenchmarksModels & quants

Descripción

### Summary

Adds ench/quality_suite.py, a checkable evaluation suite designed for head-to-head model and quantization comparisons on the same box:
- 15 objective questions with verifiable right answers across arithmetic, factual recall, Python synthesis, bug diagnosis, and strict instruction following.
- Deliberately excludes subjective or aesthetic questions requiring manual grading.
- Measures pass rate alongside warm steady-state decode speed.
- Supports configurable --port, --url, and run label.

### What it leaves alone
- Inference engine, server, and existing benchmarks are untouched. Zero dependencies outside standard library.

En el sitio

Enlaces a install, modelos, releases.