Pull requests / #2300
#2300 bench: add hosted Codex quality baseline for the quant study
open · @Jhacarreiro · 0 comentários · No GitHub
BenchmarksNVIDIA / CUDAModels & quantsDocumentationLinux
Descrição
## Title Add community Codex quality baseline for the #1850 battery ## Summary Adds a measured external quality baseline for the seven suites in the RTX 5070 Ti quantization study (#1850), using Codex CLI 0.162.1 configured as `gpt-6.1-sol` with `ultra` reasoning. Five Bugs finished with **25/25 visible and 25/25 hidden checks (100% each)** after corrections identified during the debugging and validation workflow. HumanEval+ passed 156/164 first attempts and 160/164 after one allowed repair per failure; Tool Eval Bench scored 90.2% mean with 0.4 percentage-point sample SD over five 92-case trials. ## What changed - Adds `bench/results/2026-10-13-community-codex-quality-baseline/README.md` with all seven raw quality scores, harness pins, coverage and protocol limits. - Adds `benchmark-scores.json` with exact score counts, the five-trial tool grade matrix, needle grid, safe artifact hashes and separately recorded original/recovered tool results. - Adds `five-bugs-final-validation.json` with all 25 visible and 25 hidden outcomes after corrections, the five corrected solution hashes and the t4 checker qualification. ## Extra Notes This is a results-only submission. The hosted model used an adapted CLI transport; it does not measure Strata inference, GPU performance or a normalized OVERALL score. MMLU covers only 100 abstract algebra questions. The five largest needle inputs used a separate transport after a frontend limit, with exact logical prompts and about 222K actual input tokens. One tool transport timeout was recovered using two preserved replies and one previously unanswered reply; the original report remains separately recorded. Five Bugs initially scored 25/25 visible and 20/25 hidden across independent candidates. The final 25/25 in each suite comes from grading each of five fixed corrected solutions five times, after using repository documentation and checker feedback. The report describes the corrections and the t4 quoted-attribute checker expectation; no checker was changed. The original independent-run score is preserved separately. Validation: all 172 HumanEval+ candidate pass/fail results agreed with independent stock EvalPlus 0.3.1 Linux grading; all 460 tool cells and 25 needle cells were audited; the original score JSON is byte-identical to the frozen public export. The final Five Bugs record contains 25 unique task/trial pairs, each passing both checkers, with hashes matching the five corrected files. The report uses a distinct quality schema and filenames, outside `tools/strata_bench.py`'s directory-based `results.json` reader. No engine or server code changes.
No site
Links install, modelos, releases.