Pull requests / #2300

#2300 bench: add hosted Codex quality baseline for the quant study

open · @Jhacarreiro · 0 comentarios · En GitHub

BenchmarksNVIDIA / CUDAModels & quantsDocumentationLinux

Descripción

## Title

Add community Codex quality baseline for the #1850 battery

## Summary

Adds a measured external quality baseline for the seven suites in the RTX 5070 Ti quantization study (#1850), using Codex CLI 0.162.1 configured as `gpt-6.1-sol` with `ultra` reasoning. Five Bugs finished with **25/25 visible and 25/25 hidden checks (100% each)** after corrections identified during the debugging and validation workflow. HumanEval+ passed 156/164 first attempts and 160/164 after one allowed repair per failure; Tool Eval Bench scored 90.2% mean with 0.4 percentage-point sample SD over five 92-case trials.

## What changed

- Adds `bench/results/2026-10-13-community-codex-quality-baseline/README.md` with all seven raw quality scores, harness pins, coverage and protocol limits.
- Adds `benchmark-scores.json` with exact score counts, the five-trial tool grade matrix, needle grid, safe artifact hashes and separately recorded original/recovered tool results.
- Adds `five-bugs-final-validation.json` with all 25 visible and 25 hidden outcomes after corrections, the five corrected solution hashes and the t4 checker qualification.

## Extra Notes

This is a results-only submission. The hosted model used an adapted CLI transport; it does not measure Strata inference, GPU performance or a normalized OVERALL score. MMLU covers only 100 abstract algebra questions. The five largest needle inputs used a separate transport after a frontend limit, with exact logical prompts and about 222K actual input tokens. One tool transport timeout was recovered using two preserved replies and one previously unanswered reply; the original report remains separately recorded.

Five Bugs initially scored 25/25 visible and 20/25 hidden across independent candidates. The final 25/25 in each suite comes from grading each of five fixed corrected solutions five times, after using repository documentation and checker feedback. The report describes the corrections and the t4 quoted-attribute checker expectation; no checker was changed. The original independent-run score is preserved separately.

Validation: all 172 HumanEval+ candidate pass/fail results agreed with independent stock EvalPlus 0.3.1 Linux grading; all 460 tool cells and 25 needle cells were audited; the original score JSON is byte-identical to the frozen public export. The final Five Bugs record contains 25 unique task/trial pairs, each passing both checkers, with hashes matching the five corrected files. The report uses a distinct quality schema and filenames, outside `tools/strata_bench.py`'s directory-based `results.json` reader. No engine or server code changes.

En el sitio

Enlaces a install, modelos, releases.