Proving Office publishes research on the quality of AI evaluations: where benchmark errors come from, how they spread into the benchmarks derived from them, and whether corrections reach those copies. Each study is released with its code, data and stated limitations. Claims are limited to what the released evidence supports.
8 October 2026
MMLU-Redux flags 78 parent keys of MMLU-Pro items as wrong or as having no correct answer, and 74 of those keys remain unchanged in MMLU-Pro. The article argues that a correction to a parent benchmark should trigger review of every item derived from it. A 100-item experiment found no established effect of lineage on finding defects. It did help reviewers say where a defect entered: 12 of 14 provisional defects with lineage, against 3 of 14 without. All reviewers and adjudicators were LLMs, and no human expert has checked the labels, so these results are provisional.
Article · Code and data · Archived release v1.0.1 (DOI 10.5281/zenodo.23231363)
Archived report
A published test of AI-security guard products. It is kept here as part of the research archive and is not an ongoing service.
Report (PDF)
These reports contain candidate corrections. A report counts as confirmed only when the maintainers accept it; no reply is not treated as agreement.
Proving Office is an independent research site written by Pasha. It studies how AI benchmarks are maintained: how errors are found, where they came from, and whether the fix reaches every benchmark that inherited them. Current work focuses on item-level lineage, where one benchmark reuses questions from another. Each study publishes its code, data and limitations, and problems found in a benchmark are reported to its maintainers. Proving Office does not sell testing or rankings.