Experiments with MMLU-derived items did not establish that lineage helps auditors find more defects. They did expose a maintenance gap: audit findings rarely reach the items that inherit them.
2026-10-08
How to read the claims below. Each claim falls into one of five kinds:
MMLU-Pro question 3542 asks for the relative populations of two electron-spin levels at “L-band” and “W-band” microwave frequencies. It names bands rather than frequencies, so strictly it is underspecified. Under the standard band ranges, though (L-band 1 to 2 GHz, W-band about 94 to 95 GHz), the ratio exp(−hν/kT) at 300 K is about 0.9998 at L-band and 0.9851 at W-band. The keyed answer gives 0.9850 “at L-band”, which is the W-band value, and no option matches. (Observed, by calculation under a stated convention.)
The error did not start in MMLU-Pro. The question, its four original options and its key come unchanged from MMLU. MMLU-Redux, a 2024 re-annotation of MMLU, flagged the parent item as “bad question clarity”, noting the missing band information. MMLU-Pro kept the item, added six distractors, and still carries the key in its February 2026 snapshot. By their documentation, two translated MMLU datasets cover the parent in 14 and 41 languages, and MMLU-ProX covers the MMLU-Pro item in 29. We have not checked whether those copies still carry the error.
The pattern is common in this family. Redux flags 78 parent keys of MMLU-Pro items as wrong or as having no correct answer, and 74 of those flagged keys survive unchanged in the February 2026 MMLU-Pro snapshot. For 54 of the 78, Redux’s proposed answer is already one of MMLU-Pro’s ten options. (Observed.) That does not make 74 errors, because Redux labels can be wrong too (section 6). It makes 74 items that should have been re-reviewed when the parent was flagged, and it shows that the flags did not reach the dataset built from those parents.
Audits are done on snapshots, but benchmark items live in families: they are copied, filtered, expanded, translated and re-annotated. A correction that stops at the dataset boundary leaves its copies behind.
We drew 100 MMLU-derived MMLU-Pro items at random (seed 20261008, excluding an earlier 50-item pilot) and had them reviewed. Fourteen were judged defective. (Provisional (LLM).) Classified by where the defect entered:
| Where the defect entered | Defects |
|---|---|
| MMLU-Pro’s added options (a new distractor that is also correct) | 9 |
| Already present in the MMLU parent | 4 |
| Upstream source (a stale fact) | 1 |
For a maintainer, the split decides where a fix belongs. An inherited defect has to be fixed in the parent and every descendant. A defect created by MMLU-Pro’s option expansion needs fixing only in MMLU-Pro and its own derivatives, and the parent’s translations are unaffected. A typical expansion defect is question 6268: “Fats normally contain triacylglycerols at concentrations of”, keyed “> 95%”. The expansion added “95-100%”, which names the same interval.
We had set out to answer a stronger question: does showing an auditor an item’s lineage help them find defects?
Design. Three arms reviewed the same 100 items. Each arm was four isolated LLM agents with 25 items each. - A, item-only: saw the MMLU-Pro question, options and key. - B, lineage-aware: additionally saw the MMLU parent, which options were copied or added, any MMLU-Redux annotation, and the item’s version history. - C, extra effort: saw what A saw, plus a mandatory protocol to solve the item before reading the key, check every option, and look up every fact.
Two independent LLM adjudicators reviewed every flagged item plus 12 random unflagged ones. A third broke ties. They agreed at κ = 0.80 on defect vs no defect and 0.82 on origin. The rubric and thresholds were frozen before the draw.
| A. Item-only | B. Lineage | C. Extra effort | |
|---|---|---|---|
| Defects found (of 14) | 8 | 12 | 9 |
| Origin correctly named (of 14) | 3 | 12 | 4 |
What did not survive. - Detection. B’s lead over A is not distinguishable from chance (exact McNemar p = 0.125). It vanishes on the 3 defects both adjudicators rated “confirmed”, where all arms found all 3. B’s lead over the extra-effort arm vanishes when the reference labels are rebuilt by adjudicators who never saw lineage. With 14 defects, a 10-point difference could not have been detected reliably. - A second run disagreed. A separate, unpublished second run on the same family (different snapshots, sample and adjudication) found the opposite ordering. Its materials have not been deposited or independently checked, so we do not report its rates. Two runs disagreeing at this size is what an unknown effect looks like. We do not claim that lineage helps auditors find defects, or that it does not. - The pre-registered decision rule. Our rule declared the result “publishable”, but that was not informative. The leg that passed (localization) was one only arm B could win.
What survived. - Using the diff. When B was shown the parent-child diff, it used it. On the 12 defects it found, it named the right origin for all 12. A constant guess of “added options” would have been right for 7. - The open question. That is the comparison B was built to win, because only B saw the parent. Whether showing lineage to the auditor beats simply joining it to the findings afterwards, with a script, is untested. (Open.)
The practical conclusion does not depend on that open question. Whoever attaches the lineage, a finding without it cannot be routed to the right place.
Four separate propositions, with different evidence behind each:
The smallest tool that does this takes a corrected or disputed item and returns its known descendants. For each one, it reports the transformation on each edge, whether the questioned content was inherited or added, and a ranked queue. Here is its output for question 3542, abridged:
| Rank | Dataset | Item | Questioned content | Note |
|---|---|---|---|---|
| 0 | MMLU-Redux | parent annotation | inherited | labeled “bad question clarity”; add that no option matches under standard bands |
| 1 | MMLU-Pro | 3542 | inherited | key unchanged since late 2024 |
| 3 | MMMLU, Global-MMLU | translations of the parent | unverified | documented coverage only |
| 3 | MMLU-ProX | translations of 3542 | unverified | documented coverage only |
For question 6268, the added-option defect, the same tool marks the parent’s translations as unaffected and routes the fix to MMLU-Pro and MMLU-ProX only.
The tool also returns duplicates. For the item in section 6, it lists two MMLU-Pro items with identical options, 1932 and 1933. MMLU-Pro duplicates are already reported. A January 2025 Hugging Face discussion reported 316 duplicate rows and was closed as resolved. A September 2025 discussion reported 158 exact duplicate pairs, including 1932 and 1933. Our count, 157 MMLU parents with two identical-option copies in the February 2026 snapshot, reproduces the second report on the MMLU-derived subset. It matters here only because a fix applied to one copy and not the other is a fix half done.
Coverage, stated plainly. - Lineage was reconstructed by matching stems, exactly or with a close fuzzy match. Heavily rewritten items are missed. - The data come from GitHub snapshots, not the current Hugging Face release, which was unreachable from our environment. - The translations were not joined item by item.
MMLU-Pro question 1932 asks what passive personality jurisdiction is, and keys “jurisdiction based on the nationality of the victims”. MMLU-Redux labels the parent’s key wrong and proposes “where the offence was committed”. The key is right. The American Society of International Law’s benchbook says of this basis that “what matters … is the nationality of the victim or person at whom the conduct at issue was directed.” (Observed, against a cited source.) A pipeline that pushed Redux’s correction downstream would have broken two MMLU-Pro items and every translation of the parent.
So corrections should travel along lineage as evidence attached to items, with their source and status, and should be applied only after review. The same queue that tracks a defect should track the correction.
Recording provenance is a solved representation problem, and nothing
here is offered as a new data model. - W3C PROV-O defines entities,
activities and agents, and relations such as
prov:wasDerivedFrom and prov:wasGeneratedBy. -
The Croissant 1.1 dataset-metadata standard recommends PROV-O. It allows
provenance on datasets, files, RecordSets and Fields, and “for the
finest level of granularity … individual data values.” - Inherited
benchmark errors are also documented. Land & Bikel (2026) show
errors from GSM8K and MMLU-Pro reappearing downstream.
The gap is in practice. Audit findings are published as lists of items in one snapshot, not as edges in a family, so the representation exists but audit workflows do not consult it.
The provisional defects need human confirmation, and the cheapest credible source is the people who maintain the benchmarks. We submitted the clearest cases on 8 October 2026, in one discussion to the MMLU-Pro maintainers and, separately, to the MMLU-Redux maintainers: - a key whose stated ground is contradicted by Cooper v. Oklahoma, the Supreme Court case its facts are drawn from (936); - an added option that names the same interval as the key (6268); - an added “Y-linked inheritance” option that is correct by definition (6599); - the underspecified spin-population item, where no option matches under standard band conventions (3542); - the two Redux annotations discussed above.
Each submission went in with a re-review queue of known descendants, generated and published before posting. We also asked the MMLU-Pro maintainers how they identify affected items when an upstream key changes, since their answer tests the gap this note describes. Their published policy is to keep dataset contents stable and document known issues, so “confirmed, documented” is a likely outcome.
Arguable cases are held back until an expert has read them.
Status at publication: Six candidate corrections were submitted on 8 October 2026 and await maintainer review (MMLU-Pro discussion, MMLU-Redux issue). We will report each outcome as “confirmed and fixed”, “confirmed, documented”, “acknowledged as ambiguous”, “rejected with reason” or “no response as of [date]”, always against the full denominator of six submissions. We will not count silence as confirmation. A maintainer’s confirmation validates that item. It does not validate the experiment’s rates.
The comparison that would settle the open question has three arms on independently human-validated defects: - A: item-only review. - A+J: the same findings with lineage joined afterwards by a script. - B: review with lineage visible.
The primary outcome is correct origin, comparing B with A+J. The secondary outcomes are detection, measured minutes, correction quality, and whether the right descendants were queued. A properly powered version needs several hundred defects and human reviewers, so it is a study to run with a benchmark organization, not a precondition for the maintenance practice above.
We did not show that lineage makes auditors better at finding errors, and a second run suggests the question is still open. We did find that defects in a derived benchmark enter at identifiable steps, that in this family known corrections mostly did not reach the datasets built from a corrected item, and that some corrections are themselves wrong. Linking audit findings and corrections to item-level derivation chains can be a cheap way to route each fix to every place it belongs, and to stop a bad fix from spreading.