The question
A taxonomy of failure is only worth something if two annotators applying it to the same transcript land in roughly the same place. MAST's taxonomy sorts failures into fourteen cells covering specification, inter-agent misalignment, and verification. It is a useful piece of work and it is increasingly used as a reference. So the narrow question is worth asking directly: do language models agree with each other when they apply it?
Agreement is measured with Krippendorff's α throughout, cluster-bootstrapped over traces. 1.0 is perfect; 0 is chance. The field convention treats 0.667 as the floor for annotation you would build on.
The correction, first
Rev 3.1 ran six open-weight coders at 2–12B parameters and found α = 0.173. It controlled the obvious confounds and controlled them well: every coder passed a framing-sensitivity gate before any trace was coded, and a pre-registered instrument comparison showed the shape of the question moved α by only 0.103 (free-form 0.149 against per-cell 0.046).
It could not rule out the plainest confound: every coder was small. Rev 4 ran the identical frame, prompt and parser, reused as literal objects rather than rewritten, against larger models.
The null was about model scale, not about the taxonomy. Nothing in rev 3.1 is retracted; every clause of it still holds. But the sentence a reader would take from it, this taxonomy cannot be annotated reliably, is not supported by the data and is refuted by rev 4. The rev 3.1 write-up now opens with a correction banner.
Scale was named in rev 3.1's own limitations section, written down, and not treated as urgent. One variable the study had already noticed overturned its headline two days later. The limitations section is where the next experiment is, not where the caveats go to die.
Finding 2: agreement scales with annotator size
| Tier | Coders | Pooled α | 95% CI |
|---|---|---|---|
| T1 open-weight small (2–12B) | 6 | 0.173 | [0.127, 0.207] |
| T2 open-weight large (70B-class) | 2 | 0.360 | n/a |
| T3 frontier | 4 | 0.598 | [0.499, 0.670] |
The gradient is smooth, not stepped. Agreement roughly doubles from T1 to T2 and rises again to T3, with no discontinuity that would suggest a capability threshold. The middle tier earns its place: it turns small models fail, frontier models work into a continuum. Cross-tier agreement sitting at 0.362 says the tiers are annotating the same thing at different precision, not annotating past each other.
Every frontier coder passed the same framing gate, with shifts of 0.000–0.029 against a 0.30 bar fixed before collection, markedly less frame-sensitive than the small models. Without rev 3.1's gate and instrument work, this result would be ambiguous between "frontier models are better annotators" and "frontier models follow instructions better."
Findings 1 and 3: the reference is the unstable thing
MAST's released dataset contains duplicated trace payloads carrying conflicting annotations: the same trace, annotated more than once, with the annotations disagreeing. Nobody designed that as a reliability experiment, but structurally it is one, and it can be scored.
The self-agreement interval contains zero: on the file's own repeats, the published annotation is not distinguishable from chance agreement with itself. Meanwhile four frontier models, annotating independently, agree with each other substantially, and with those labels not at all.
The construction gives the published labels their best case. All 31 duplicated groups are included, one of which agrees perfectly, and that pair pulls the figure up. Scored on the 30 conflicting groups alone the reference drops to α = 0.041. We report the higher number.
This is not a claim that the models are right. There is no ground truth here, by construction, which is precisely what makes the taxonomy hard. It is a claim about which reference is more stable, and its useful property is that it is measurable without any human ground truth at all. You only need the same annotators to look twice.
The human reference material is 8 annotation blocks across 19 records, not 19 independent annotations. Any inter-rater claim built on it must establish substrate independence first, or it is measuring copied columns. The α above is computed over duplicated trace units precisely to avoid that trap, and the bootstrap clusters on traces.
The second limit is n. The figure rests on 31 duplicated traces, roughly 5% of the corpus, almost all Magentic/GAIA plus one AppWorld pair. That subset is not a random sample of systems. It does not change the ceiling reading, because a ceiling is a statement about what the file reproduces, but the number should not be quoted as a corpus-wide property.
No coder was excluded by the marginal-rate gate. Marking rates ran 0.146–0.229, a tight band and much closer to the corpus's own ~0.16 than the open-weight roster's 16-fold spread. The convergence is not an artifact of coders marking wildly different amounts of the taxonomy.
What is not claimed
- Not that frontier models annotate this taxonomy correctly. There is no ground truth here.
- Not that the taxonomy is reliable. 0.598 is below the 0.667 conventional floor. This is not solved. It is a different world from 0.173, which is a different claim.
- Not that MAST's annotators were careless. Finding 1 is a measurement on a released artifact; the mechanism behind it is not established here.
- Not a general claim about LLM annotation. One taxonomy, one corpus, short-to-medium traces.
Limits
No Google coder. The pre-registered gemini-3.6-flash was
recorded UNAVAILABLE, because its endpoint refuses reasoning.enabled=false, which
the instrument requires. It was not substituted: swapping in a different Google model
after the other coders' numbers were visible is exactly the researcher discretion
pre-registration exists to prevent. T3 keeps four coders across two
vendors, so the decisive test remains scoreable, but vendor coverage is a real gap.
The family effect is under-powered at both scales. Within-family α
0.642 against between-family 0.428 at frontier, but on two pairs, reported with no
verdict. Rev 3.1's equivalent was AMBIGUOUS at 0.127 after its admission gate excluded
both members of one family. Future rosters need three members per family so a gate can
take one and still leave a pair. The strongest pair in the study is
gpt-5.6-sol against gpt-5.6-terra at 0.735; the weakest
frontier pairing is claude-opus-5 against llama-3.3-70b at 0.212.
n = 24 traces, short-to-medium only (≤8,000 tokens), three strata. Total collection cost $3.00 across 324 calls, against a pre-registered $75 cap.
The harness bug that nearly became a finding
The first admission run reported frontier models failing the output-format gate: one at 20% parseability, one returning no content at all. It would have been written up as frontier models cannot follow this format.
It was wrong. max_tokens=300 was being consumed entirely by reasoning
tokens: finish_reason=length, completion_tokens=300, empty
content. The same call with reasoning disabled answers in five tokens.
All data collected on the pre-fix instrument was discarded before any statistic was
computed.
1. finish_reason == "length" is a failed call, not a
parsed one. A truncated list otherwise parses into a shorter cell vector and quietly
lowers α, indistinguishable from a coder that marked fewer cells.
2. Reasoning is disabled explicitly on every call for every coder, which the tier comparison requires anyway: comparing tiers across different reasoning settings is not comparing like with like.
This is the third appearance of one failure mode in this lab, each time in a different costume. The cheap check is to open one raw payload before believing any claim about a model's competence.
Protocols & data
Both studies were pre-registered before collection. Every statistic on this page ships with the raw model output it was computed from, the scorer that computed it, and that scorer's committed output, so each number can be checked against the artifact it was computed from. Recomputing a statistic end-to-end from the raw model calls additionally needs the lab's collector stack (the modules that build the frame and admission files and call the model APIs), which is not shipped in this repository.
- Pre-registration · rev 4 frontier annotator agreement, 2026-08-12
- Pre-registration · rev 3.1 open-weight annotator agreement, 2026-08-10
- Results · rev 4 D1, D2, D4 scored
- Results · rev 3.1 carries the correction banner
- Synthesis · the three findings
- results-rev4.json scorer output, machine readable
- results-rev3.json scorer output, machine readable
- rev4_score.py · rev3_score.py · kripp.py the scorers, and the α implementation they call
- raw/ 57 JSONL files, every model call behind every statistic
All three findings rest on MAST's public released dataset (AG2 and Magentic traces) plus model output generated for these studies. No private corpus is involved.