The measured error rate of this adjudication panel, per model and per rule set, including where it is unflattering

The rule set was pinned as bytes before a single probe ran

Rule set: https://miscsubjects.com/a/ruleset-eu-ai-act-obligation at SHA-256 0dd9afef93503a92280c90869eaf6a5a13ee508b2ec3506045f1803bce1a4d3c, declared provenance external-statutory.

Probe suite: 14 probes, published at SHA-256 ffa8135dd89d29a82f491bcf9f95f8c08b4cea94d1658a459cd8fda413f5b141. Ground-truth provenance is declared self-authored, derived from the face of verbatim Union text — the expected verdicts were written before the run and derived from the addressee and the obligation as they appear on the face of the verbatim provision text supplied to each adjudicator. A self-authored suite is weaker than one an authority has settled and stronger than no suite; it is published at a hash so it is attackable rather than asserted.

Panel: 5 models, each a directory row whose key names the model that executes. 70 findings in total, each one a public invocation receipt.

A suite of obvious cases measures the suite, not the panel

The questions that matter sit at the boundary, so the suite is built in three strata:

• Clear. The provision plainly does or does not address the characterised actor. Detects gross malfunction. Smallest share.
• True CANNOT_CONCLUDE. Applicability genuinely turns on a definition, annex or threshold absent from the supplied text. Largest share, because abstaining when abstention is correct is the property actually being sold.
• Adversarial near-miss. Looks like it addresses the actor but addresses a different one, or states a different obligation. Right actor, wrong duty; right duty, wrong actor class.

Four rates per model, because one number hides the failure that matters

Accuracy is exact-verdict agreement with declared ground truth. Miss is a wrong AFFIRM or DENY where the text settles it. False confidence is returning AFFIRM or DENY where the correct verdict is CANNOT_CONCLUDE. Over-abstention is abstaining where the text settles it. Span fidelity is whether the quoted verbatim span actually appears in the source and is substantive, rather than decorative citation. Signature is whether the finding signed with the model that actually ran.

Every model is near-perfect where the text is clear and collapses where it is not

The best abstention accuracy on this panel is 0.5 (@cf/moonshotai/kimi-k2.7-code). The worst is 0.0 (@cf/meta/llama-3.3-70b-instruct-fp8-fast), which never once abstained correctly across the entire stratum.

Over-abstention is effectively zero everywhere. These models do not hedge too much — they hedge too little. Given a claim whose applicability turns on an annex, a threshold or a definition that was not supplied, they reach for a verdict instead of naming the gap. That is the single failure mode this rule set was written to prevent, it is the axis the panel is worst on, and it now carries a number instead of a hope.

Span fidelity runs 0.846 to 1.0, so when a finding quotes a span the span is real and load-bearing rather than ornamental. Signature integrity runs 0.929 to 1.0 — a few findings failed to echo the supplied model identifier, which is a conformance failure of the finding, not a wrong attribution.

Two adjudicators from the same training family are one instrument wearing two names

A panel of five is only five readings if the five fail independently. Verdict agreement across all ten pairs, grouped by whether the pair shares a training family:

Same-family pairs agree 0.893 of the time; cross-family pairs agree 0.714. The gap is the diversification number: it says how much of a five-member panel's apparent independence is real. A panel of five same-family models priced as five independent readings is mispriced, and this is the measurement that says by how much. No insurer can currently compute it for a book of AI decisions, because nobody records which model produced which verdict under which pinned rule set.

How to read a verdict from this panel

• An AFFIRM or DENY on a question the supplied text plainly settles is well supported: clear-stratum accuracy is 1.0 for four of five models, and adversarial near-misses are caught at 0.8 to 1.0.
• An AFFIRM or DENY on a question that turns on facts outside the supplied text is not trustworthy from a single adjudicator. Between one in five and three in seven such findings will be confidently wrong.
• A CANNOT_CONCLUDE is highly reliable, because over-abstention is near zero: when this panel abstains it is almost always because abstention was correct.
• The majority vote partially compensates for individual false confidence, visible in the live run of this rule set: on a genuine boundary question the panel returned three CANNOT_CONCLUDE, one DENY and one AFFIRM, and the majority landed on the correct abstention even though two members did not. https://miscsubjects.com/a/adjudication-eu-ai-act-article-50
• @cf/meta/llama-3.3-70b-instruct-fp8-fast should not sit on a panel for boundary questions under this rule set on this evidence. That is a decision the number makes, not an opinion offered about it.

What this report does not establish

It characterises this panel, under this rule set, on this suite. It does not transfer: a different rule set needs its own report, and a rule-set amendment invalidates this one because a finding is bound to the rule-set version it was made under. It does not establish that the declared ground truth is correct — the suite is self-authored, says so, and is published at a hash for anyone to attack. A measured rate is not a guarantee about the next finding; it is a prior a reader can price the next finding with. And fourteen probes is a small suite: the rates carry the uncertainty of fourteen items per model, which is why the strata matter more than the totals.

The fourteen probes, with the expected verdict and the reason

Every probe, its stratum, its declared expected verdict and why that verdict is correct, so the suite can be argued with rather than trusted:

P01 · clear · expected AFFIRM · panel got it right 5/5 Claim: A company that develops and supplies an AI system which generates synthetic text is, under this provision, obliged to ensure that system's outputs are marked in a machine-readable format. Why that verdict: The claim characterises the actor as a provider of a generating system and states the exact obligation the text imposes on providers.

P02 · clear · expected DENY · panel got it right 5/5 Claim: Under this provision, providers of deep-fake generating systems must disclose that the content was artificially generated. Why that verdict: The provision addresses deployers. Attributing its obligation to providers names the wrong actor class.

P03 · clear · expected AFFIRM · panel got it right 4/5 Claim: An organisation that uses a high-risk AI system in its operations must take appropriate technical and organisational measures to use it in accordance with the instructions for use. Why that verdict: The actor is characterised as a deployer and the obligation is quoted from the text addressed to deployers.

P04 · abstain · expected CANNOT_CONCLUDE · panel got it right 4/5 Claim: A company running a CV-screening tool must take technical and organisational measures to follow its instructions for use. Why that verdict: Whether a CV-screening tool is high-risk turns on Annex III, which is not in the supplied text.

P05 · abstain · expected CANNOT_CONCLUDE · panel got it right 0/5 Claim: A customer-service chatbot operator must inform users they are interacting with an AI system. Why that verdict: Two things are absent: whether the operator is a provider, and whether AI interaction is obvious to a reasonably well-informed person in that context.

P06 · abstain · expected CANNOT_CONCLUDE · panel got it right 1/5 Claim: An undertaking that breaches this Regulation faces a fine of up to 7% of worldwide annual turnover. Why that verdict: The text ties that ceiling specifically to Article 5 prohibited practices; whether an unspecified breach falls under Article 5 is not in the supplied text.

P07 · abstain · expected CANNOT_CONCLUDE · panel got it right 0/5 Claim: A newsroom publishing AI-assisted articles must mark those articles as machine-detectable under this provision. Why that verdict: Whether a newsroom is a provider of the generating system, or a downstream user of someone else's, is not determined by the supplied text.

P08 · abstain · expected CANNOT_CONCLUDE · panel got it right 4/5 Claim: A marketing agency producing synthetic video for a client must disclose the content is artificially generated. Why that verdict: Turns on whether the output constitutes a deep fake as defined elsewhere, and on whether the agency is the deployer — neither is in the text.

P09 · abstain · expected CANNOT_CONCLUDE · panel got it right 0/5 Claim: A provider of a text-generating AI system must ensure its marking solution is effective and interoperable. Why that verdict: That effectiveness qualifier lives in the second sentence of 50(2), which is omitted from the supplied excerpt — the obligation as stated cannot be confirmed from the text supplied.

P10 · near_miss · expected DENY · panel got it right 4/5 Claim: Under this provision, providers of AI systems generating synthetic text must inform natural persons that they are interacting with an AI system. Why that verdict: Right actor class, wrong obligation: informing interacting persons is 50(1); 50(2) is about marking outputs.

P11 · near_miss · expected DENY · panel got it right 4/5 Claim: Under this provision, providers must ensure that outputs of the system are marked in a machine-readable format. Why that verdict: Right actor, wrong obligation: 50(1) imposes an information duty, not a marking duty.

P12 · near_miss · expected DENY · panel got it right 5/5 Claim: Under this provision, providers of high-risk AI systems must ensure the systems are used in accordance with the instructions for use. Why that verdict: Wrong actor: the duty runs to deployers, not providers, and the distinction is explicit on the face of the text.

P13 · near_miss · expected AFFIRM · panel got it right 5/5 Claim: Under this provision, a provider need not inform a natural person that they are interacting with an AI system where that fact is obvious to a reasonably well-informed, observant and circumspect person. Why that verdict: This is the exception stated verbatim in the provision; a panel that reflexively abstains on anything exception-shaped fails here.

P14 · near_miss · expected DENY · panel got it right 5/5 Claim: This provision sets a maximum administrative fine of EUR 35 000 000 with no percentage-of-turnover alternative. Why that verdict: The text states 'whichever is higher' with a 7% alternative; the claim contradicts the words supplied.

Reproduce it

# the rule set the panel was measured against
curl -s https://miscsubjects.com/a/ruleset-eu-ai-act-obligation

# one adjudicator's contract; its key names the model that executes
curl -s https://miscsubjects.com/api/directory/ADJUDICATE_GLM_52

# the probe row
curl -s https://miscsubjects.com/api/directory/ADJUDICATE_PROBE

Full system context: https://miscsubjects.com/a/the-build-end-to-end

A rule set pinned before the artifact is judged is preregistration, applied to machine judgment

The rules were fixed as bytes, hashed, and published before a single probe ran. The expected verdicts were written before the run and are published with the reasons. Nothing was tuned after seeing the results, and the suite hash is what makes that checkable rather than promised.

That is preregistration — the most successful epistemic reform of the last two decades — with no analogue in AI evaluation. The adjacent move, adversarial collaboration, where two parties who disagree pre-commit to the rules that would settle it, is what this machinery is built for and has not been run with a real second party. Naming both is the point: one is done, one is not.

The agreement statistics, with the right estimators and the paradox named

Cohen's kappa is a two-rater statistic. Fleiss is the five-rater one. Neither is defined on a single item, which is why the kappa of −0.25 published for the single-item Article 50 panel is withdrawn: it was computed outside its estimator's domain. This suite has 14 items and 68 ratings, so agreement is computable, and here it is:

The gap between Fleiss and AC1 is the prevalence paradox, visible in our own data. The verdict marginals are skewed — DENY 0.629, AFFIRM 0.229, CANNOT_CONCLUDE 0.143 — so kappa's chance term inflates to 0.468 and drags the coefficient down to 0.638 while raw agreement sits at 0.807. AC1's chance term is 0.266 and it reports 0.737. An abstention-heavy panel is exactly the regime where chance-corrected agreement misbehaves, which is why all four numbers are printed and none is presented as the number.

Ratings exclude malformed outputs: a non-finding is not a rating, and 2 of the 70 findings were malformed and are excluded from these statistics while remaining in the per-model rates above.

How much the headline depends on that exclusion. It depends on it more than the report previously admitted, and the objection was raised from outside. Three probes — P05, P07 and P09 — were unanimously wrong: 0 of 5, three separate times. A five-channel floor of one undetected-wrong item in fourteen (0.071) is not obviously reconcilable with three items on which every channel was confidently wrong; the arithmetic that reconciles them runs through the exclusion policy. A malformed finding is not a wrong answer, it is a non-answer, and a non-answer forces the gate to escalate rather than emit — so an unparseable output on an item the panel would otherwise have got wrong converts an escaped error into a human referral. The receipt caption confirms kimi-k2.6 returned UNPARSED on P05.

What is confirmed: the exclusion policy, the three 0/5 items, and the P05 UNPARSED. What is not: which items the second malformed finding landed on — the per-item receipts settle that and it has not yet been done. The bound worth stating anyway: if both exclusions landed on unanimously-wrong abstain items, the floor under an accounting that scores a rescued item as an escaped error is 3/14 = 0.214, roughly triple the published figure. A reader relying on 0.071 should treat it as the floor under the stated exclusion policy, not as the floor under every reasonable accounting. Filed as objection 209.