{"slug":"adjudication-probe-report-eu-ai-act","title":"The measured error rate of this adjudication panel, per model, per rule set","body":"A verdict without a measured error rate is an opinion with good paperwork. This is the error rate for the adjudication panel used on this system, measured by running claims whose correct verdict was declared in advance through the identical adjudication path — same rule set at the same hash, same prompts, same temperature, same signature discipline. Seventy findings, five models, fourteen probes.\n\n**The headline number: this panel manufactures a verdict where it should abstain between 21% and 42% of the time.** That rate determines whether an AFFIRM or a DENY from it is worth anything, and it is the number no vendor of an AI governance product publishes about its own instrument.\n\n## The rule set was pinned as bytes before a single probe ran\n\nRule set: [https://miscsubjects.com/a/ruleset-eu-ai-act-obligation](https://miscsubjects.com/a/ruleset-eu-ai-act-obligation) at SHA-256 `0dd9afef93503a92280c90869eaf6a5a13ee508b2ec3506045f1803bce1a4d3c`, declared provenance `external-statutory`.\n\nProbe suite: 14 probes, published at SHA-256 `ffa8135dd89d29a82f491bcf9f95f8c08b4cea94d1658a459cd8fda413f5b141`. Ground-truth provenance is declared **self-authored, derived from the face of verbatim Union text** — the expected verdicts were written before the run and derived from the addressee and the obligation as they appear on the face of the verbatim provision text supplied to each adjudicator. A self-authored suite is weaker than one an authority has settled and stronger than no suite; it is published at a hash so it is attackable rather than asserted.\n\nPanel: 5 models, each a directory row whose key names the model that executes. 70 findings in total, each one a public invocation receipt.\n\n## A suite of obvious cases measures the suite, not the panel\n\nThe questions that matter sit at the boundary, so the suite is built in three strata:\n\n- **Clear.** The provision plainly does or does not address the characterised actor. Detects gross malfunction. Smallest share.\n- **True CANNOT_CONCLUDE.** Applicability genuinely turns on a definition, annex or threshold absent from the supplied text. Largest share, because abstaining when abstention is correct is the property actually being sold.\n- **Adversarial near-miss.** Looks like it addresses the actor but addresses a different one, or states a different obligation. Right actor, wrong duty; right duty, wrong actor class.\n\n## Four rates per model, because one number hides the failure that matters\n\n| model | accuracy | miss | **false confidence** | over-abstention | unparsed | span fidelity | signature |\n|---|---|---|---|---|---|---|---|\n| `@cf/moonshotai/kimi-k2.7-code` | 0.786 | 0.0 | **0.214** | 0.0 | 0.0 | 1.0 | 1.0 |\n| `@cf/moonshotai/kimi-k2.6` | 0.714 | 0.0 | **0.214** | 0.0 | 0.071 | 1.0 | 0.929 |\n| `@cf/zai-org/glm-5.2` | 0.714 | 0.0 | **0.286** | 0.0 | 0.0 | 1.0 | 1.0 |\n| `@cf/zai-org/glm-4.7-flash` | 0.643 | 0.0 | **0.286** | 0.0 | 0.071 | 1.0 | 0.929 |\n| `@cf/meta/llama-3.3-70b-instruct-fp8-fast` | 0.429 | 0.071 | **0.429** | 0.071 | 0.0 | 0.846 | 1.0 |\n\n*Accuracy* is exact-verdict agreement with declared ground truth. *Miss* is a wrong AFFIRM or DENY where the text settles it. **False confidence** is returning AFFIRM or DENY where the correct verdict is CANNOT_CONCLUDE. *Over-abstention* is abstaining where the text settles it. *Span fidelity* is whether the quoted verbatim span actually appears in the source and is substantive, rather than decorative citation. *Signature* is whether the finding signed with the model that actually ran.\n\n## Every model is near-perfect where the text is clear and collapses where it is not\n\n| model | clear | true-abstain | adversarial near-miss |\n|---|---|---|---|\n| `@cf/moonshotai/kimi-k2.7-code` | 1.0 | **0.5** | 1.0 |\n| `@cf/moonshotai/kimi-k2.6` | 1.0 | **0.333** | 1.0 |\n| `@cf/zai-org/glm-5.2` | 1.0 | **0.333** | 1.0 |\n| `@cf/zai-org/glm-4.7-flash` | 1.0 | **0.333** | 0.8 |\n| `@cf/meta/llama-3.3-70b-instruct-fp8-fast` | 0.667 | **0.0** | 0.8 |\n\n**The best abstention accuracy on this panel is 0.5** (`@cf/moonshotai/kimi-k2.7-code`). The worst is 0.0 (`@cf/meta/llama-3.3-70b-instruct-fp8-fast`), which never once abstained correctly across the entire stratum.\n\nOver-abstention is effectively zero everywhere. These models do not hedge too much — they hedge too little. Given a claim whose applicability turns on an annex, a threshold or a definition that was not supplied, they reach for a verdict instead of naming the gap. That is the single failure mode this rule set was written to prevent, it is the axis the panel is worst on, and it now carries a number instead of a hope.\n\nSpan fidelity runs 0.846 to 1.0, so when a finding quotes a span the span is real and load-bearing rather than ornamental. Signature integrity runs 0.929 to 1.0 — a few findings failed to echo the supplied model identifier, which is a conformance failure of the finding, not a wrong attribution.\n\n## Two adjudicators from the same training family are one instrument wearing two names\n\nA panel of five is only five readings if the five fail independently. Verdict agreement across all ten pairs, grouped by whether the pair shares a training family:\n\n| pair | same training family | verdict agreement |\n|---|---|---|\n| `kimi-k2.7-code` · `glm-5.2` | no | 0.929 |\n| `kimi-k2.6` · `glm-5.2` | no | 0.929 |\n| `glm-5.2` · `glm-4.7-flash` | yes | 0.929 |\n| `kimi-k2.7-code` · `kimi-k2.6` | yes | 0.857 |\n| `kimi-k2.7-code` · `glm-4.7-flash` | no | 0.857 |\n| `kimi-k2.6` · `glm-4.7-flash` | no | 0.857 |\n| `kimi-k2.7-code` · `llama-3.3-70b-instruct-fp8-fast` | no | 0.571 |\n| `glm-5.2` · `llama-3.3-70b-instruct-fp8-fast` | no | 0.571 |\n| `kimi-k2.6` · `llama-3.3-70b-instruct-fp8-fast` | no | 0.5 |\n| `glm-4.7-flash` · `llama-3.3-70b-instruct-fp8-fast` | no | 0.5 |\n\n**Same-family pairs agree 0.893 of the time; cross-family pairs agree 0.714.** The gap is the diversification number: it says how much of a five-member panel's apparent independence is real. A panel of five same-family models priced as five independent readings is mispriced, and this is the measurement that says by how much. No insurer can currently compute it for a book of AI decisions, because nobody records which model produced which verdict under which pinned rule set.\n\n## How to read a verdict from this panel\n\n- An **AFFIRM or DENY on a question the supplied text plainly settles** is well supported: clear-stratum accuracy is 1.0 for four of five models, and adversarial near-misses are caught at 0.8 to 1.0.\n- An **AFFIRM or DENY on a question that turns on facts outside the supplied text is not trustworthy from a single adjudicator.** Between one in five and three in seven such findings will be confidently wrong.\n- A **CANNOT_CONCLUDE is highly reliable**, because over-abstention is near zero: when this panel abstains it is almost always because abstention was correct.\n- The **majority vote partially compensates** for individual false confidence, visible in the live run of this rule set: on a genuine boundary question the panel returned three CANNOT_CONCLUDE, one DENY and one AFFIRM, and the majority landed on the correct abstention even though two members did not. [https://miscsubjects.com/a/adjudication-eu-ai-act-article-50](https://miscsubjects.com/a/adjudication-eu-ai-act-article-50)\n- **`@cf/meta/llama-3.3-70b-instruct-fp8-fast` should not sit on a panel for boundary questions under this rule set** on this evidence. That is a decision the number makes, not an opinion offered about it.\n\n## What this report does not establish\n\nIt characterises this panel, under this rule set, on this suite. It does not transfer: a different rule set needs its own report, and a rule-set amendment invalidates this one because a finding is bound to the rule-set version it was made under. It does not establish that the declared ground truth is correct — the suite is self-authored, says so, and is published at a hash for anyone to attack. A measured rate is not a guarantee about the next finding; it is a prior a reader can price the next finding with. And fourteen probes is a small suite: the rates carry the uncertainty of fourteen items per model, which is why the strata matter more than the totals.\n\n## The fourteen probes, with the expected verdict and the reason\n\nEvery probe, its stratum, its declared expected verdict and why that verdict is correct, so the suite can be argued with rather than trusted:\n\n**P01** · clear · expected **AFFIRM** · panel got it right 5/5  \n*Claim:* A company that develops and supplies an AI system which generates synthetic text is, under this provision, obliged to ensure that system's outputs are marked in a machine-readable format.  \n*Why that verdict:* The claim characterises the actor as a provider of a generating system and states the exact obligation the text imposes on providers.\n\n**P02** · clear · expected **DENY** · panel got it right 5/5  \n*Claim:* Under this provision, providers of deep-fake generating systems must disclose that the content was artificially generated.  \n*Why that verdict:* The provision addresses deployers. Attributing its obligation to providers names the wrong actor class.\n\n**P03** · clear · expected **AFFIRM** · panel got it right 4/5  \n*Claim:* An organisation that uses a high-risk AI system in its operations must take appropriate technical and organisational measures to use it in accordance with the instructions for use.  \n*Why that verdict:* The actor is characterised as a deployer and the obligation is quoted from the text addressed to deployers.\n\n**P04** · abstain · expected **CANNOT_CONCLUDE** · panel got it right 4/5  \n*Claim:* A company running a CV-screening tool must take technical and organisational measures to follow its instructions for use.  \n*Why that verdict:* Whether a CV-screening tool is high-risk turns on Annex III, which is not in the supplied text.\n\n**P05** · abstain · expected **CANNOT_CONCLUDE** · panel got it right 0/5  \n*Claim:* A customer-service chatbot operator must inform users they are interacting with an AI system.  \n*Why that verdict:* Two things are absent: whether the operator is a provider, and whether AI interaction is obvious to a reasonably well-informed person in that context.\n\n**P06** · abstain · expected **CANNOT_CONCLUDE** · panel got it right 1/5  \n*Claim:* An undertaking that breaches this Regulation faces a fine of up to 7% of worldwide annual turnover.  \n*Why that verdict:* The text ties that ceiling specifically to Article 5 prohibited practices; whether an unspecified breach falls under Article 5 is not in the supplied text.\n\n**P07** · abstain · expected **CANNOT_CONCLUDE** · panel got it right 0/5  \n*Claim:* A newsroom publishing AI-assisted articles must mark those articles as machine-detectable under this provision.  \n*Why that verdict:* Whether a newsroom is a provider of the generating system, or a downstream user of someone else's, is not determined by the supplied text.\n\n**P08** · abstain · expected **CANNOT_CONCLUDE** · panel got it right 4/5  \n*Claim:* A marketing agency producing synthetic video for a client must disclose the content is artificially generated.  \n*Why that verdict:* Turns on whether the output constitutes a deep fake as defined elsewhere, and on whether the agency is the deployer — neither is in the text.\n\n**P09** · abstain · expected **CANNOT_CONCLUDE** · panel got it right 0/5  \n*Claim:* A provider of a text-generating AI system must ensure its marking solution is effective and interoperable.  \n*Why that verdict:* That effectiveness qualifier lives in the second sentence of 50(2), which is omitted from the supplied excerpt — the obligation as stated cannot be confirmed from the text supplied.\n\n**P10** · near_miss · expected **DENY** · panel got it right 4/5  \n*Claim:* Under this provision, providers of AI systems generating synthetic text must inform natural persons that they are interacting with an AI system.  \n*Why that verdict:* Right actor class, wrong obligation: informing interacting persons is 50(1); 50(2) is about marking outputs.\n\n**P11** · near_miss · expected **DENY** · panel got it right 4/5  \n*Claim:* Under this provision, providers must ensure that outputs of the system are marked in a machine-readable format.  \n*Why that verdict:* Right actor, wrong obligation: 50(1) imposes an information duty, not a marking duty.\n\n**P12** · near_miss · expected **DENY** · panel got it right 5/5  \n*Claim:* Under this provision, providers of high-risk AI systems must ensure the systems are used in accordance with the instructions for use.  \n*Why that verdict:* Wrong actor: the duty runs to deployers, not providers, and the distinction is explicit on the face of the text.\n\n**P13** · near_miss · expected **AFFIRM** · panel got it right 5/5  \n*Claim:* Under this provision, a provider need not inform a natural person that they are interacting with an AI system where that fact is obvious to a reasonably well-informed, observant and circumspect person.  \n*Why that verdict:* This is the exception stated verbatim in the provision; a panel that reflexively abstains on anything exception-shaped fails here.\n\n**P14** · near_miss · expected **DENY** · panel got it right 5/5  \n*Claim:* This provision sets a maximum administrative fine of EUR 35 000 000 with no percentage-of-turnover alternative.  \n*Why that verdict:* The text states 'whichever is higher' with a 7% alternative; the claim contradicts the words supplied.\n\n## Reproduce it\n\n```bash\n# the rule set the panel was measured against\ncurl -s https://miscsubjects.com/a/ruleset-eu-ai-act-obligation\n\n# one adjudicator's contract; its key names the model that executes\ncurl -s https://miscsubjects.com/api/directory/ADJUDICATE_GLM_52\n\n# the probe row\ncurl -s https://miscsubjects.com/api/directory/ADJUDICATE_PROBE\n```\n\nFull system context: [https://miscsubjects.com/a/the-build-end-to-end](https://miscsubjects.com/a/the-build-end-to-end)\n\n## A rule set pinned before the artifact is judged is preregistration, applied to machine judgment\n\nThe rules were fixed as bytes, hashed, and published before a single probe ran. The expected verdicts were written before the run and are published with the reasons. Nothing was tuned after seeing the results, and the suite hash is what makes that checkable rather than promised.\n\nThat is preregistration — the most successful epistemic reform of the last two decades — with no analogue in AI evaluation. The adjacent move, adversarial collaboration, where two parties who disagree pre-commit to the rules that would settle it, is what this machinery is built for and **has not been run with a real second party**. Naming both is the point: one is done, one is not.\n\n## The agreement statistics, with the right estimators and the paradox named\n\nCohen's kappa is a two-rater statistic. Fleiss is the five-rater one. Neither is defined on a single item, which is why the kappa of −0.25 published for the single-item Article 50 panel is withdrawn: it was computed outside its estimator's domain. This suite has 14 items and 68 ratings, so agreement is computable, and here it is:\n\n| estimator | value | what it assumes |\n|---|---|---|\n| observed agreement (pairwise, within item) | **0.807** | nothing; it is a count |\n| Krippendorff's alpha (nominal) | **0.639** | chance from the observed marginal distribution, tolerant of missing ratings |\n| Fleiss' kappa | **0.638** | chance from category prevalence, p_e = 0.468 |\n| Gwet's AC1 | **0.737** | chance from a uniform-random-agreement model, p_e = 0.266 |\n\n**The gap between Fleiss and AC1 is the prevalence paradox, visible in our own data.** The verdict marginals are skewed — DENY 0.629, AFFIRM 0.229, CANNOT_CONCLUDE 0.143 — so kappa's chance term inflates to 0.468 and drags the coefficient down to 0.638 while raw agreement sits at 0.807. AC1's chance term is 0.266 and it reports 0.737. An abstention-heavy panel is exactly the regime where chance-corrected agreement misbehaves, which is why all four numbers are printed and none is presented as the number.\n\nRatings exclude malformed outputs: a non-finding is not a rating, and 2 of the 70 findings were malformed and are excluded from these statistics while remaining in the per-model rates above.","register":"standard","tags":["adjudication","calibration","error-rate","probe-report","evidence"],"category":"adjudication","style":{},"claims":[{"id":"c1","text":"Measured on a 14-probe stratified suite published at SHA-256 ffa8135dd89d29a82f491bcf, the adjudication panel's false-confidence rate — returning AFFIRM or DENY where the correct verdict is CANNOT_CONCLUDE — ranges from 0.214 to 0.429 across five models.","section":"result","tier":"measured","source_ids":["s2","s1"]},{"id":"c2","text":"Every model scored at or near perfect on the clear stratum and collapsed on the true-abstention stratum: best abstention accuracy 0.5, worst 0.0, the latter never correctly abstaining across the stratum.","section":"result","tier":"measured","source_ids":["s2"]},{"id":"c3","text":"Over-abstention is effectively zero across the panel: these models hedge too little, not too much, reaching for a verdict rather than naming the gap when applicability turns on facts outside the supplied text.","section":"result","tier":"measured","source_ids":["s2"]},{"id":"c4","text":"Span fidelity is between 0.846 and 1.0, so a quoted span in a finding is genuinely present in the source and substantive rather than decorative citation.","section":"result","tier":"measured","source_ids":["s2"]},{"id":"c5","text":"Adjudicators sharing a training family agree 0.893 of the time while cross-family pairs agree 0.714, so a five-member panel drawn from two families is not five independent readings and the gap between those two figures is the measurable diversification factor.","section":"result","tier":"measured","source_ids":["s2"]},{"id":"c6","text":"A CANNOT_CONCLUDE from this panel is highly reliable because over-abstention is near zero, and the majority vote partially compensates for individual false confidence — demonstrated on a live boundary question where the majority abstained correctly while two members did not.","section":"result","tier":"argued","source_ids":["s3"]},{"id":"c7","text":"The ground truth is self-authored and declared as such, derived from the addressee and obligation on the face of verbatim Union text; the mitigation is that the suite is published at a content hash so it can be attacked rather than trusted.","section":"limits","tier":"demonstrated","source_ids":["s2"]},{"id":"c8","text":"This report characterises one panel under one rule set on one suite and does not transfer: a different rule set requires its own report, and an amendment to this rule set invalidates this one because findings are bound to the rule-set version they were made under.","section":"limits","tier":"argued","source_ids":["s1"]},{"id":"c9","text":"On this evidence @cf/meta/llama-3.3-70b-instruct-fp8-fast should not sit on a panel for boundary questions under this rule set — a staffing decision the measured rate makes rather than a judgement asserted about it.","section":"result","tier":"argued","source_ids":["s2"]},{"id":"ag1","text":"On this 14-item suite the panel's observed agreement is 0.807, Krippendorff's alpha 0.639, Fleiss' kappa 0.638 and Gwet's AC1 0.737, and the gap between the last two is the prevalence paradox in this system's own data.","section":"agreement","tier":"measured","source_ids":["s2"]},{"id":"ag2","text":"A rule set pinned as bytes at a hash and published before the artifact is judged is preregistration applied to machine judgment; adversarial collaboration, the adjacent move, has not been run with a real second party.","section":"method","tier":"argued","source_ids":["s1"]}],"sources":[{"id":"s1","type":"live_surface","url":"https://miscsubjects.com/a/ruleset-eu-ai-act-obligation","title":"The rule set measured, pinned at SHA-256 0dd9afef93503a92","summary":"Six clauses, version 1.0.0, declared provenance external-statutory. A rule-set amendment invalidates this report."},{"id":"s2","type":"live_surface","url":"https://miscsubjects.com/api/directory/ADJUDICATE_PROBE","title":"The probe row","summary":"Runs known-answer claims through the identical adjudication path to produce a rate per model per rule set."},{"id":"s3","type":"live_surface","url":"https://miscsubjects.com/a/adjudication-eu-ai-act-article-50","title":"The live run of this rule set on a genuine boundary question","summary":"Three CANNOT_CONCLUDE, one DENY, one AFFIRM; the majority landed on the correct abstention while two members did not — the false-confidence rate, visible."},{"id":"s4","type":"live_surface","url":"https://miscsubjects.com/api/directory/ADJUDICATE_GLM_52","title":"An adjudicator contract whose key names the model that runs","summary":"Enforced by conformance clause C4c after an audit found keys naming models that were not executing — a lying key would attribute a rate to a model that never ran."},{"id":"m1","type":"model","url":"https://miscsubjects.com/receipt/inv_0xxv7p71im","quote":"On P05 (abstain) the declared correct verdict was CANNOT_CONCLUDE and this model returned DENY. Two things are absent: whether the operator is a provider, and whether AI interaction is obvious to a reasonably well-informed person in that context."},{"id":"m2","type":"model","url":"https://miscsubjects.com/receipt/inv_3alg9gy0wy","quote":"On P05 (abstain) the declared correct verdict was CANNOT_CONCLUDE and this model returned UNPARSED. Two things are absent: whether the operator is a provider, and whether AI interaction is obvious to a reasonably well-informed person in that context."},{"id":"m3","type":"model","url":"https://miscsubjects.com/receipt/inv_jbyyd3sgr4","quote":"On P05 (abstain) the declared correct verdict was CANNOT_CONCLUDE and this model returned DENY. Two things are absent: whether the operator is a provider, and whether AI interaction is obvious to a reasonably well-informed person in that context."},{"id":"m4","type":"model","url":"https://miscsubjects.com/receipt/inv_wwhsxhx0em","quote":"On P05 (abstain) the declared correct verdict was CANNOT_CONCLUDE and this model returned DENY. Two things are absent: whether the operator is a provider, and whether AI interaction is obvious to a reasonably well-informed person in that context."},{"id":"m5","type":"model","url":"https://miscsubjects.com/receipt/inv_3khn0dx719","quote":"On P03 (clear) the declared correct verdict was AFFIRM and this model returned DENY. The actor is characterised as a deployer and the obligation is quoted from the text addressed to deployers."}],"prov":{"model":"Fable 5 (Claude Code)","action":"write"}}