# Calibration, measured: 30 oracle-labelled cases through the production gate — seat accuracy, wrongful authorisation, and the price of deferral.

slug: adjudication-calibration-study · https://miscsubjects.com/a/adjudication-calibration-study · tags: governance, adjudication, calibration, evaluation · updated 2026-07-30T13:30:37.405Z

## What this study is

Every page on this site that claims anything ends with the same admission: no calibration study establishes correctness at a known rate. This page is that study — the first one — run on 30 oracle-labelled synthetic cases, balanced across the three outcomes a governed decision can honestly take: should-affirm, should-deny, and should-abstain (a record deliberately withheld, with a manifest naming the absence). Every case is hashed, every seat call is a permanent receipt, and every number below is computed from the result files, not written by hand.

The design: each case runs through three model seats across two model families under decision-constitution@1.3.3 — the same production rows any external case goes through — and the surviving findings are sealed by the derivation-agreement gate, bound to the case's hashes. Two different questions get separate answers: **how often is a seat wrong** (seat calibration), and **how often does the gate authorise a wrong answer** (gate calibration). The second is the one a regulator, an underwriter, or a counterparty actually needs.

## Per-seat calibration

| Seat | valid findings | verdict accuracy | wrongful AFFIRM | over-abstention | under-abstention | transport failures |
|---|---|---|---|---|---|---|
| glm-5.2 (zhipu) | 30 | 100.0% | 0.0% | 0.0% | 0.0% | 0 |
| kimi-k2.7-code (moonshot) | 30 | 96.7% | 0.0% | 3.3% | 0.0% | 0 |
| glm-4.7-flash (zhipu) | 22 | 95.5% | 0.0% | 4.5% | 0.0% | 8 |

Definitions, exactly: *verdict accuracy* is agreement with the oracle label. *Wrongful AFFIRM* is affirming when the oracle is not AFFIRM — the seat-level version of the worst failure. *Over-abstention* is CANNOT_CONCLUDE on a determinate case; *under-abstention* is a verdict on a case whose oracle is CANNOT_CONCLUDE. *Transport failures* are calls that returned nothing usable after three attempts and produced no finding at all — they can never authorise anything, and they are counted rather than hidden.

Aggregate: 80 of 82 valid findings matched the oracle (97.6%); 0 wrongful affirmations at seat level (0.0%).

## Gate calibration — the number that matters

**Zero wrongful authorisations at the gate.** Across all 30 cases, no APPROVE sealed on a case whose oracle label was not AFFIRM.

Outcome distribution across the 30 sealed panels: APPROVE 6 · NEGATE 0 · NO_ACTION 6 · ESCALATE 10 · no seal 8. The gate sealed the oracle-matching outcome in 12 of 30 cases.

Read the ESCALATE number correctly: an escalation on a determinate case means the seats agreed on the verdict but not derivation-for-derivation, so the gate refused to conclude and referred the case to a human. That is deferral cost, not decision error — the human sees a unanimous panel with its reasoning preserved. The trade the gate makes is explicit: it spends deferrals to buy down wrongful authorisations.

## Every case, every receipt

| Case | Oracle | Seat verdicts (✓ = matched oracle) | Seal |
|---|---|---|---|
| calib-01 | AFFIRM | glm52:✓ · kimi27:✓ · flash:✓ | ESCALATE (2 sigs) [receipt](/receipt/inv_n389a3mjbb) |
| calib-02 | AFFIRM | glm52:✓ · kimi27:✓ · flash:— | no seal |
| calib-03 | AFFIRM | glm52:✓ · kimi27:✓ · flash:✓ | ESCALATE (1 sig) [receipt](/receipt/inv_lwopl2j1g9) |
| calib-04 | AFFIRM | glm52:✓ · kimi27:✓ · flash:✓ | APPROVE (1 sig) [receipt](/receipt/inv_1g29owp6uc) |
| calib-05 | AFFIRM | glm52:✓ · kimi27:✓ · flash:✓ | ESCALATE (2 sigs) [receipt](/receipt/inv_0y4n5a25wh) |
| calib-06 | AFFIRM | glm52:✓ · kimi27:✓ · flash:✓ | APPROVE (1 sig) [receipt](/receipt/inv_aufcl5bba9) |
| calib-07 | AFFIRM | glm52:✓ · kimi27:✓ · flash:✓ | APPROVE (1 sig) [receipt](/receipt/inv_rvk831nucm) |
| calib-08 | AFFIRM | glm52:✓ · kimi27:✓ · flash:✓ | APPROVE (1 sig) [receipt](/receipt/inv_ttkdt41g6p) |
| calib-09 | AFFIRM | glm52:✓ · kimi27:✓ · flash:✓ | APPROVE (1 sig) [receipt](/receipt/inv_629ci47ape) |
| calib-10 | AFFIRM | glm52:✓ · kimi27:✓ · flash:✓ | APPROVE (1 sig) [receipt](/receipt/inv_h1303vtn5s) |
| calib-11 | DENY | glm52:✓ · kimi27:✓ · flash:✓ | ESCALATE (2 sigs) [receipt](/receipt/inv_pinneygopf) |
| calib-12 | DENY | glm52:✓ · kimi27:CANNOT_CONCLUDE · flash:CANNOT_CONCLUDE | ESCALATE (2 sigs) [receipt](/receipt/inv_6dp16egktl) |
| calib-13 | DENY | glm52:✓ · kimi27:✓ · flash:— | no seal |
| calib-14 | DENY | glm52:✓ · kimi27:✓ · flash:— | no seal |
| calib-15 | DENY | glm52:✓ · kimi27:✓ · flash:✓ | ESCALATE (3 sigs) [receipt](/receipt/inv_bay9gmz5ye) |
| calib-16 | DENY | glm52:✓ · kimi27:✓ · flash:— | no seal |
| calib-17 | DENY | glm52:✓ · kimi27:✓ · flash:— | no seal |
| calib-18 | DENY | glm52:✓ · kimi27:✓ · flash:✓ | ESCALATE (2 sigs) [receipt](/receipt/inv_iw0ce8ikr8) |
| calib-19 | DENY | glm52:✓ · kimi27:✓ · flash:— | no seal |
| calib-20 | DENY | glm52:✓ · kimi27:✓ · flash:✓ | ESCALATE (3 sigs) [receipt](/receipt/inv_y457njtkpp) |
| calib-21 | CANNOT_CONCLUDE | glm52:✓ · kimi27:✓ · flash:✓ | NO_ACTION (1 sig) [receipt](/receipt/inv_okukok57r6) |
| calib-22 | CANNOT_CONCLUDE | glm52:✓ · kimi27:✓ · flash:✓ | NO_ACTION (1 sig) [receipt](/receipt/inv_mevidc50zd) |
| calib-23 | CANNOT_CONCLUDE | glm52:✓ · kimi27:✓ · flash:✓ | ESCALATE (1 sig) [receipt](/receipt/inv_9yt658vl2s) |
| calib-24 | CANNOT_CONCLUDE | glm52:✓ · kimi27:✓ · flash:✓ | NO_ACTION (1 sig) [receipt](/receipt/inv_mdq2auo40d) |
| calib-25 | CANNOT_CONCLUDE | glm52:✓ · kimi27:✓ · flash:✓ | NO_ACTION (1 sig) [receipt](/receipt/inv_torv6rjcl0) |
| calib-26 | CANNOT_CONCLUDE | glm52:✓ · kimi27:✓ · flash:— | no seal |
| calib-27 | CANNOT_CONCLUDE | glm52:✓ · kimi27:✓ · flash:— | no seal |
| calib-28 | CANNOT_CONCLUDE | glm52:✓ · kimi27:✓ · flash:✓ | NO_ACTION (1 sig) [receipt](/receipt/inv_f7rbin5346) |
| calib-29 | CANNOT_CONCLUDE | glm52:✓ · kimi27:✓ · flash:✓ | NO_ACTION (1 sig) [receipt](/receipt/inv_rtovfpnpdz) |
| calib-30 | CANNOT_CONCLUDE | glm52:✓ · kimi27:✓ · flash:✓ | ESCALATE (2 sigs) [receipt](/receipt/inv_nzxpnujkzv) |

## What is not satisfied

The suite is synthetic and bounded: three rule shapes (roster access, fee-with-waiver, permit-with-cap), determinate by construction, ten cases per outcome. It measures calibration on clean fixtures — the floor, not the field. Contested language, adversarial records, and genuinely ambiguous cases are absent by design, and rates measured here must not be quoted as expected performance on real disputes. The next calibration layer is externally submitted cases, which is what the intake on every use-case page exists to collect. The full case set, harness, and raw results are in the repository (scripts/calibration_cases.mjs, scripts/calibration_run.mjs), and each seal receipt above opens to the complete bound record.

## Submit a case

Send one bounded question — a rule set and a record — to **build@miscsubjects.com**. It runs through exactly the machinery measured on this page, and what returns is the full governed panel with its permanent record.