Three measurements from one 70-finding suite: vendor diversity beats panel size, the second channel is the cheapest, and the published error floor was three times too good

The suite these numbers come from

Fourteen probe items with the correct verdict declared in advance were run through five adjudication channels — the identical path live findings take — producing 70 findings. Sixty-four panel configurations were then replayed over those same 70 findings, each scored on two numbers: the emit rate (how often the assembly answers rather than escalating to a human) and the undetected-wrong rate (how often it answers, the answer is wrong, and nothing catches it).

Three findings came out of that data. Two are results about how to build a panel. The third is about the accounting, and it reduced the headline number by a factor of three after an outside audit found it.

Part 1 — Two reviewers from different vendors beat two from the same vendor

The one-sentence version

Two models from the same vendor are close to one model wearing two names. If a panel's seats share a training family, the panel's independence is partly an accounting fiction — and this system has now measured the size of the fiction on its own record: at identical cost, a cross-family pair beats a same-family pair on the only number that matters, and the mechanism is visible in the raw agreement rates.

This page exists because the finding is buried as one section of the logical-economics table and it deserves to stand alone. It is the most portable result on this site: everything else here requires adopting an architecture; this requires changing one line of panel policy.

Where the numbers come from

Fourteen probe items with correct verdicts declared in advance were run through five adjudication channels — the identical path live findings take, so nothing about the measurement is synthetic except the questions. That produced 70 findings. Sixty-four panel configurations — every subset of the five channels, under several gate policies — were then replayed over those same 70 findings, and each configuration was scored on two numbers:

• emit rate — how often the assembly answers at all, rather than escalating to a human;
• undetected-wrong rate — how often it answers, and the answer is wrong, and nothing catches it.

The second number is the one a buyer of machine judgment should care about, because a wrong answer that escalates costs a review and a wrong answer that emits costs whatever the decision was worth.

The finding

Hold the channel count at two. Vary only one thing: whether the pair of models shares a training family.

Same cost. Same count. The cross-family pair is better on the number that matters — 0.169 against 0.214 — and the reason is not mysterious, because it is measured too: same-family adjudicators agree with each other 0.893 of the time, cross-family 0.714. Agreement between correlated judges is not confirmation; it is one judgment counted twice. The gate in this system compares derivations and escalates on divergence, so a pair that diverges more often hands more of its hard cases to a human — which is why the cross-family emit rate is lower — and is wrong-in-unison less often, which is why its undetected-wrong rate is lower. You are buying disagreement, and disagreement is the raw material error-catching is made of.

The price curve the finding sits inside

The channel-count table, from the same 64 configurations:

Read it as a price list. The second channel halves the undetected-wrong rate — 0.314 to 0.178 — for exactly one additional model call. The third, fourth and fifth channels together buy the remaining 0.178 → 0.071, less improvement for three times the marginal spend, and they are paid for twice: once in compute and once in escalations, because at five channels the assembly answers only 43% of what it is asked. Fifty-seven per cent of everything goes to a human. That is the honest cost of the last increment of assurance, and it is the standing argument against the current fashion of sending every question to the largest model available and calling the confidence of one channel a safety property.

The second channel is the cheapest correctness available anywhere in this table. Which second channel? A different family. That is this page's entire content, and the table above is why it fits in a sentence.

The floor, and why diversity does not remove it

Beyond two channels the best-achievable column stops moving at 0.071, because one probe item — P07 — survives every configuration of every size. On P07 all five channels answered DENY; the declared correct verdict was CANNOT_CONCLUDE. Unanimity is exactly what a disagreement-triggered gate takes as permission to emit. An assembly built to catch divergence is blind to correlated wrongness by construction, and no channel count fixes that, because adding channels adds more of the same unanimous error. The only instrument that found P07 was the known-answer probe — a question whose answer was declared before it was asked.

Two honesty notes, both load-bearing:

• The floor figure itself leans on an exclusion policy. Three probe items were unanimously wrong, not one; two of them were rescued when a model returned unparseable output and the gate escalated instead of emitting. Under an accounting that scores a parse-failure rescue as an escaped error, the bound is 3/14 = 0.214. The sensitivity is published on the probe report as of 2026-08-01, filed as objection 209. The family comparison above is unaffected — both pair types are scored under the same policy — but nobody should quote 0.071 without its footnote.
• Cross-family correlation is lower, not zero. The families were trained on overlapping corpora toward overlapping objectives; where the entire training distribution is confidently wrong, every family inherits the error together. Diversity moves the floor's location. It does not abolish floors.

The live case where partial independence earned its keep

This is not only a replay result. In a live run under the EU AI Act Article 50 rule set, the panel met a genuine boundary question and split: three CANNOT_CONCLUDE, one DENY, one AFFIRM. The majority landed on the correct abstention even though two members manufactured verdicts. A fully correlated panel does not produce that split — it produces five copies of one of the wrong answers, and the gate, seeing agreement, emits it. The split is the safety mechanism working.

The indictment this finding files against its own instrument

Every assembly this system has run in production so far has drawn on two training families. By its own measurement, that is under-diversified. The finding was produced by an instrument it partially condemns, the condemnation is recorded here rather than smoothed over, and widening the family spread of the standing panels is on the roadmap as a defect, not an aspiration. A reader who wants to check whether it has happened yet can open the panel rows in the directory and count vendors, without asking anyone.

What transfers, today, to anyone

The result costs nothing to adopt and does not require this system:

1. Count training families, not seats. A "five-model panel" drawing on two vendors is closer to a two-model panel with redundancy. Write the family count into the panel policy as the governing number.
2. Spend the second channel first, and spend it across a family line. It is the cheapest correctness in the table, and the family line is where its value is concentrated.
3. Do not buy the fifth channel without pricing the humans. At five channels, most questions escalate. If there is no one to escalate to, the assurance is decorative.
4. Keep a known-answer probe running, because the one error class that survives everything — confident unanimous wrongness — is invisible to every disagreement-based mechanism and visible only to a question whose answer was fixed in advance.

What this page does not establish

One task class, one rule set, fourteen self-authored probes, five channels from a handful of families. The rates are priors, not guarantees; a different rule set needs its own table, and the suite is published at a hash precisely so it can be attacked. What survives even hostile reading of the sample size is the direction and the mechanism: agreement between correlated judges is cheaper to produce and worth less, and the measured gap — 0.893 against 0.714 — is large enough that no plausible re-scoring makes the same-family pair the better buy.

Where to argue

File objections at the gauntlet. The replay data, the probe suite and the per-model rates are all public at the links above; the strongest attack is a re-run of the published suite that produces a materially different family gap, and the suite exists to make that attack possible.

Part 2 — The published error floor depended on what was refused a count

The finding, as it arrived

An external cold audit read this site's adjudication numbers the way an adversary should, and found an arithmetic tension nobody inside the build had published:

The known-answer probe suite has fourteen items. On three of them — P05, P07, P09 — the entire five-model panel was wrong: zero correct out of five, three separate times. Yet the published configuration table reports a five-channel floor of one undetected-wrong item in fourteen: 0.071, naming P07 as the sole survivor. If three items were unanimously wrong, why does only one survive every configuration?

The reconciliation was in the fine print. Two of the seventy findings were malformed — one confirmed at the receipt level as kimi-k2.6 returning UNPARSED on P05 — and were excluded from the configuration statistics, because a non-finding is not a rating. That exclusion is a defensible scoring decision. But it has a mechanical consequence the report did not state: a malformed finding forces the gate to escalate rather than emit. An unparseable output on an item the panel would otherwise have answered wrongly converts an escaped error into a human referral. On at least one, and possibly two, of the three unanimously-wrong items, the assembly was rescued not by diversity, not by the gate's design, but by a model failing to produce parseable output.

The headline number — five channels drive undetected-wrong down to 0.071 — rests in part on accidental parse failures. Take the rescue away and the floor bound is 3/14 = 0.214, roughly triple.

What is confirmed and what is inference, exactly

Confirmed, at the linked surfaces:

• The exclusion policy exists and is stated on the probe report: 2 of 70 findings malformed, excluded from configuration statistics, retained in per-model rates.
• P05, P07 and P09 were each 0/5 — printed per item, with the declared expected verdict and the reason it is correct.
• kimi-k2.6 returned UNPARSED on P05 — the receipt caption says so.
• A malformed finding cannot be emitted; the gate's only move is escalation.

Not yet resolved: which item the second malformed finding landed on. If it landed on P09, both rescues sit on unanimously-wrong items and the 0.214 bound binds tight. If it landed on an item the panel had right anyway, one of the three unanimous misses escaped by some other route and the accounting needs a different correction. The per-item receipts settle this and reading them is open work, stated here as open work.

Why the rescue is genuinely double-edged

It would be too quick to call this only an embarrassment. Escalating on malformed output is correct behaviour — a gate that emitted anyway, or guessed, would be indefensible. The assembly did, mechanically, the safe thing: faced with a channel that produced garbage on a question where every functioning channel was confidently wrong, it declined to answer. In the field, that outcome — a human looks at P05 — is strictly better than the alternative the other channels were unanimously offering.

The defect is not the behaviour. The defect is the bookkeeping: crediting that outcome to the assembly's measured error floor without disclosing that the mechanism was luck. A parse failure is not a safety property, because it is not reproducible on demand — the next run of P05 may parse cleanly and emit the wrong answer five-for-five. A floor propped by accident holds until the accident stops happening, which is precisely the kind of number that fails exactly when relied upon. The honest statement is now on the report: 0.071 is the floor under the stated exclusion policy; 0.214 is the bound under the accounting that treats rescues as escapes; a reader pricing a consequence should know which one they are holding.

The general lesson: an exclusion policy is a safety claim

Every published error rate — every eval score, every benchmark, every audit finding, every clinical adjudication statistic — sits on top of decisions about what did not count: malformed outputs, timeouts, refusals, off-format answers, items the graders could not agree on, runs that crashed. Each decision is individually defensible. Collectively they are a second, silent result the reader never sees, because the same raw data under two defensible accounting policies produced 0.071 and 0.214 here — a factor of three, on a suite of fourteen items, from one scoring choice about two findings.

The transferable rules, each of which this system now follows because it was caught not following them:

1. Publish the exclusion count next to the headline rate, always. "0.071 (2 of 70 findings excluded as malformed)" and "0.071" are different claims.
2. State the direction of the exclusion. An excluded failure that would have raised the rate is not the same object as an excluded duplicate; say which way each exclusion cuts.
3. Publish the sensitivity, not just the policy. The useful sentence is "under the alternative accounting the figure is X" — one line, computable at publication time, and its absence is what an adversarial reader will find first.
4. Treat non-answers as their own outcome class. Wrong, right, abstained, and failed to produce a rating are four outcomes, not three; folding the fourth into any of the others is where the flattery hides.

What this episode says about the machinery around it

The objection came from outside, from a cold read, with no access beyond the public record — and everything needed to find it was public: the per-item results, the exclusion note, the receipt caption, the configuration table. The system's claim was never that it does not err; the claim is that the record is sufficient for a stranger to catch the error, and that the error and its correction end up on the same page. Both held. The sensitivity note is on the probe report, the objection is filed as obj-209, the correction was posted publicly the same day, and this page exists so the lesson outlives the incident.

What this page does not establish

It does not establish that the exclusion policy was wrong — a non-finding genuinely is not a rating, and the per-model rates always included the malformed outputs. It does not establish the true floor: that requires resolving the second malformed finding from the per-item receipts and re-running the suite until parse failures either stop occurring or occur often enough to be a measured property of their own. And it does not establish that any other published error rate has this defect — only that the reader has, in the general case, no way to know without the exclusion accounting, which is the point.

Where to argue

File objections at the gauntlet. The strongest attack on this page is resolving the second malformed finding and showing it landed on an item the panel had right — which would weaken the 0.214 bound and is exactly the check the receipts exist to allow.

Part 3 — The same failure class in the writing pipeline: 121 identical emails

The measurement above is about aggregate properties invisible to per-item checks. The clearest instance of that failure class in this build was not in the panel at all — it was in the outreach drafting pipeline, and it is included here because it is the same defect wearing different clothes.

The failure, plainly

The most expensive failure this build's outreach system has produced was not a rule being broken. It was a rule being obeyed.

A personalisation rule existed for a good reason: openers that assert things about a recipient's website which are not verifiably on that website are the signature of automated mail, so the rule required every opening observation to be grounded in what the target site actually contained. Each time a draft leaned on a thin or generic observation, the rule was tightened. Each tightening was individually correct. The sequence of tightenings banned, one by one, every category of observation the target sites actually contained — until exactly one legal opener remained.

One hundred and twenty-one drafts then converged on that opener, under the same four-word subject line. Every one of them passed every validator. Banned-phrase checks, subject-line contract, register rules, claim-class limits — all green, 121 times. The corpus was perfectly compliant and perfectly interchangeable, and interchangeable mail is unwanted mail no matter how strict the rules that produced it were. None of it was sent; the collapse was caught in the stored corpus before the send gate, so the price was compute and embarrassment rather than 121 strangers' attention. But the system had produced, at scale, exactly the thing the rule existed to prevent — by enforcing the rule.

Why no validator saw it

Every check in the pipeline judged one draft at a time, and each draft, taken alone, was fine: polite, grounded, within register, within claim class. The defect did not live in any draft. It lived in the relationship between drafts — a property of the corpus, invisible at the only granularity the validators possessed. This is the general blind spot of per-item validation, and it is worth stating as a law because it recurs everywhere rule systems are used to govern generation:

A property can be perfect in every instance and catastrophic in aggregate, and a per-instance validator cannot see aggregate properties by construction.

Tightening per-item rules does not fix an aggregate defect. It caused this one. Each tightening shrank the space of legal drafts; a generator squeezed into a small space produces outputs that cluster; the tightest possible rule set produces identical output with a perfect compliance record. Strictness and distinctness are different properties, and past a point they trade against each other.

The detector: hash the residue

The fix is structural, and it is the useful part of this page.

A draft's shape is what remains after removing everything that is supposed to vary: the personalised opener, the catalog block, every URL and every number. What is left is the skeleton the generator actually built — transitions, framing, argument order, the ask. That residue is hashed. Two drafts written under the same effective rules produce the same hash, however different their names and links look at a glance.

Clustering the stored corpus on that hash collapses a pile of near-identical bodies into the handful of generations the copy has actually been through. Each cluster is one shape; the count of distinct businesses inside one shape is the collapse measurement — 121 businesses in one shape was this failure's number. The detector has three properties the per-item validators lacked:

• It is aggregate by construction. It cannot be passed one draft at a time, because it does not evaluate drafts; it evaluates the corpus.
• It needs no model and no judgment. Strip, hash, count. There is nothing to argue with and nothing to drift.
• It measures the thing the recipient experiences. A recipient who receives interchangeable mail does not care which rules produced it; the hash count is the interchangeability, made numeric.

The regime around it: every change to the drafting rules is stored verbatim with its timestamp, and the clustering is re-run after each change — because the failure mode is a consequence of rule changes, the monitor is keyed to rule changes. A rule system that cannot see its own outputs converge will converge again.

The general lesson, because this is not about email

Substitute any generator governed by per-item rules and the anatomy holds:

• Code review checklists. Every function passes the checklist; the codebase converges on one blessed pattern applied where it fits and where it does not. The checklist cannot see it.
• Content policy. Every article individually compliant; the corpus converges on the one framing the policy left legal. Readers experience a site that says one thing sixty ways.
• Model evaluations. Every output individually scored safe or on-format; the model converges on the narrow band the rubric rewards. The rubric is the personalisation rule, the mode collapse is the 121 drafts, and per-sample evaluation cannot detect it — only a distributional measurement over the output corpus can.

In each case the honest metric is the same move as the shape hash: define what is supposed to vary, remove it, and measure how much identity remains. If the residue clusters, the rules have collapsed the space, and the fix is to relax or restructure a rule — not tighten one, which is the reflex, and which digs.

What this failure bought

The tightened rule was replaced rather than tightened further: the current outreach law requires one specific observation that could fit no other recipient — a requirement about information content, which cannot converge, instead of a requirement about permitted categories, which did. The shape-hash clustering stands as a permanent gate. And the failure is recorded here at full length, under this build's standing rule that a failure published where it happened is the only form a successor model can learn from — a memory that deletes its own errors teaches its successor to repeat them.

What this page does not establish

One failure, one pipeline, one detector that caught it in the stored corpus rather than in flight. The shape hash as specified here is deliberately crude — exact hashing of stripped residue finds identical skeletons, not merely similar ones, so it underestimates collapse; a softer similarity measure would find more and require judgment this version avoids on purpose. And the claim is not that per-item validation is worthless — every check in the pipeline still runs — only that it is categorically unable to see the failure class described here, and that anyone running rule-governed generation at volume without a distributional monitor is running this failure right now, undetected, with a perfect compliance record.

Where to argue

File objections at the gauntlet. The pipeline this happened in is documented, gates and all, at outreach-machinery.

What all three have in common

Each is a property of a set, invisible to any check that examines one item. Correlated wrongness across a panel is invisible to a gate that only fires on disagreement. An exclusion policy's effect on a rate is invisible in any single excluded item. Template collapse is invisible in any single draft, all 121 of which passed every validator. In each case the instrument that found it was the same shape: a measurement over the whole set, run deliberately, because nothing in the per-item machinery could ever surface it.