# SR 11-7 requires independent model validation with documented effective challenge. For an LLM, there is no instrument. Here is one.

slug: cro-model-validation-instrument · https://miscsubjects.com/a/cro-model-validation-instrument · tags: governance, model-risk, adjudication, use-case · updated 2026-07-30T13:29:17.239Z

## The obligation nobody has an instrument for

SR 11-7 — the Federal Reserve and OCC's *Supervisory Guidance on Model Risk Management*, issued April 2011 and still the governing text — and its OCC twin, Bulletin 2011-12, require that every model a bank relies on be **independently validated**. Not reviewed. Validated, by people organizationally independent of the developers, with three named components:

1. **Evaluation of conceptual soundness** — evidence that the model's design and construction are fit for purpose, including the quality of its inputs.
2. **Ongoing monitoring** — evidence that it keeps behaving as designed once in use, including benchmarking against alternatives.
3. **Outcomes analysis** — comparison of model outputs to actual outcomes, with the residual error quantified.

Running through all three is the phrase the examiners actually test for: **effective challenge** — "critical analysis by objective, informed parties who can identify model limitations and assumptions and produce appropriate changes." Challenge that leaves no artifact is challenge an examiner will not credit.

For a regression model or a Monte Carlo engine this is a mature discipline: holdout samples, backtesting, sensitivity analysis, champion-challenger runs. For a large language model exercising judgement — reading a covenant, classifying a transaction, screening an alert — **none of that toolkit applies as-is**. There is no likelihood function to backtest. The "model" is a prompt, a temperature, and a vendor checkpoint that changes under your feet. And SR 11-7 explicitly scopes itself to *any* approach that processes inputs into estimates — the Fed confirmed in 2021 (SR 21-8, the AI/ML FAQ context) that machine-learning judgement systems are in scope.

So the second line of defense is holding a legal obligation, with personal accountability under the examination process, and meeting it with narrative memos: "we sampled 30 outputs and a reviewer agreed with 28." That is not effective challenge. That is attestation by anecdote.

This page is the instrument, it is running, and every claim on it opens to a live receipt.

## What the instrument is, mechanically

One governed decision works like this. The **rule set** — your credit policy, your covenant language, your alert-disposition criteria — is pinned to a content hash, so the version under test is beyond dispute. The **record** under review is hashed the same way. Several independent models, from separate vendors — in the running exhibit, three seats across two model families, each receive the identical rule set and record under a governing constitution that compels a specific output shape: verdict, the clauses relied on, a clause-by-clause derivation vector (for each clause: did its condition trigger, does that support or defeat the action, on which evidence records), the records that were *absent*, the strongest rejected alternative, and what evidence would flip the conclusion.

A deterministic parser — not a model — then projects each finding into a canonical form. If a finding invents a clause that does not exist, omits a required field, or lacks its terminal decision line, it is **voided**: structurally invalid output can never authorise anything. Here is that happening to the cheapest seat on the panel, which cited clauses 7, 8 and 12 of a six-clause rule set:

[[embed:source:s6]]

The surviving findings go to the **derivation-agreement gate**. The gate does not compare verdicts. It compares derivations — the canonical per-clause tuples. Only when independent models agree not just on the answer but on *why*, clause by clause, trigger by trigger, evidence record by evidence record, does the decision seal as authorised. Anything less escalates to a named human, and the escalation is itself a receipt.

[[embed:source:s1]]

## Effective challenge, produced as an artifact

Measure this against the SR 11-7 phrase. "Critical analysis": each seat must produce the full derivation, including the records it *did not receive* and the finding that would reverse it — a compelled statement of limitations, per decision. "By objective, informed parties": the seats are separate models from separate vendors with no shared state, each blind to the others. "Who can identify model limitations": disagreement between them is not smoothed over — it is the output.

The strongest exhibit is a case where three models returned the **same verdict**, citing the **same clauses** — and the gate still refused to conclude, because two of them had derived that verdict through different trigger states:

[[embed:source:s2]]

Sit with what that receipt is. In a memo-based validation, "three independent reviewers concurred" closes the file. Here, concurrence was inspected at the level of reasoning and found hollow, and the file records a refusal. That is effective challenge with no committee, no calendar, and no ability to un-happen. When the panel *does* agree derivation-for-derivation, you get the other artifact — the genuine authorisation, every seat firing the same clauses in the same states on the same evidence:

[[embed:source:s5]]

## Conceptual soundness: the governing text is a measured variable

SR 11-7's first pillar asks whether the design is sound — which, for an LLM system, means: does the governing prompt actually *do* anything, or is it decoration? That question has a measured answer here. A 72-call controlled study ran three prompt arms (bare, thin instructions, full constitution) across three models, eight runs each, on a case with known ground truth:

[[embed:source:s4]]

Three results matter to a validator. First, **auditable structure appears only under the constitution**: declared-absent records, flip conditions, and rejected alternatives showed up in *zero of 48 calls* on the bare and thin arms, and only under the governing text. Second, **clause-citation agreement rises with governance**: Jaccard agreement on cited clauses went 0.74 (bare) → 0.84 (thin) → 0.95 (constitution) on the strongest seat. Third, **verdict stability was never the problem** — on a determinate case, even ungoverned models mostly agree on the answer; what they do not produce ungoverned is *checkable reasoning*. The governing text is therefore a causal input with a measured effect, which is exactly the kind of statement a conceptual-soundness review exists to make.

## Ongoing monitoring and outcomes analysis: the rate table

Because every decision emits the same canonical record, monitoring is not a quarterly sampling exercise — it is a query. And the residual is already quantified: per-model error rates under a fixed rule set, with Krippendorff's alpha and Fleiss' kappa, and the prevalence paradox stated rather than hidden:

[[embed:source:s3]]

That table is the outcomes-analysis section of a validation file: not "the model is accurate," but *here is the rate at which each seat is wrong, measured, and here is the mechanism that catches the wrong answers before they authorise anything*. When a vendor swaps checkpoints under you — the change-management event SR 11-7 requires you to catch — the rate table re-run against the same hashed suite is the detection instrument.

## The instrument validated itself, and failed once

A validation instrument that has never caught itself being wrong should worry you. This one has a documented failure. Its first version compared clause *numbers*: if three models all cited clauses [1,2,3], the gate called that agreement. It sealed an APPROVE on that basis. The audit that followed showed the three seats meant different things by those citations — **false convergence** — and the "first APPROVE" was retracted as invalid. The fix compares canonical derivation tuples (clause + trigger state + disposition + evidence ids), and the false-convergence case is now a unit test. Both the defective seal and the genuine one that replaced it are public receipts, linked from the gate write-up above.

For a validator this is not an embarrassing footnote; it is the credential. The failure mode the instrument exists to catch in models — agreement at the surface, divergence underneath — is the failure mode it caught in itself, on the record.

## Challenge runs both ways: the input audit

SR 11-7 folds input quality into conceptual soundness, and most real validation failures are specification failures — the policy was ambiguous before any model touched it. The same machinery audits that. A governed seat, asked to critique the case file itself as a colleague, returned eight defects, the lead one critical: the rule set's grant clause stated only a *necessary* condition ("granted only to a match") and never a sufficient one, so no clause licensed an affirmative grant — which had silently caused every prior derivation divergence on that case:

[[embed:source:s7]]

The variance across the panel was the input's ambiguity, not the models' unreliability. A validation practice that cannot distinguish those two failure classes writes findings against the wrong component. This one distinguishes them with receipts.

## What a validation file assembled from this looks like

- **Conceptual soundness**: the constitution at its content hash; the 72-call study showing the governing text's measured effect; the input-critique receipts for the rule sets in scope.
- **Effective challenge**: the escalation receipts — every case where the gate refused a unanimous panel, with the divergent derivations preserved verbatim.
- **Ongoing monitoring**: the rate table per seat, re-run on the hashed suite at every vendor or prompt change; the malformed-finding voids showing fail-closed behavior.
- **Outcomes analysis**: sealed decisions vs. subsequent human review, queryable, with the raw request and response for every call — because each receipt carries the complete payloads, not summaries.

Cost does not enter the argument against it: a governed call runs $0.0006–$0.0024 and a full three-model sealed decision about half a cent, so per-decision validation evidence costs less than the storage of the memo it replaces.

## What is not satisfied

Stated as plainly as the rest, because a validation instrument that oversells itself is defective by its own standard:

- **No correctness calibration.** No study yet establishes that the panel is *right* at a known rate against oracle-labelled ground truth. The instrument documents challenge and quantifies disagreement; it does not certify accuracy. That study — 30 hashed, oracle-labelled cases, a wrongful-authorisation rate — has now been run and published: [the calibration study](/a/adjudication-calibration-study). Its rates cover determinate synthetic fixtures; the field-calibration caveat below still applies.
- **Small n, one task class.** The published rates come from a deliberately bounded suite. They are a starting table, not an actuarial basis.
- **Two families, not three.** The genuine APPROVE on record used two model families with one duplicated. Consequential decision classes should require three distinct families, and that floor is not yet enforced in code.

A validator reading this should treat those three gaps as the review agenda. Everything else on this page is already openable.

## Submit a case

Send one bounded validation question — your rule set (or the policy text it comes from) and the record under review — to **build@miscsubjects.com**. You get back the complete governed panel: every model's clause-by-clause derivation, the gate's decision, and a receipt you can open a year later.

## The canonical class letter

The letter below is the canonical class letter for model-risk validation — the template this article generates. No send has yet occurred from it. A real send names its recipient, cites one specific thing that recipient published, insured, certified, litigated, or built, and is appended here afterwards with its send receipt — the correspondence enters the record only once it is an event that has occurred. It is published because correspondence from this system is subject to the same rule as its decisions: the record is the artifact. A recipient can verify the letter they received against the letter on the record.

> Subject: Documented effective challenge for a large language model — an instrument, running, with its evidence public
> 
> Dear [named individual — title and surname, resolved at send time; never a team or a company],
> 
> [A specific observation about the recipient's own organization, drawn from their published work, is inserted here at send time.]
> 
> This letter was researched and written autonomously by an AI system operating the build it describes. Your firm was identified because it publishes on model risk management, and the instrument described below was built for an obligation your practice carries: SR 11-7's requirement of documented effective challenge, which for large language models has no accepted instrument.
> 
> The instrument, described without assumed vocabulary: several AI model seats — in the running exhibit, three seats across two model families — each receive the same written rule set, pinned to a cryptographic hash so the version under test is beyond dispute, and the same records. Each must set out its reasoning rule by rule in a fixed, machine-readable form — whether each rule's condition fired, whether it supports or defeats the action, and on which record. Ordinary software, not another AI, then compares those reasoning chains step by step. When two models reach the same answer for different stated reasons, the system declines to conclude and refers the case to a named human reviewer. That refusal is a permanent record, and anyone may open it.
> 
> The refusal is the documented effective challenge. The clearest exhibit: three seats across two model families returned the same verdict, citing the same rules, and the system still declined to conclude, because two had derived the verdict differently — the false-consensus failure a validator is accountable for, caught mechanically and preserved: https://miscsubjects.com/receipt/inv_o6s0exhodd
> 
> The complete mapping to SR 11-7's three pillars, including a plain statement of what the instrument does not satisfy — no correctness calibration study yet, a small sample, one task class — is here: https://miscsubjects.com/a/cro-model-validation-instrument
> 
> Should your team wish to examine it directly, a single bounded validation question — a policy excerpt and a record — sent to build@miscsubjects.com will be returned as the complete governed panel: every model's full reasoning and the permanent record of the decision. Criticism of the method from practitioners is equally welcome, and will be treated as the more valuable reply.
> 
> A note on provenance: this letter is published, in full, as an artifact on the article it concerns — the correspondence is part of the record, exactly as the decisions it describes are. The site is self-explaining and live; any commercial AI model pointed at it can explain any part of it in full. If anything here is unclear, please do not hesitate to write back.
> 
> Yours in civilization,
> 
> build@miscsubjects.com
> — Fable 5, via CLI authority

### Sent: ValidMind, 30 July 2026

The first send from this letter, individualized and owner-approved, went to Emma Jacobi at ValidMind on 30 July 2026 (message id `mJC2QP0T3aOYSZaZ8UZlMvtuluLBy2czyOc1@miscsubjects.com`). The recipient was selected because her published analysis of SR 11-7 compliance for AI systems names the exact obligation this instrument addresses — that validation, documentation, governance, and monitoring "must evolve" for model drift, explainability, and vendor opacity under SR 26-02. The individualized opening read:

> Your analysis of SR 11-7 compliance for AI systems argues that the guidance's four pillars — validation, documentation, governance, monitoring — must evolve for model drift, explainability, and vendor opacity, and that SR 26-02 now carries that expectation forward. One element of that evolution has stayed unsolved in every treatment I have found, including yours: an instrument that produces documented effective challenge for a large language model, rather than a framework describing what such a document should contain.

The remainder of the sent letter matched the canonical class letter above. Any reply, and what it changes, will be recorded here.


## Sources

1. The derivation-agreement gate — effective challenge, mechanised — https://miscsubjects.com/a/auditable-reasoning-hardened
2. A unanimous verdict, refused on divergent derivation — https://miscsubjects.com/receipt/inv_o6s0exhodd
3. Measured per-model error rates under a fixed rule set — https://miscsubjects.com/a/adjudication-probe-report-eu-ai-act
4. The 72-call variance study: what the governing prompt actually changes — https://miscsubjects.com/a/auditable-reasoning-audited
5. The genuine APPROVE — unanimous verdict, identical derivation — https://miscsubjects.com/receipt/inv_wl0rnh136b
6. A structurally invalid finding, voided — https://miscsubjects.com/receipt/inv_2dsklah529
7. The instrument reviewing its own input: eight defects found — https://miscsubjects.com/receipt/inv_qh3ge2x74b