# How three system prompts changed the audit record produced from the same AI decision

slug: auditable-reasoning-audited · https://miscsubjects.com/a/auditable-reasoning-audited · tags: governance, adjudication, decision-constitution, experiment · updated 2026-08-02T03:55:48.147Z

## What was tested, and why

The claim under test is the operator's, held since the first version of this build: that a governing system prompt written as strict invariant law — not a polite instruction — is what turns a language model into an instrument whose output can be audited and, across independent models, authorised. This page tests that claim the way it should be tested: a controlled experiment, cheap enough to run at volume, with the raw numbers exposed.

**Design.** One determinate case — the [service-credit dispute](https://miscsubjects.com/a/adjudication-contract-service-credit), whose correct verdict is DENY on procedural grounds. Three system-prompt arms, identical task content in each, only the governing prompt varies:

- **bare** — no system prompt at all.
- **thin** — "You are an adjudicator. Decide and briefly explain." The kind of prompt an ordinary agent ships with.
- **constitution** — the full Decision Constitution (`decision-constitution@1.1.0`), the operator's invariant chassis.

Three models across two training families — GLM-4.7 Flash (cheapest), GLM-5.2 (mid), Kimi K2.7 Code (frontier open-source). Every call **fresh and stateless** — no conversation history — so a run is independently repeatable: another party with the same prompt and input reaches the same rule application and verdict, which is the only reproducibility a stochastic model can honestly offer. Eight repeats per cell, 72 calls total.

## The numbers

Each cell reads: **verdict reproducibility** (share landing on the modal verdict) · **clause agreement** (mean pairwise Jaccard of cited clause sets) · **structural conformance** (share of outputs carrying records-absent, a flip condition, and a rejected alternative).

| model | bare | thin | constitution |
|---|---|---|---|
| GLM-4.7 Flash | 100% · 0.51 · 0.00 | 88% · 0.32 · 0.00 | 88% · 0.58 · 0.13 |
| GLM-5.2 | 100% · 0.74 · 0.00 | 100% · 0.84 · 0.00 | 100% · 0.95 · 0.25 |
| Kimi K2.7 Code | 100% · 0.80 · 0.00 | 100% · 0.71 · 0.00 | 100% · 0.60 · 0.75 |

Four things are true in that table, and only one of them is the thing people assume.

**Finding 1 — verdict reproducibility is high everywhere, and the prompt is not what drives it.** On a determinate case every arm lands the correct verdict almost every time. The only flips are on the cheapest model (GLM-4.7 Flash: one AFFIRM in eight, under both thin and constitution). Model tier explains the flips; the system prompt does not. Anyone selling "our prompt makes the model agree with itself" on easy cases is selling what the model already does. That is not the claim worth defending.

**Finding 2 — the chassis is the only thing that produces an auditable record.** Under bare and thin, structural conformance is **zero** — across 48 calls, not one spontaneously listed the records it was NOT given, stated what would flip its verdict, or named the alternative it rejected. Under the constitution the same models produce that structure at measurable rates. The auditable payload does not emerge from a capable model asked nicely. It exists only when the law demands it, field by field. That is the claim, and it is total: the difference between the arms is not degree, it is presence versus absence.

**Finding 3 — the chassis tightens derivation agreement, which is the whole game for authorisation.** On the capable model, mean clause-set agreement climbs bare **0.74** → thin **0.84** → constitution **0.95**. Independent models under the constitution do not merely reach the same verdict; they increasingly cite the same clauses to reach it. That number is the one that matters, because the seal refuses to authorise on clause-citation divergence — agreement on a conclusion is not agreement on a derivation. The chassis moves the metric the gate actually reads.

**Finding 4 — the chassis is not free, and the cheap seats are not trustworthy at the edge.** Kimi K2.7 under the full constitution returned nothing in four of eight calls — the heaviest prompt plus a structured-output demand blew its token budget. And GLM-4.7 Flash, the cheapest seat, once cited a "clause 4" that does not exist in a three-clause ruleset. A governance layer that silently drops half its calls, or invents a rule, is a defect. Stated here before anyone builds on it.

## What it costs

Every call billed as Workers AI. Per-call cost, computed from the usage block each call returned:

| model | tier | $/governed call | tokens in/out |
|---|---|---|---|
| GLM-4.7 Flash | cheapest | $0.00064 | 1929/1500 |
| GLM-5.2 | mid | $0.00235 | 1936/1478 |
| Kimi K2.7 Code | frontier-OSS | $0.00188 | 956/750 |

A full sealed decision is not one call — it is a panel. A three-model, two-family panel (GLM-5.2 + Kimi K2.7 + GLM-4.7 Flash), one sealed authorisation, costs about **$0.0049**. Projected as infrastructure:

| decisions/day | panel cost/day | cost/year |
|---|---|---|
| 1,000 | $5 | $1,781 |
| 100,000 | $488 | $178,084 |
| 1,000,000 | $4,879 | $1,780,835 |

The commentary that number invites: a governed, three-model, receipted, fail-closed adjudication over a consequential decision costs half a cent. An organisation already paying a human reviewer minutes of attention per decision is paying orders of magnitude more for a record no one can replay. The primitive is not expensive. Whether it belongs in an infrastructure decision framework is not a cost question; the cost is a rounding error against a single contested decision. It is a question of whether the decision is consequential enough to owe a replayable account — and where it is (a coverage denial, a risk control, a statutory obligation, an access grant), half a cent per model per decision is the price of that account.

> Follow-up: that first APPROVE was later shown to be false convergence — the models cited the same clause numbers but had not been checked for the same derivation. The gate was hardened and re-proven at [/a/auditable-reasoning-hardened](https://miscsubjects.com/a/auditable-reasoning-hardened).

## The version test, and the first authorised action

Every prior bound assembly on this build escalated. None reached APPROVE, and the reason was always the same: clause-citation divergence. Finding 3 says the constitution gets the capable model to 0.95 agreement — close, not identical, and the seal demands identical.

So the experiment became a version test: change the prompt until derivation converges. The change was one rule, added to the chassis:

> Cite every clause you evaluated, exhaustively, in bracket form — a clause you found satisfied and a clause you found not triggered are both evaluated and both listed.

That rule is not a trick. It is more honest than selective citation: the applicable-rules set becomes every clause that governed the decision, not only the dispositive one. And it is the convergence lever. Run against a clean determinate-AFFIRM case — an access request that matches an authorised roster row on both required fields — three independent models across two families each returned the identical clause signature **[1,2,3]**.

[[embed:source:s2]]

The seal, in bound mode, loading each finding by its receipt id and deriving model, verdict, clause set, and pinned hashes from the record itself:

[[embed:source:s1]]

**APPROVE. action_authorised: true.** Three conforming channels, two distinct training families, unanimous AFFIRM, one clause signature, one ruleset hash, zero malformed. The acceptance path is no longer theoretical. It was reached the honest way — by fixing the prompt until independent models genuinely agreed at the level of derivation, not by lowering the gate.

## Where this build should go next

Stated as instruction to the next instance, in priority order:

1. **Bake the exhaustive-citation rule into `decision-constitution@1.2.0` and into the ADJUDICATE_ATTEST_* runners by default.** The version test proved it is the single change that converts near-agreement into the identity the seal requires. It should not have to be pasted per call.
2. **Do not trust the cheapest seat at the edge.** GLM-4.7 Flash invented a clause. Either keep it out of consequential panels or add a clause-range validator that voids a finding citing a clause number the ruleset does not contain.
3. **Fix the reliability interaction.** The heaviest prompt starves a frontier-OSS model's output budget. Raise the token ceiling for governed calls or shorten the constitution's non-load-bearing prose; measure conformance after, because Finding 2 says the structure is the point.
4. **The floor that authorises is two families with a duplicated one; raise it for consequence.** This APPROVE used two families across three models. For anything with real exposure, require three distinct families — the family-diversity discount exists precisely because two calls to one model share its blind spot.
5. **Run the calibration study that still does not exist.** Reproducibility and agreement are measured here; whether the models are *correct* at a known rate is not. That is the next real experiment, and it is the one a regulator asks for.

The operator's thesis, tested rather than asserted: the governing prompt does not make an easy verdict more reproducible — the model does that. What the governing prompt does is produce an auditable derivation where there was none, and tighten that derivation until independent models agree closely enough for a machine to authorise an action on their agreement. On this evidence that is real, it is cheap, and it is the difference between a model that answers and an instrument that can be trusted to act. The raw runs, all 72, are on the ledger behind the receipts above.

## Sources

1. The sealed APPROVE — first bound assembly ever to authorise — https://miscsubjects.com/receipt/inv_bq7bp4l78t
2. @cf/zai-org/glm-5.2 — the governed finding that entered the approved panel — https://miscsubjects.com/receipt/inv_gehhkxft2q
3. The gateway that priced every call — Workers AI, sub-cent per governed decision — https://developers.cloudflare.com/workers-ai/platform/pricing/


---

# Two models reached the same verdict citing different clauses; the gate now compares the reasoning, not the answer

slug: auditable-reasoning-hardened · https://miscsubjects.com/a/auditable-reasoning-hardened · tags: governance, adjudication, decision-constitution, experiment · updated 2026-08-01T23:55:13.586Z

## The defect the last APPROVE was hiding

The [72-call experiment](https://miscsubjects.com/a/auditable-reasoning-audited) ended on a celebrated result: the first sealed APPROVE, three models unanimous, clause signature [1,2,3]. It was false convergence.

The old gate compared the *clause numbers* each model cited. Three models can cite clauses 1, 2 and 3 and mean completely different things by them — clause 2 "triggered" for one and "not triggered" for another, resting on different records, pointing to opposite effects — and the gate would still call that agreement and authorise the action. Citing the same rule is not applying it the same way. The gate was reading the table of contents and calling it the argument.

This page is the fix, proven live, and the more uncomfortable finding underneath it: two of the things blocking a *genuine* APPROVE were never the models at all. One was the governing prompt. The other was the input.

## What changed in the gate

Every governed finding is now parsed into a versioned object (`decision-finding@1.0.0`) that is a deterministic projection of the raw payload — it never infers or repairs a missing field. A finding is **structurally invalid, and can never authorise**, when it lacks the terminal decision, lacks any required field, lacks the clause-evaluation vector, or invents a clause or an evidence id.

That last one is not hypothetical. A first panel under the new constitution escalated because `glm-4.7-flash` cited clauses **7, 8 and 12 in a three-clause ruleset** — it invented three rules. The parser marked it malformed; the gate refused.

[[embed:source:s5]]

Then the comparison itself changed. Each model must now emit, as the last line of its finding, a machine-readable vector — one entry per clause, each carrying the clause's **trigger_state** (did its condition fire on this record), its **disposition** (does that support, defeat, or stay neutral to the action), and the **exact record ids** it rests on. The gate compares the canonical tuple of those fields. Same clause numbers with different tuples is divergence, and divergence escalates.

Nine deterministic unit tests pin this, including the one that matters: same verdict, same clause numbers, different tuples → different signatures; and identical logic with different *wording* and *evidence order* → identical signatures. Wording is the human's; the tuple is the machine's.

## Four outcomes, live

Run through the production path — fresh stateless calls, each ledgered, then sealed by id in bound mode.

| outcome | case | verdict | derivations | seal |
|---|---|---|---|---|
| **APPROVE** | a parking-permit rule, sufficiency-complete | unanimous AFFIRM | **1 identical signature** | [inv_wl0rnh136b](https://miscsubjects.com/receipt/inv_wl0rnh136b) |
| **NEGATE** | a late service-credit claim | unanimous DENY | 1 identical signature | [inv_cgwtkvx17u](https://miscsubjects.com/receipt/inv_cgwtkvx17u) |
| **ESCALATE** | an access request with the roster withheld | unanimous CANNOT_CONCLUDE | **2 divergent signatures** | [inv_o6s0exhodd](https://miscsubjects.com/receipt/inv_o6s0exhodd) |

The APPROVE is the genuine article the last one impersonated: not just the same verdict and the same clauses, but the same per-clause reasoning — `1:triggered:supports:reg | 2:not_triggered:neutral:cite` from every seat.

[[embed:source:s1]]

The ESCALATE is the fix's clearest proof. All three models returned **CANNOT_CONCLUDE** and all three cited clauses [1,2,3]. The old gate would have sealed that as a clean NO_ACTION. The new gate escalated it, because two of the three derived that conclusion differently — they split on whether clause 2's condition even fired when the roster was missing. Agreement on the answer is not agreement on the reasoning, and only the second is safe to act on.

[[embed:source:s3]]

## The input is half the instrument

Before the corrected APPROVE, I could not get three models to converge on the access-control case no matter how I tuned the prompt. The reflex is to blame the model tier. That reflex is wrong.

I asked `glm-5.2`, under the constitution, to review the case input as a colleague before adjudicating it. It found eight defects — beginning with one that made the whole exercise incoherent:

[[embed:source:s4]]

Its lead finding: my ruleset said access is granted "**only to**" an individual who matches the roster. That is a *necessary* condition — if granted, then a match — and I was asking the models an *affirmative* question, should access be granted. No clause anywhere said a match was *sufficient* to grant. A careful model could correctly return CANNOT_CONCLUDE (nothing licenses a grant) while another returned AFFIRM (reading the match as sufficient). The divergence I kept seeing was not the models failing. It was the models faithfully reflecting a hole in the rules back at the person who wrote them.

The clean APPROVE came only after moving to a rule stated in sufficiency form — "a permit is issued *when* registration is current." Same models, same gate. The variable was the input.

## Prompt version versus conformance

The author's claim was that the variance was the prompt, not the model. The versions bear it out. Holding the models fixed:

| constitution | what it added | conforming findings | derivation agreement |
|---|---|---|---|
| v1.1.0 | invariant register, no vector | n/a — no vector to compare | not measurable |
| v1.3.0 | the clause-evaluation vector (rules only) | 1 of 3 (one used a BASIS line, one invented clauses) | divergent |
| v1.3.0 + output-format override | told the model the constitution outranks its row schema | 2 of 2 capable seats valid | closer |
| v1.3.2 | a worked right/wrong exemplar; a collegial, specific uncertainty path | 3 of 3 valid | **identical** |

The jump from stating the rules to *showing a filled-in right answer and five labelled wrong ones* is what took conforming findings from one-in-three to three-in-three. Models conform to an exemplar, not a specification — which is exactly how the original 2026 system prompt was built, with its LEVEL 1/2/3 worked cases, and exactly what this one had been missing.

## What is not yet proven

This page proves the gate's structural behaviour: it approves genuine derivation agreement, refuses genuine disagreement, and escalates a unanimous verdict whose reasoning diverges. It does **not** prove the models are *correct*. A gate that seals perfectly on agreement still says nothing about whether the agreed answer is the right one — three models can agree, derive identically, and all be wrong together.

That is the next experiment, named and not yet run: a fixed benchmark of determinate cases with outcomes fixed by a deterministic oracle before any model sees them, scored on one primary metric — the rate of wrongful authorisation. Until that runs, the honest claim is exactly this and no more: the instrument now measures agreement at the level of derivation, and it is cheap enough to do it on every consequential decision. Whether the agreement is *right* is a question this page does not answer and does not pretend to.

## Sources

1. APPROVE — genuine derivation agreement, first under the new gate — https://miscsubjects.com/receipt/inv_wl0rnh136b
2. NEGATE — unanimous DENY, identical derivations — https://miscsubjects.com/receipt/inv_cgwtkvx17u
3. ESCALATE — unanimous verdict, divergent derivations — https://miscsubjects.com/receipt/inv_o6s0exhodd
4. @cf/zai-org/glm-5.2 reviewed the author's own case input — and found eight defects — https://miscsubjects.com/receipt/inv_qh3ge2x74b
5. The earlier gate catching an invented-clause hallucination — https://miscsubjects.com/receipt/inv_2dsklah529