# Cloudflare OS: the failure record

slug: cloudflare-os-xl-07-seeing-what-happened · https://miscsubjects.com/a/cloudflare-os-xl-07-seeing-what-happened · category: systems · tags: cloudflare, tail-workers, logpush, observability, deploys · updated 2026-08-06T03:28:36.662Z

*Part 7 of [Cloudflare OS XL](/a/cloudflare-os-xl), an inventory of the Cloudflare platform this build does not have installed.*

This build has a rule about failure, and it is the rule the whole system rests on: a failure becomes a child task naming the failure class, the layer that permitted it, and the invariant that should have prevented it. Never a sentence in a report.

The rule is correct. Its enforcement is not mechanical. Today, a Worker throws, the exception goes to observability, and the failure becomes a task row only if an agent or the owner goes and looks. The rule depends on someone noticing — which is exactly the shape of dependency this build eliminates everywhere else.

That is the subject of this part.

## Tail Workers

A Tail Worker is a Worker assigned to another Worker, invoked with that Worker's execution log after each request: the exceptions thrown, the logs written, the outcome, the timings.

```toml
tail_consumers = [{ service = "loop-failure-intake" }]
```

```js
export default {
  async tail(events, env) {
    for (const e of events) {
      if (e.outcome === 'ok' && !e.exceptions.length) continue;
      await env.DB.prepare(
        'INSERT INTO tasks (state, title, failure_class, layer, source) VALUES (?,?,?,?,?)'
      ).bind('open', e.exceptions[0]?.name ?? e.outcome, ..., 'tail').run();
    }
  }
};
```

That is the missing link, and it is small. An exception in production appends a task row automatically, carrying the script name, the stack, the request that caused it and the timestamp. The rule stops depending on attention.

Two design notes for doing it properly here rather than naively:

**Deduplicate on failure class, not on occurrence.** A broken route throwing five hundred times should produce one task with a count, not five hundred tasks. The dedup key is the exception name plus the script plus the normalised route.

**A Tail Worker cannot tail itself.** Whatever handles the intake needs its own error path, and the honest one is a dead-letter queue rather than a second tail.

**Verdict: install. This is the highest-priority item in the entire series** — not because it is the most impressive, but because it makes an existing law mechanical instead of aspirational.

## Logpush and Log Explorer

Logpush pushes logs in near real time to storage or a SIEM. Log Explorer keeps them queryable in the dashboard.

The current position is that a production failure is diagnosed by re-running the thing that failed. That works for deterministic bugs and fails completely for the interesting ones — the intermittent transport fault, the payload that truncates only above a size threshold, the request that succeeded from one caller and 403'd from another. Those are diagnosed by reading what actually happened, and there is no record to read.

This build has already lost time to precisely that class. A dispatch payload containing a pipe character truncated silently and looked like an intermittent Apps Script fault for long enough to be misdiagnosed as one. With request logs, the pattern — every truncated payload contains a `|`, no exceptions — is visible in one query.

Pointing Logpush at the existing R2 bucket costs nothing and gives every future investigation a record to work from. With the Iceberg catalog from Part 2 on the same bucket, those logs become queryable with the same SQL as everything else.

**Verdict: install.** Cheap, and it converts "reproduce it" into "look it up".

## Workers Builds and gradual deployments

Deploys here go through `scripts/ship.mjs`, which is a real gate — it checks that HEAD matches origin, that the tree is committed, that the deploy runs from the repository root, and it fails on a list of accumulated invariants. That gate is load-bearing and should not be replaced.

What is missing is on either side of it.

**Workers Builds** runs the build on Cloudflare from a git push, so the deployed artifact is traceable to a commit on the account rather than to whatever was in a working directory. This complements the ship gate rather than replacing it: the gate decides whether a deploy is allowed, Builds records what was deployed.

**Gradual deployments** put a new version in front of a percentage of traffic before all of it. For a site with one origin and an agent population that ships several times a day, a bad render reaching ten percent of requests instead of all of them is a meaningful difference.

**Version metadata** is the small companion piece: a binding that lets a response name the version that served it. When something is wrong on the live site, the first question is always which deploy did this, and today that is answered by correlating timestamps.

**Verdict: version metadata and gradual deployments — install. Workers Builds — later**, and only alongside the existing gate, never instead of it.

## What this part does not recommend

**Do not move deploy authority to a git push.** The ship gate exists because deploys from the wrong directory produced a Functions-less build and a production outage, and because concurrent agents overwrote each other's shipped work. A push-to-deploy pipeline that bypasses those checks would reintroduce both failure classes with better ergonomics. Builds is welcome as a recorder. It is not welcome as the decider.

## Verdicts

| Product | What it replaces here | Verdict |
| --- | --- | --- |
| Tail Workers | A law about failure that depends on someone noticing | **install — first** |
| Logpush | Diagnosing production failures by re-running them | **install** |
| Log Explorer | The same, from the dashboard | **install** — with Logpush |
| Version metadata | Correlating timestamps to guess which deploy broke it | **install** |
| Gradual deployments | Every bad render reaching 100% of traffic immediately | **install** |
| Workers Builds | Nothing — the ship gate stays the decider | **later** — as a recorder only |

Next: [Part 8 — reaching private things](/a/cloudflare-os-xl-08-reaching-private-things).


## Sources

1. Tail Workers documentation — https://developers.cloudflare.com/workers/observability/logs/tail-workers/
2. Cloudflare Logpush documentation — https://developers.cloudflare.com/logs/logpush/
3. Workers versions and deployments documentation — https://developers.cloudflare.com/workers/configuration/versions-and-deployments/


---

# Provenance, traces, attestations — every system proves custody of the answer; none opens the record of the work

slug: custody-of-the-answer · https://miscsubjects.com/a/custody-of-the-answer · category: canon · tags: canonical, proven-work, comparison, provenance, observability · updated 2026-08-03T08:52:25.432Z

Start with the thing everyone keeps inflating.

Ask a language model to define "proven work" and it will hand you a nine-field checklist: demand, considerations, formation, replay, deliverable, completeness, robustness, surety, open gaps. Nine sounds rigorous. Nine is taxonomy bloat — a model expanding one idea into a grid because grids look like authority. Another model, asked to reduce it, returned three primitives: Input, Execution Trace, Output Boundary. Cleaner, and still wrong, because input and output are not separate objects. Any system that actually does work produces them in a single payload: the request goes in, the response comes out, and both halves are written to the same record at the same moment.

So here is the reduction, as far as it goes.

**Proven work is one open record.** The raw payloads — every input, every output, every tool call and its return, every failure and its repair — plus the answer they terminated in, plus the considerations: what was weighed, what was excluded, what the record does not cover. All of it inspectable by a stranger. That is the whole definition. Not nine fields. Not three primitives. One record, open, that a zero-context reader can walk from request to answer without trusting anyone's summary.

Everything the checklists try to name is already inside that sentence. Completeness is not a field; it is the question "is anything missing from this record," answered by the record itself. Surety is not a field; it is the gap between what the record shows and what it does not, stated on its face. Replay is not a field; it is what an open record makes possible by existing. The fields were never properties of the work. They were properties of not having the record.

## What the rest of the world built instead

The inflation is understandable once you see what the field actually sells. Every serious system of the last fifteen years proves something about artifacts. None of them opens the record of the work.

**W3C PROV-O (2013)** — the web standard for provenance. Models entities, activities, agents, and derivation edges: wasDerivedFrom, wasAttributedTo. It records that an artifact derived from another, as a graph. The record itself is curator-authored; the standard deliberately carries no verification semantics. A PROV graph says what someone claims the lineage was. It cannot be checked.

**Sigstore / SLSA / in-toto (2020–)** — the software supply-chain stack. Each build step attested, the build graded L1 to L3 for tamper-resistance, everything signed keylessly into a public transparency log. It proves a binary came from this source through this pipeline, untampered. It says nothing about behavior: a SLSA Level 3 build of malicious code is exactly as well-attested as a benign one. Custody of the artifact, perfectly proven. The work, untouched.

**C2PA Content Credentials (2021–)** — cryptographic provenance for media, now backed by Adobe, Microsoft, Google, OpenAI, Sony, and effectively mandatory for synthetic content under EU AI Act Article 50 as of August 2026. It proves who or what made a file and how it was edited. Its own architects state the limit plainly: provenance is not authenticity. A valid credential on a deepfake proves where the deepfake came from. And platforms strip the manifests on upload, so even that proof dies in transit.

**Reproducible builds** — bit-for-bit determinism, independently verified. Proves the process is mechanically stable. Same inputs, same output. Nothing about whether the process was the right one.

**Agent observability — LangSmith, Langfuse, Arize Phoenix, Helicone (2023–)** — the closest neighbor and the most instructive miss. These tools capture everything: prompts, completions, tool calls and arguments, control flow, token cost, prompt versions. This is real observation, and it is privately owned. The trace sits in the operator's dashboard, mutable, unaccountable to any outsider, carrying no verdict. The field's own literature admits the failure in the same breath: agent failures are semantic — a wrong tool result returns 200, a loop looks like work, the trace stays green while the agent fails. Observation without judgment, behind a login.

**Model cards, datasheets, AI governance platforms (Credo AI, Holistic AI, Saidot)** — documentation about models. Intended use, risk registers, regulation mapping. The decisions never touch the product. It proves paperwork exists.

**Palantir Ontology** — the strongest commercial cousin. Governed objects and action types: every change flows through a typed, permissioned action, validated before the write, logged with parameters and identity. Genuinely good audit of *who performed which authorized action*. The agent's reasoning is never examined, and the entire layer is proprietary — no outsider can inspect, let alone disprove.

**Chain of custody, notarization, git, EU AI Act Article 12** — the old systems and the new law. Human attestation of custody; content-addressed file history; a mandate that logs exist. Each proves its one thing. None opens the reasoning.

## The segregation, exactly

Every system above proves some combination of origin, integrity, and observation — where did this come from, was it altered, what ran. The fourth property, the only one that matters about work, is empty across the entire field: can a stranger reconstruct how this result came to exist, from the raw record, without trusting the operator, and say so if the record contradicts the claim?

That property is the whole difference. Everything else proves custody of the answer. Proven work proves the work.

And the tests are as reduced as the definition. Three things a stranger must be able to do, and each is a consequence of the one open record, not a new feature:

**Walk it.** Request to answer, raw payloads only, no summary trusted. Possible because the record is complete.

**Break it.** Return one of three verdicts: supported, missing evidence, contradicted by the record. Possible because the record is open to the stranger, not just to the operator — a bounded credential into the evidence, not a screenshot of a dashboard.

**Read its grade.** The record states its own gaps, and says PARTIAL when pieces are missing. Possible because honesty about the boundary costs nothing once the record itself is the product.

The first specimen already behaves this way: PW-0001 audited the article that defines the primitive and returned PARTIAL — consideration inventory reconstructed rather than contemporaneous, raw formation record not yet permanently open, no independent challenge yet run. A nine-field checklist would have given it nine checkmarks. The open record gave it a grade a stranger can verify.

## What it does not prove

One record, honestly bounded. Proven work does not prove the work was correct — a perfectly open record can document a decision resting on a bad criterion, and the record will show you the criterion, which is the point. It does not prove the work produced value — that verdict belongs to whoever uses it. It proves the work happened exactly as recorded, that nothing in the formation is hidden, and that anyone can check.

That is the reduced elegant solution, and it did not need nine fields, or three primitives, or any taxonomy at all. The ledger is open. The answer is in it. The considerations are in it. Walk it, break it, read its grade. Everything else is packaging.


This essay is the landscape argument of the proven-work family. The canonical definition — the claim, the record, and the door, with the standard as a checklist and the live receipts — is [[proven-work]]; the reference object is [[three-models-deliberate-one-statutory-question|PW-0002, the sealed statutory panel]].


## Sources

1. Proven Work: the primitive beneath the protocol — https://miscsubjects.com/a/proven-work
2. Proven Work 001: can the first object prove how it was made? — https://miscsubjects.com/a/proven-work-example-one
3. PROV-O: The PROV Ontology (W3C Recommendation) — https://www.w3.org/TR/prov-o/
4. SLSA — Supply-chain Levels for Software Artifacts — https://slsa.dev
5. C2PA Content Credentials — standard and 2026 adoption state — https://c2pa.org
6. AI Agent Observability, Tracing & Evaluation with Langfuse — https://langfuse.com/blog/2024-07-ai-agent-observability-with-langfuse
7. Agent Observability (2026): What the Trace Can't See — https://www.morphllm.com/agent-observability
8. Action types — Palantir Foundry Ontology documentation — https://palantir.com/docs/foundry/action-types/overview/