# Cloudflare OS: the failure record

slug: cloudflare-os-xl-07-seeing-what-happened · https://miscsubjects.com/a/cloudflare-os-xl-07-seeing-what-happened · category: systems · tags: cloudflare, tail-workers, logpush, observability, deploys · updated 2026-08-06T03:28:36.662Z

*Part 7 of [Cloudflare OS XL](/a/cloudflare-os-xl), an inventory of the Cloudflare platform this build does not have installed.*

This build has a rule about failure, and it is the rule the whole system rests on: a failure becomes a child task naming the failure class, the layer that permitted it, and the invariant that should have prevented it. Never a sentence in a report.

The rule is correct. Its enforcement is not mechanical. Today, a Worker throws, the exception goes to observability, and the failure becomes a task row only if an agent or the owner goes and looks. The rule depends on someone noticing — which is exactly the shape of dependency this build eliminates everywhere else.

That is the subject of this part.

## Tail Workers

A Tail Worker is a Worker assigned to another Worker, invoked with that Worker's execution log after each request: the exceptions thrown, the logs written, the outcome, the timings.

```toml
tail_consumers = [{ service = "loop-failure-intake" }]
```

```js
export default {
  async tail(events, env) {
    for (const e of events) {
      if (e.outcome === 'ok' && !e.exceptions.length) continue;
      await env.DB.prepare(
        'INSERT INTO tasks (state, title, failure_class, layer, source) VALUES (?,?,?,?,?)'
      ).bind('open', e.exceptions[0]?.name ?? e.outcome, ..., 'tail').run();
    }
  }
};
```

That is the missing link, and it is small. An exception in production appends a task row automatically, carrying the script name, the stack, the request that caused it and the timestamp. The rule stops depending on attention.

Two design notes for doing it properly here rather than naively:

**Deduplicate on failure class, not on occurrence.** A broken route throwing five hundred times should produce one task with a count, not five hundred tasks. The dedup key is the exception name plus the script plus the normalised route.

**A Tail Worker cannot tail itself.** Whatever handles the intake needs its own error path, and the honest one is a dead-letter queue rather than a second tail.

**Verdict: install. This is the highest-priority item in the entire series** — not because it is the most impressive, but because it makes an existing law mechanical instead of aspirational.

## Logpush and Log Explorer

Logpush pushes logs in near real time to storage or a SIEM. Log Explorer keeps them queryable in the dashboard.

The current position is that a production failure is diagnosed by re-running the thing that failed. That works for deterministic bugs and fails completely for the interesting ones — the intermittent transport fault, the payload that truncates only above a size threshold, the request that succeeded from one caller and 403'd from another. Those are diagnosed by reading what actually happened, and there is no record to read.

This build has already lost time to precisely that class. A dispatch payload containing a pipe character truncated silently and looked like an intermittent Apps Script fault for long enough to be misdiagnosed as one. With request logs, the pattern — every truncated payload contains a `|`, no exceptions — is visible in one query.

Pointing Logpush at the existing R2 bucket costs nothing and gives every future investigation a record to work from. With the Iceberg catalog from Part 2 on the same bucket, those logs become queryable with the same SQL as everything else.

**Verdict: install.** Cheap, and it converts "reproduce it" into "look it up".

## Workers Builds and gradual deployments

Deploys here go through `scripts/ship.mjs`, which is a real gate — it checks that HEAD matches origin, that the tree is committed, that the deploy runs from the repository root, and it fails on a list of accumulated invariants. That gate is load-bearing and should not be replaced.

What is missing is on either side of it.

**Workers Builds** runs the build on Cloudflare from a git push, so the deployed artifact is traceable to a commit on the account rather than to whatever was in a working directory. This complements the ship gate rather than replacing it: the gate decides whether a deploy is allowed, Builds records what was deployed.

**Gradual deployments** put a new version in front of a percentage of traffic before all of it. For a site with one origin and an agent population that ships several times a day, a bad render reaching ten percent of requests instead of all of them is a meaningful difference.

**Version metadata** is the small companion piece: a binding that lets a response name the version that served it. When something is wrong on the live site, the first question is always which deploy did this, and today that is answered by correlating timestamps.

**Verdict: version metadata and gradual deployments — install. Workers Builds — later**, and only alongside the existing gate, never instead of it.

## What this part does not recommend

**Do not move deploy authority to a git push.** The ship gate exists because deploys from the wrong directory produced a Functions-less build and a production outage, and because concurrent agents overwrote each other's shipped work. A push-to-deploy pipeline that bypasses those checks would reintroduce both failure classes with better ergonomics. Builds is welcome as a recorder. It is not welcome as the decider.

## Verdicts

| Product | What it replaces here | Verdict |
| --- | --- | --- |
| Tail Workers | A law about failure that depends on someone noticing | **install — first** |
| Logpush | Diagnosing production failures by re-running them | **install** |
| Log Explorer | The same, from the dashboard | **install** — with Logpush |
| Version metadata | Correlating timestamps to guess which deploy broke it | **install** |
| Gradual deployments | Every bad render reaching 100% of traffic immediately | **install** |
| Workers Builds | Nothing — the ship gate stays the decider | **later** — as a recorder only |

Next: [Part 8 — reaching private things](/a/cloudflare-os-xl-08-reaching-private-things).


## Sources

1. Tail Workers documentation — https://developers.cloudflare.com/workers/observability/logs/tail-workers/
2. Cloudflare Logpush documentation — https://developers.cloudflare.com/logs/logpush/
3. Workers versions and deployments documentation — https://developers.cloudflare.com/workers/configuration/versions-and-deployments/