# Cloudflare OS: the ledger as a table

slug: cloudflare-os-xl-02-ledger-as-a-table · https://miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table · category: systems · tags: cloudflare, pipelines, r2, ledger, analytics · updated 2026-08-06T03:28:33.400Z

*Part 2 of [Cloudflare OS XL](/a/cloudflare-os-xl), an inventory of the Cloudflare platform this build does not have installed.*

The central claim of this build is that nothing is ever overwritten and every action appends a hash-chained audit row. That claim is true. The rows exist, in a D1 database called `loop-shared-events` and in R2 as receipt files.

Then someone asks a question of it — how many outbound sends went to a domain whose MX record failed, per week, since May — and the answer is produced by pulling files and counting them in a script. The ledger is a record. It is not yet a table anybody can query.

Five products close the distance between those two things.

## Pipelines

Pipelines is Cloudflare's streaming ingest: data arrives over HTTP or from a Worker binding, is transformed with SQL, and is delivered to R2 as Apache Iceberg tables or as Parquet and JSON files. It is in open beta.

Today, every ledger append is a D1 `INSERT` executed inside the request that caused it. That has three costs. It puts a write in the hot path of the thing being recorded. It makes the ledger's throughput a function of D1's write throughput. And it produces rows, not columns — which is why analytical questions are answered by export-and-count.

With a pipeline, the Worker writes an event to a binding and returns. The pipeline batches, transforms and lands it in R2 in a columnar format. The record is still append-only and still hash-chained; it is simply stored as something a query engine can read.

The natural first candidates here are the three highest-volume event streams: agent turns, tool invocations, and outbound send receipts.

**Verdict: install, for agent turns first.** It is beta, so it belongs on the stream where a gap would be survivable, not on the audit chain.

## R2 Data Catalog and R2 SQL

R2 Data Catalog is a managed Apache Iceberg catalog built into an R2 bucket. R2 SQL is a distributed SQL engine that queries it. Together they are the reason the previous section says "Iceberg" rather than "Parquet files in a bucket": Iceberg gives the pile of files a schema, a snapshot history and a table identity, and R2 SQL means you do not have to bring your own engine to read it.

Enabling the catalog on an existing bucket is one command.

```
wrangler r2 bucket catalog enable miscsubjects-ledger
```

What this changes for this build is the nature of an audit. The audit chain is the build's trust mechanism; it is what makes the claim "nothing is ever overwritten" checkable rather than asserted. But a trust mechanism that can only be verified by a bespoke script is verified by whoever wrote the script. A ledger as an Iceberg table can be queried by anyone with the credential, including a model, including an outside auditor, with a plain `SELECT`.

There is a second, quieter benefit. Time-travel is a property of Iceberg, not something this build would have to implement: the table can be read as of a snapshot. "What did the ledger say on 3 August" stops being a question about backups.

**Verdict: install.** Low cost, and it converts an existing asset into a queryable one without moving it out of R2.

## R2 event notifications

An object lands in R2 and a message appears on a queue. That is the whole feature, and it is missing from a build that has three queues already.

Right now, assets get processed because a cron woke up and looked. Generated hero images, ArcAds output, absorbed repositories, uploaded references — each of those arrives in a bucket and then waits for a scheduled sweep to notice. The sweep runs every minute, which is fast enough to feel instant and is still the wrong mechanism: it polls whether or not anything happened, and it cannot tell you *why* it processed something.

```
wrangler r2 bucket notification create miscsubjects-store --event-type object-create --queue loop-tasks
```

With that, the arrival of the object *is* the trigger. The queue message carries the bucket, the key and the event type, so the consumer knows exactly what changed rather than diffing a listing.

**Verdict: install.** It is one command per bucket and it deletes polling code.

## Analytics Engine

Analytics Engine accepts unlimited-cardinality analytics written from a Worker and queried with SQL. Writes are non-blocking and effectively free; you get one dataset binding and you write data points with blobs, doubles and an index.

```toml
[[analytics_engine_datasets]]
binding = "METRICS"
dataset = "loop_metrics"
```

```js
env.METRICS.writeDataPoint({
  blobs: [toolName, modelId, agentName, outcome],
  doubles: [latencyMs, tokensIn, tokensOut, costUsd],
  indexes: [agentName],
});
```

This build already tries to answer cost and latency questions — there is a `COST_REPORT` row, a governor, and per-model accounting. Those work by reading the ledger back and aggregating it, which means the cost of asking a cost question scales with the size of the ledger.

Analytics Engine is the correct tool for that specific class of question because it is designed for high-cardinality dimensions. Per-tool, per-model, per-agent, per-outcome, forever, at a write cost that does not compete with the request. The ledger keeps being the record of what happened; Analytics Engine becomes the record of how much it cost and how long it took.

The one constraint worth knowing before adopting it: it is a metrics store, not an event store. Data points are sampled at high volume and are not the audit trail. Do not put anything in it that has to be exact.

**Verdict: install.** It answers the cost question this build keeps asking, and it does not compete with the ledger for that role.

## What this part does not recommend

**Do not move the audit chain off D1.** The hash chain's value is that each row commits to the previous one at write time, inside a transaction, in the same request that performed the act. Streaming it through a batching pipeline first would put a gap between the act and the commitment, and the gap is exactly what the chain exists to close. Pipelines belongs on the high-volume observational streams. The chain stays where it is.

## Verdicts

| Product | What it replaces here | Verdict |
| --- | --- | --- |
| Pipelines | Per-row D1 inserts in the hot path for high-volume streams | **install** — agent turns first |
| R2 Data Catalog | A ledger auditable only by a bespoke script | **install** |
| R2 SQL | Export-and-count in a local script | **install** — with the catalog |
| R2 event notifications | A cron sweep that polls buckets every minute | **install** |
| Analytics Engine | Cost and latency questions answered by re-reading the ledger | **install** |
| Pipelines *for the audit chain* | Nothing — it would weaken it | **no** |

Next: [Part 3 — running real code](/a/cloudflare-os-xl-03-running-real-code).


## Sources

1. Cloudflare Pipelines documentation — https://developers.cloudflare.com/pipelines/
2. R2 Data Catalog documentation — https://developers.cloudflare.com/r2/data-catalog/
3. R2 SQL documentation — https://developers.cloudflare.com/r2-sql/
4. Workers Analytics Engine documentation — https://developers.cloudflare.com/analytics/analytics-engine/