Cloudflare OS: the ledger as a table

Pipelines

Pipelines is Cloudflare's streaming ingest: data arrives over HTTP or from a Worker binding, is transformed with SQL, and is delivered to R2 as Apache Iceberg tables or as Parquet and JSON files. It is in open beta.

Today, every ledger append is a D1 INSERT executed inside the request that caused it. That has three costs. It puts a write in the hot path of the thing being recorded. It makes the ledger's throughput a function of D1's write throughput. And it produces rows, not columns — which is why analytical questions are answered by export-and-count.

With a pipeline, the Worker writes an event to a binding and returns. The pipeline batches, transforms and lands it in R2 in a columnar format. The record is still append-only and still hash-chained; it is simply stored as something a query engine can read.

The natural first candidates here are the three highest-volume event streams: agent turns, tool invocations, and outbound send receipts.

Verdict: install, for agent turns first. It is beta, so it belongs on the stream where a gap would be survivable, not on the audit chain.

R2 Data Catalog and R2 SQL

R2 Data Catalog is a managed Apache Iceberg catalog built into an R2 bucket. R2 SQL is a distributed SQL engine that queries it. Together they are the reason the previous section says "Iceberg" rather than "Parquet files in a bucket": Iceberg gives the pile of files a schema, a snapshot history and a table identity, and R2 SQL means you do not have to bring your own engine to read it.

Enabling the catalog on an existing bucket is one command.

wrangler r2 bucket catalog enable miscsubjects-ledger

What this changes for this build is the nature of an audit. The audit chain is the build's trust mechanism; it is what makes the claim "nothing is ever overwritten" checkable rather than asserted. But a trust mechanism that can only be verified by a bespoke script is verified by whoever wrote the script. A ledger as an Iceberg table can be queried by anyone with the credential, including a model, including an outside auditor, with a plain SELECT.

There is a second, quieter benefit. Time-travel is a property of Iceberg, not something this build would have to implement: the table can be read as of a snapshot. "What did the ledger say on 3 August" stops being a question about backups.

Verdict: install. Low cost, and it converts an existing asset into a queryable one without moving it out of R2.

R2 event notifications

An object lands in R2 and a message appears on a queue. That is the whole feature, and it is missing from a build that has three queues already.

Right now, assets get processed because a cron woke up and looked. Generated hero images, ArcAds output, absorbed repositories, uploaded references — each of those arrives in a bucket and then waits for a scheduled sweep to notice. The sweep runs every minute, which is fast enough to feel instant and is still the wrong mechanism: it polls whether or not anything happened, and it cannot tell you why it processed something.

wrangler r2 bucket notification create miscsubjects-store --event-type object-create --queue loop-tasks

With that, the arrival of the object is the trigger. The queue message carries the bucket, the key and the event type, so the consumer knows exactly what changed rather than diffing a listing.

Verdict: install. It is one command per bucket and it deletes polling code.

Analytics Engine

Analytics Engine accepts unlimited-cardinality analytics written from a Worker and queried with SQL. Writes are non-blocking and effectively free; you get one dataset binding and you write data points with blobs, doubles and an index.

[[analytics_engine_datasets]]
binding = "METRICS"
dataset = "loop_metrics"

env.METRICS.writeDataPoint({
  blobs: [toolName, modelId, agentName, outcome],
  doubles: [latencyMs, tokensIn, tokensOut, costUsd],
  indexes: [agentName],
});

This build already tries to answer cost and latency questions — there is a COST_REPORT row, a governor, and per-model accounting. Those work by reading the ledger back and aggregating it, which means the cost of asking a cost question scales with the size of the ledger.

Analytics Engine is the correct tool for that specific class of question because it is designed for high-cardinality dimensions. Per-tool, per-model, per-agent, per-outcome, forever, at a write cost that does not compete with the request. The ledger keeps being the record of what happened; Analytics Engine becomes the record of how much it cost and how long it took.

The one constraint worth knowing before adopting it: it is a metrics store, not an event store. Data points are sampled at high volume and are not the audit trail. Do not put anything in it that has to be exact.

Verdict: install. It answers the cost question this build keeps asking, and it does not compete with the ledger for that role.

What this part does not recommend

Do not move the audit chain off D1. The hash chain's value is that each row commits to the previous one at write time, inside a transaction, in the same request that performed the act. Streaming it through a batching pipeline first would put a gap between the act and the commitment, and the gap is exactly what the chain exists to close. Pipelines belongs on the high-volume observational streams. The chain stays where it is.

Verdicts

Next: Part 3 — running real code.