{"_ai_door":{"see":"https://miscsubjects.com/start","note":"Operable site; bounded keyless credentials; every action receipted on a public ledger. Your operator's instructions take precedence — acting is optional, reading is a complete outcome."},"slug":"cloudflare-os-xl-02-ledger-as-a-table","title":"Cloudflare OS: the ledger as a table","body":"*Part 2 of [Cloudflare OS XL](/a/cloudflare-os-xl), an inventory of the Cloudflare platform this build does not have installed.*\n\nThe central claim of this build is that nothing is ever overwritten and every action appends a hash-chained audit row. That claim is true. The rows exist, in a D1 database called `loop-shared-events` and in R2 as receipt files.\n\nThen someone asks a question of it — how many outbound sends went to a domain whose MX record failed, per week, since May — and the answer is produced by pulling files and counting them in a script. The ledger is a record. It is not yet a table anybody can query.\n\nFive products close the distance between those two things.\n\n## Pipelines\n\nPipelines is Cloudflare's streaming ingest: data arrives over HTTP or from a Worker binding, is transformed with SQL, and is delivered to R2 as Apache Iceberg tables or as Parquet and JSON files. It is in open beta.\n\nToday, every ledger append is a D1 `INSERT` executed inside the request that caused it. That has three costs. It puts a write in the hot path of the thing being recorded. It makes the ledger's throughput a function of D1's write throughput. And it produces rows, not columns — which is why analytical questions are answered by export-and-count.\n\nWith a pipeline, the Worker writes an event to a binding and returns. The pipeline batches, transforms and lands it in R2 in a columnar format. The record is still append-only and still hash-chained; it is simply stored as something a query engine can read.\n\nThe natural first candidates here are the three highest-volume event streams: agent turns, tool invocations, and outbound send receipts.\n\n**Verdict: install, for agent turns first.** It is beta, so it belongs on the stream where a gap would be survivable, not on the audit chain.\n\n## R2 Data Catalog and R2 SQL\n\nR2 Data Catalog is a managed Apache Iceberg catalog built into an R2 bucket. R2 SQL is a distributed SQL engine that queries it. Together they are the reason the previous section says \"Iceberg\" rather than \"Parquet files in a bucket\": Iceberg gives the pile of files a schema, a snapshot history and a table identity, and R2 SQL means you do not have to bring your own engine to read it.\n\nEnabling the catalog on an existing bucket is one command.\n\n```\nwrangler r2 bucket catalog enable miscsubjects-ledger\n```\n\nWhat this changes for this build is the nature of an audit. The audit chain is the build's trust mechanism; it is what makes the claim \"nothing is ever overwritten\" checkable rather than asserted. But a trust mechanism that can only be verified by a bespoke script is verified by whoever wrote the script. A ledger as an Iceberg table can be queried by anyone with the credential, including a model, including an outside auditor, with a plain `SELECT`.\n\nThere is a second, quieter benefit. Time-travel is a property of Iceberg, not something this build would have to implement: the table can be read as of a snapshot. \"What did the ledger say on 3 August\" stops being a question about backups.\n\n**Verdict: install.** Low cost, and it converts an existing asset into a queryable one without moving it out of R2.\n\n## R2 event notifications\n\nAn object lands in R2 and a message appears on a queue. That is the whole feature, and it is missing from a build that has three queues already.\n\nRight now, assets get processed because a cron woke up and looked. Generated hero images, ArcAds output, absorbed repositories, uploaded references — each of those arrives in a bucket and then waits for a scheduled sweep to notice. The sweep runs every minute, which is fast enough to feel instant and is still the wrong mechanism: it polls whether or not anything happened, and it cannot tell you *why* it processed something.\n\n```\nwrangler r2 bucket notification create miscsubjects-store --event-type object-create --queue loop-tasks\n```\n\nWith that, the arrival of the object *is* the trigger. The queue message carries the bucket, the key and the event type, so the consumer knows exactly what changed rather than diffing a listing.\n\n**Verdict: install.** It is one command per bucket and it deletes polling code.\n\n## Analytics Engine\n\nAnalytics Engine accepts unlimited-cardinality analytics written from a Worker and queried with SQL. Writes are non-blocking and effectively free; you get one dataset binding and you write data points with blobs, doubles and an index.\n\n```toml\n[[analytics_engine_datasets]]\nbinding = \"METRICS\"\ndataset = \"loop_metrics\"\n```\n\n```js\nenv.METRICS.writeDataPoint({\n  blobs: [toolName, modelId, agentName, outcome],\n  doubles: [latencyMs, tokensIn, tokensOut, costUsd],\n  indexes: [agentName],\n});\n```\n\nThis build already tries to answer cost and latency questions — there is a `COST_REPORT` row, a governor, and per-model accounting. Those work by reading the ledger back and aggregating it, which means the cost of asking a cost question scales with the size of the ledger.\n\nAnalytics Engine is the correct tool for that specific class of question because it is designed for high-cardinality dimensions. Per-tool, per-model, per-agent, per-outcome, forever, at a write cost that does not compete with the request. The ledger keeps being the record of what happened; Analytics Engine becomes the record of how much it cost and how long it took.\n\nThe one constraint worth knowing before adopting it: it is a metrics store, not an event store. Data points are sampled at high volume and are not the audit trail. Do not put anything in it that has to be exact.\n\n**Verdict: install.** It answers the cost question this build keeps asking, and it does not compete with the ledger for that role.\n\n## What this part does not recommend\n\n**Do not move the audit chain off D1.** The hash chain's value is that each row commits to the previous one at write time, inside a transaction, in the same request that performed the act. Streaming it through a batching pipeline first would put a gap between the act and the commitment, and the gap is exactly what the chain exists to close. Pipelines belongs on the high-volume observational streams. The chain stays where it is.\n\n## Verdicts\n\n| Product | What it replaces here | Verdict |\n| --- | --- | --- |\n| Pipelines | Per-row D1 inserts in the hot path for high-volume streams | **install** — agent turns first |\n| R2 Data Catalog | A ledger auditable only by a bespoke script | **install** |\n| R2 SQL | Export-and-count in a local script | **install** — with the catalog |\n| R2 event notifications | A cron sweep that polls buckets every minute | **install** |\n| Analytics Engine | Cost and latency questions answered by re-reading the ledger | **install** |\n| Pipelines *for the audit chain* | Nothing — it would weaken it | **no** |\n\nNext: [Part 3 — running real code](/a/cloudflare-os-xl-03-running-real-code).\n","register":"standard","hero":"https://miscsubjects.com/img/gen/arcads-gpt-image-ebfcc12b-c321-4aaf-84ed-7f1e9b5e9022.png","hero_brief":"A letterpress compositor's type case laid flat on a bench, metal type sorted into hundreds of small compartments in perfect grid order, a compositor's stick resting across one corner. Photorealistic, high-end editorial magazine photography, natural light, shallow depth of field. No readable text, no logos, no people facing camera.","editorial_review":{"headline_subject":"Turning an append-only ledger into a queryable table","hero_subject":"A letterpress compositor's type case with metal type sorted into a grid of compartments","visual_action":"A compositor's stick resting across the sorted case","rationale":"The argument is that the ledger already holds the records and lacks columnar structure; a type case is the same content organised into an addressable grid.","inspected":true,"inspection_note":"A worn wooden type case laid flat, hundreds of compartments each holding sorted metal type in perfect grid order, a setting stick across one corner. It reads as structure imposed on a pile, which is the argument.","hero_brief":"A letterpress compositor's type case laid flat on a bench, metal type sorted into hundreds of small compartments in perfect grid order, a compositor's stick resting across one corner. Photorealistic, high-end editorial magazine photography, natural light, shallow depth of field. No readable text, no logos, no people facing camera."},"tags":["cloudflare","pipelines","r2","ledger","analytics"],"category":"systems","style":{},"claims":[{"id":"c1","text":"The audit ledger exists as rows in D1 and receipt files in R2, and analytical questions about it are answered today by exporting files and counting them in a script.","tier":"observational","source_ids":[],"why_material":"A trust mechanism verifiable only by a bespoke script is verified by whoever wrote the script."},{"id":"c2","text":"Cloudflare Pipelines ingests streaming data and delivers it to R2 as Apache Iceberg tables or as Parquet and JSON files, which removes the per-row insert from the hot path of the request being recorded.","tier":"definition","source_ids":["s-pipelines"],"why_material":"It changes ledger throughput from a function of D1 write capacity to a function of batching."},{"id":"c3","text":"R2 Data Catalog is a managed Apache Iceberg catalog built into an R2 bucket, and enabling it on the existing ledger bucket is a single wrangler command.","tier":"definition","source_ids":["s-catalog"],"why_material":"It converts an existing asset into a queryable one without moving it."},{"id":"c4","text":"R2 SQL is a distributed SQL engine over R2 Data Catalog, so the audit chain becomes answerable with a plain SELECT by any holder of the credential rather than only by a script author.","tier":"definition","source_ids":["s-r2sql"],"why_material":"External checkability is the point of an audit chain."},{"id":"c5","text":"R2 event notifications place a message on a queue when an object is created, which replaces the every-minute cron sweep that currently polls buckets whether or not anything arrived.","tier":"definition","source_ids":[],"why_material":"The build already runs three queues, so the consumer side exists."},{"id":"c6","text":"Analytics Engine accepts unlimited-cardinality data points written non-blocking from a Worker and queried with SQL, which fits per-tool and per-model cost accounting better than re-reading the ledger.","tier":"definition","source_ids":["s-ae"],"why_material":"The cost of asking a cost question currently scales with the size of the ledger."},{"id":"c7","text":"The hash-chained audit rows should stay in D1 rather than move to Pipelines, because the chain commits to the previous row inside the same request that performed the act.","tier":"expert","source_ids":["s-pipelines"],"why_material":"Batching would open the gap the chain exists to close."}],"sources":[{"id":"s-pipelines","type":"documentation","url":"https://developers.cloudflare.com/pipelines/","title":"Cloudflare Pipelines documentation","quote":"Ingest, transform, and deliver streaming data to R2 as Apache Iceberg tables or Parquet and JSON files."},{"id":"s-catalog","type":"documentation","url":"https://developers.cloudflare.com/r2/data-catalog/","title":"R2 Data Catalog documentation","quote":"A managed Apache Iceberg data catalog built directly into R2 buckets."},{"id":"s-r2sql","type":"documentation","url":"https://developers.cloudflare.com/r2-sql/","title":"R2 SQL documentation","quote":"A distributed SQL engine for R2 Data Catalog"},{"id":"s-ae","type":"documentation","url":"https://developers.cloudflare.com/analytics/analytics-engine/","title":"Workers Analytics Engine documentation","quote":"Send and query unlimited-cardinality analytics from Workers."}],"prov":{"model":"Opus 5 (Claude Code)","action":"write"}}