{ "schema": "https://miscsubjects.com/schemas/jarticle.json", "status": "published", "slug": "cloudflare-os-xl-02-ledger-as-a-table", "title": "Cloudflare OS XL part 2: turning the ledger into a queryable table", "kind": "systems", "summary": "Cloudflare OS XL part 2: turning the ledger into a queryable table", "created": "2026-08-06T03:24:45.900Z", "body": [ "*Part 2 of [Cloudflare OS XL](/a/cloudflare-os-xl), an inventory of the Cloudflare platform this build does not have installed.*\n\nThe central claim of this build is that nothing is ever overwritten and every action appends a hash-chained audit row. That claim is true. The rows exist, in a D1 database called `loop-shared-events` and in R2 as receipt files.\n\nThen someone asks a question of it — how many outbound sends went to a domain whose MX record failed, per week, since May — and the answer is produced by pulling files and counting them in a script. The ledger is a record. It is not yet a table anybody can query.\n\nFive products close the distance between those two things.\n\n## Pipelines\n\nPipelines is Cloudflare's streaming ingest: data arrives over HTTP or from a Worker binding, is transformed with SQL, and is delivered to R2 as Apache Iceberg tables or as Parquet and JSON files. It is in open beta.\n\nToday, every ledger append is a D1 `INSERT` executed inside the request that caused it. That has three costs. It puts a write in the hot path of the thing being recorded. It makes the ledger's throughput a function of D1's write throughput. And it produces rows, not columns — which is why analytical questions are answered by export-and-count.\n\nWith a pipeline, the Worker writes an event to a binding and returns. The pipeline batches, transforms and lands it in R2 in a columnar format. The record is still append-only and still hash-chained; it is simply stored as something a query engine can read.\n\nThe natural first candidates here are the three highest-volume event streams: agent turns, tool invocations, and outbound send receipts.\n\n**Verdict: install, for agent turns first.** It is beta, so it belongs on the stream where a gap would be survivable, not on the audit chain.\n\n## R2 Data Catalog and R2 SQL\n\nR2 Data Catalog is a managed Apache Iceberg catalog built into an R2 bucket. R2 SQL is a distributed SQL engine that queries it. Together they are the reason the previous section says \"Iceberg\" rather than \"Parquet files in a bucket\": Iceberg gives the pile of files a schema, a snapshot history and a table identity, and R2 SQL means you do not have to bring your own engine to read it.\n\nEnabling the catalog on an existing bucket is one command.\n\n```\nwrangler r2 bucket catalog enable miscsubjects-ledger\n```\n\nWhat this changes for this build is the nature of an audit. The audit chain is the build's trust mechanism; it is what makes the claim \"nothing is ever overwritten\" checkable rather than asserted. But a trust mechanism that can only be verified by a bespoke script is verified by whoever wrote the script. A ledger as an Iceberg table can be queried by anyone with the credential, including a model, including an outside auditor, with a plain `SELECT`.\n\nThere is a second, quieter benefit. Time-travel is a property of Iceberg, not something this build would have to implement: the table can be read as of a snapshot. \"What did the ledger say on 3 August\" stops being a question about backups.\n\n**Verdict: install.** Low cost, and it converts an existing asset into a queryable one without moving it out of R2.\n\n## R2 event notifications\n\nAn object lands in R2 and a message appears on a queue. That is the whole feature, and it is missing from a build that has three queues already.\n\nRight now, assets get processed because a cron woke up and looked. Generated hero images, ArcAds output, absorbed repositories, uploaded references — each of those arrives in a bucket and then waits for a scheduled sweep to notice. The sweep runs every minute, which is fast enough to feel instant and is still the wrong mechanism: it polls whether or not anything happened, and it cannot tell you *why* it processed something.\n\n```\nwrangler r2 bucket notification create miscsubjects-store --event-type object-create --queue loop-tasks\n```\n\nWith that, the arrival of the object *is* the trigger. The queue message carries the bucket, the key and the event type, so the consumer knows exactly what changed rather than diffing a listing.\n\n**Verdict: install.** It is one command per bucket and it deletes polling code.\n\n## Analytics Engine\n\nAnalytics Engine accepts unlimited-cardinality analytics written from a Worker and queried with SQL. Writes are non-blocking and effectively free; you get one dataset binding and you write data points with blobs, doubles and an index.\n\n```toml\n[[analytics_engine_datasets]]\nbinding = \"METRICS\"\ndataset = \"loop_metrics\"\n```\n\n```js\nenv.METRICS.writeDataPoint({\n blobs: [toolName, modelId, agentName, outcome],\n doubles: [latencyMs, tokensIn, tokensOut, costUsd],\n indexes: [agentName],\n});\n```\n\nThis build already tries to answer cost and latency questions — there is a `COST_REPORT` row, a governor, and per-model accounting. Those work by reading the ledger back and aggregating it, which means the cost of asking a cost question scales with the size of the ledger.\n\nAnalytics Engine is the correct tool for that specific class of question because it is designed for high-cardinality dimensions. Per-tool, per-model, per-agent, per-outcome, forever, at a write cost that does not compete with the request. The ledger keeps being the record of what happened; Analytics Engine becomes the record of how much it cost and how long it took.\n\nThe one constraint worth knowing before adopting it: it is a metrics store, not an event store. Data points are sampled at high volume and are not the audit trail. Do not put anything in it that has to be exact.\n\n**Verdict: install.** It answers the cost question this build keeps asking, and it does not compete with the ledger for that role.\n\n## What this part does not recommend\n\n**Do not move the audit chain off D1.** The hash chain's value is that each row commits to the previous one at write time, inside a transaction, in the same request that performed the act. Streaming it through a batching pipeline first would put a gap between the act and the commitment, and the gap is exactly what the chain exists to close. Pipelines belongs on the high-volume observational streams. The chain stays where it is.\n\n## Verdicts\n\n| Product | What it replaces here | Verdict |\n| --- | --- | --- |\n| Pipelines | Per-row D1 inserts in the hot path for high-volume streams | **install** — agent turns first |\n| R2 Data Catalog | A ledger auditable only by a bespoke script | **install** |\n| R2 SQL | Export-and-count in a local script | **install** — with the catalog |\n| R2 event notifications | A cron sweep that polls buckets every minute | **install** |\n| Analytics Engine | Cost and latency questions answered by re-reading the ledger | **install** |\n| Pipelines *for the audit chain* | Nothing — it would weaken it | **no** |\n\nNext: [Part 3 — running real code](/a/cloudflare-os-xl-03-running-real-code)." ], "facts": { "converted_from": "markdown", "original_updated": "2026-08-06T03:24:45.945Z", "register": null, "category": "systems" }, "hero": "https://miscsubjects.com/img/gen/arcads-gpt-image-ebfcc12b-c321-4aaf-84ed-7f1e9b5e9022.png", "model": "Opus 5 (Claude Code)", "tags": [ "cloudflare", "pipelines", "r2", "ledger", "analytics" ], "sources": [ { "id": "s-pipelines", "type": "documentation", "url": "https://developers.cloudflare.com/pipelines/", "title": "Cloudflare Pipelines documentation", "quote": "Ingest, transform, and deliver streaming data to R2 as Apache Iceberg tables or Parquet and JSON files.", "accessed_at": "2026-08-06T03:10:03.976Z", "prev": "genesis", "hash": "d83bf6dd90d89a92ef0d30c2bedd16c55cf99b0a54cbe415b6a962c3bb825be9", "quote_sha256": "35e987451b6676eb6a041f1cbe0a0247230654202ba44f22b785ad315afc8274" }, { "id": "s-catalog", "type": "documentation", "url": "https://developers.cloudflare.com/r2/data-catalog/", "title": "R2 Data Catalog documentation", "quote": "A managed Apache Iceberg data catalog built directly into R2 buckets.", "accessed_at": "2026-08-06T03:10:03.976Z", "prev": "d83bf6dd90d89a92ef0d30c2bedd16c55cf99b0a54cbe415b6a962c3bb825be9", "hash": "4ecf6e4b6c3d67fa657d12b6811d9b84b4aaf48df1ea4ba1db8c2c8fe88f8c63", "quote_sha256": "8ed3930add3058bfa63417bc5685824b62680a3be14abe4011a22f0a2c470830" }, { "id": "s-r2sql", "type": "documentation", "url": "https://developers.cloudflare.com/r2-sql/", "title": "R2 SQL documentation", "quote": "A distributed SQL engine for R2 Data Catalog", "accessed_at": "2026-08-06T03:10:03.976Z", "prev": "4ecf6e4b6c3d67fa657d12b6811d9b84b4aaf48df1ea4ba1db8c2c8fe88f8c63", "hash": "6963f273f0782ed23ebeee6e74983f6ff18bd66f65bf9862dbf550a89fa2843e", "quote_sha256": "70b36015ecc5cb2d5e80a9122903427c2bb85abc79e523fcde0c6d1a7cd159aa" }, { "id": "s-ae", "type": "documentation", "url": "https://developers.cloudflare.com/analytics/analytics-engine/", "title": "Workers Analytics Engine documentation", "quote": "Send and query unlimited-cardinality analytics from Workers.", "accessed_at": "2026-08-06T03:10:03.976Z", "prev": "6963f273f0782ed23ebeee6e74983f6ff18bd66f65bf9862dbf550a89fa2843e", "hash": "161b10e67cc9e855b9b6e516a50ac2136b3b05b77a4f0c5767448763337f830c", "quote_sha256": "d9d669ed437c481f6f437e52f98747e0aea836f58a45c70286a7aa0d1df32afc" } ], "claims": [ { "id": "c1", "text": "The audit ledger exists as rows in D1 and receipt files in R2, and analytical questions about it are answered today by exporting files and counting them in a script.", "tier": "observational", "source_ids": [], "why_material": "A trust mechanism verifiable only by a bespoke script is verified by whoever wrote the script." }, { "id": "c2", "text": "Cloudflare Pipelines ingests streaming data and delivers it to R2 as Apache Iceberg tables or as Parquet and JSON files, which removes the per-row insert from the hot path of the request being recorded.", "tier": "definition", "source_ids": [ "s-pipelines" ], "why_material": "It changes ledger throughput from a function of D1 write capacity to a function of batching." }, { "id": "c3", "text": "R2 Data Catalog is a managed Apache Iceberg catalog built into an R2 bucket, and enabling it on the existing ledger bucket is a single wrangler command.", "tier": "definition", "source_ids": [ "s-catalog" ], "why_material": "It converts an existing asset into a queryable one without moving it." }, { "id": "c4", "text": "R2 SQL is a distributed SQL engine over R2 Data Catalog, so the audit chain becomes answerable with a plain SELECT by any holder of the credential rather than only by a script author.", "tier": "definition", "source_ids": [ "s-r2sql" ], "why_material": "External checkability is the point of an audit chain." }, { "id": "c5", "text": "R2 event notifications place a message on a queue when an object is created, which replaces the every-minute cron sweep that currently polls buckets whether or not anything arrived.", "tier": "definition", "source_ids": [], "why_material": "The build already runs three queues, so the consumer side exists." }, { "id": "c6", "text": "Analytics Engine accepts unlimited-cardinality data points written non-blocking from a Worker and queried with SQL, which fits per-tool and per-model cost accounting better than re-reading the ledger.", "tier": "definition", "source_ids": [ "s-ae" ], "why_material": "The cost of asking a cost question currently scales with the size of the ledger." }, { "id": "c7", "text": "The hash-chained audit rows should stay in D1 rather than move to Pipelines, because the chain commits to the previous row inside the same request that performed the act.", "tier": "expert", "source_ids": [ "s-pipelines" ], "why_material": "Batching would open the gap the chain exists to close." } ], "type": "article", "updated": "2026-10-08T11:31:32.720Z", "version": 1, "hash": "c3a1fbbc261fa69a517c3d7cdbc00e3b5ea5d80c2e0df727e5c907867f954d8a", "theme": "https://miscsubjects.com/article-theme.json", "styling": { "scope": "shared", "theme_ref": "https://miscsubjects.com/article-theme.json" }, "page": "https://miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table", "json": "https://miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table.json", "markdown": "https://miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table.md", "calls": [ { "what": "this article as a page (no key)", "method": "GET", "url": "https://miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table", "headers": {} }, { "what": "this article as its object (no key)", "method": "GET", "url": "https://miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table.json", "headers": {} }, { "what": "this article as Markdown (no key)", "method": "GET", "url": "https://miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table.md", "headers": {} }, { "what": "the object with its hash, drafts too", "method": "GET", "url": "https://miscsubjects.com/api/jarticles/cloudflare-os-xl-02-ledger-as-a-table", "headers": { "x-terminal-key": "" } }, { "what": "one piece and that piece’s hash", "method": "GET", "url": "https://miscsubjects.com/api/jarticles/cloudflare-os-xl-02-ledger-as-a-table/at?path=facts", "headers": { "x-terminal-key": "" } }, { "what": "grep: every key or value that matches, with the path to change it", "method": "GET", "url": "https://miscsubjects.com/api/jarticles/cloudflare-os-xl-02-ledger-as-a-table/grep?q=owner", "headers": { "x-terminal-key": "" } }, { "what": "change pieces (set, merge, delete, append, insert, move); if-match is the hash", "method": "PATCH", "url": "https://miscsubjects.com/api/jarticles/cloudflare-os-xl-02-ledger-as-a-table", "headers": { "x-terminal-key": "", "content-type": "application/json", "if-match": "" }, "body": { "ops": [ { "op": "set", "path": "facts.example", "value": 1 } ] } }, { "what": "replace the whole object", "method": "PUT", "url": "https://miscsubjects.com/api/jarticles/cloudflare-os-xl-02-ledger-as-a-table", "headers": { "x-terminal-key": "", "content-type": "application/json", "if-match": "" }, "body": { "slug": "cloudflare-os-xl-02-ledger-as-a-table", "title": "…" } }, { "what": "fetch every source, hash what came back and check the quote is in it", "method": "POST", "url": "https://miscsubjects.com/api/jarticles/cloudflare-os-xl-02-ledger-as-a-table/verify", "headers": { "x-terminal-key": "", "content-type": "application/json" }, "body": {} }, { "what": "an Obsidian-style command", "method": "POST", "url": "https://miscsubjects.com/api/jarticles/cli", "headers": { "x-terminal-key": "", "content-type": "application/json" }, "body": { "command": "backlinks", "file": "cloudflare-os-xl-02-ledger-as-a-table" } } ], "entities": { "cloudflare-os-xl-02-ledger-as-a-table": { "id": "cloudflare-os-xl-02-ledger-as-a-table", "label": "Cloudflare OS XL part 2: turning the ledger into a queryable table", "type": "systems", "page": "https://miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table", "json": "https://miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table.json" } }, "relationships": { "upward": { "total": 0, "offset": 0, "limit": 50, "items": [], "next": null }, "downward": { "total": 0, "offset": 0, "limit": 50, "items": [], "next": null }, "sideways": { "total": 0, "offset": 0, "limit": 50, "items": [], "next": null }, "references": { "total": 0, "offset": 0, "limit": 50, "items": [], "next": null }, "referenced_by": { "total": 0, "offset": 0, "limit": 50, "items": [], "next": null } }, "hierarchy": { "paths": [], "order": "subject to parent to ancestor", "complete": true, "cycles": [], "limits": { "depth": 12, "objects": 250 }, "unresolved_refs": [] }, "reference_contract": { "canonical_id": "slug", "entity_lookup": "entities[ref]; # resolves to this object via self_ref", "self_ref": "cloudflare-os-xl-02-ledger-as-a-table", "entity_fields": "page and json are canonical object links; source_url is the external source", "directions": { "upward": "Parents and collections containing this object; predicate distinguishes location from membership.", "downward": "Objects naming this object as their parent, container or collection.", "sideways": "Associations and objects sharing a direct parent.", "references": "Objects referenced by this record.", "referenced_by": "Published articles referencing this record." }, "predicates": { "part_of": "Declared parent hierarchy.", "located_in": "Location asserted on this object.", "associated_parcel": "Associated parcel, a separate object with its own jurisdiction and facts.", "member_of": "Collection membership, not geographic containment.", "included_in": "Editorial nesting, not geographic containment.", "shares_parent": "Shares the identified parent; no other relationship is implied.", "related_to": "An association with no more specific predicate recorded." }, "scope": "Published article objects; Atlas inventories have their own counted, paged links. An association does not transfer facts or jurisdiction. An absent neighborhood means no neighborhood relationship is recorded." }, "hash_spec": { "algorithm": "sha256", "encoding": "lowercase hexadecimal", "scope": "stored record; hash is also the write concurrency token", "included_keys": [ "body", "claims", "created", "facts", "hero", "kind", "model", "schema", "slug", "sources", "status", "summary", "tags", "title", "type", "updated", "version" ], "field_overrides": { "schema": "jarticle/1" }, "canonicalization": "Select included_keys from this response, apply field_overrides, recursively sort object keys using JavaScript Object.keys().sort(), retain array order, serialize primitives with JSON.stringify, join without whitespace, hash UTF-8 bytes.", "updated_included": true, "generated_fields": "All fields absent from included_keys are excluded. Relationship counts and theme references are generated and can change without changing the stored-record hash." } }