--- slug: directory-ledger-pattern title: "The Directory and the Ledger: a system that lists everything it can do and keeps everything it did" kind: architecture status: published hash: 9b372df35e80b7ab657a2cf1795c2fd40bfc512cd4f52ec22ea64f8a98efdcdc created: 2026-10-09T00:13:35.292Z updated: 2026-10-09T04:18:10.347Z summary: "An AI agent is only as trustworthy as two answers: what could it have done, and what did it actually do? miscsubjects answers both with data. A Directory holds every capability as a row that contains the exact call that runs it. A Ledger keeps every request and every answer as exact bytes, stored once under their hash. This is the architecture, with the measurements." tags: ["architecture","ai-agents","tool-catalog","event-log","content-addressing","postgresql","provenance"] above: [] below: ["[[unified-dispatch-macaroon-atoms]]", "[[atomic-tooling]]", "[[task-payload-provenance-article]]"] adjacent: ["[[authority-outside-the-model]]", "[[scoped-access]]"] nested: ["[[unified-dispatch-macaroon-atoms]]", "[[atomic-tooling]]", "[[task-payload-provenance-article]]"] --- # The Directory and the Ledger: a system that lists everything it can do and keeps everything it did *An AI agent is only as trustworthy as two answers: what could it have done, and what did it actually do? miscsubjects answers both with data. A Directory holds every capability as a row that contains the exact call that runs it. A Ledger keeps every request and every answer as exact bytes, stored once under their hash. This is the architecture, with the measurements.* ## Two questions every agent system has to answer Give an AI agent the ability to act, and two questions follow it everywhere. **What could it have done?** The honest answer is usually a list of tool names in a prompt, with the real instructions scattered through code, config files and someone's memory. **What did it actually do?** The honest answer is usually the agent's own summary, which is the one witness with a reason to flatter itself. miscsubjects answers both questions with data instead of prose. The **Directory** is a table with one row for each thing the system can do, and the row contains the exact call that does it. The **Ledger** is an append-only table with one row for each thing that happened, linked to the exact bytes that were sent and received. The two are joined by the same tool names and the same call ids, so a reviewer can walk from a capability to its uses and from a recorded call back to the contract that produced it. That is the whole pattern. The rest of this article is what it takes to make it true at the scale of a real system, with the numbers measured on the live build on 9 October 2026 (UTC). ```widget { "type": "stat", "value": "24,890", "label": "rows in the Directory: 1,583 tools, 3,239 articles, 785 flows, 201 credentials, 143 tasks and twenty other kinds of thing (index built 6 October 2026)" } ``` ## Part one: the Directory ### A capability is a row that holds its own call Most tool catalogs describe a tool. A Directory row **is** the tool, in the sense that matters to a machine: it carries a complete, runnable contract. A tool row begins with a name and a description, then one column per way of calling it, then its type, kind and id. Each way column holds one JSON object that can be pasted and run as written: ``` a web call {method, url, headers, body} an MCP call {method: "MCP", server, headers, tool, arguments} a command {method: "CLI", command} several in order an array of those code to paste {method: "CODE", language, code} ``` The contract writer renders 26 of these forms for every tool: the build's own REST door, a one-line URL, MCP over HTTP, the MCP client configuration for Claude Code, two command-line clients, a webhook, an Apple Shortcut, JavaScript and Python snippets, an SDK method, a direct function call, MCP over stdio, WebSocket, server-sent events, GraphQL, SSH, and the OIP dispatch door and its dry-run form. A tool that has a live scoped token gains three credential-free links. A tool that wraps an outside service gains an **upstream** form: the service's own call, exactly as it would be made. That is 30 forms for a fully equipped row. The point is not the count. The point is that **every form is derived from one canonical call**, `{"tool": NAME, "args": {...}}`, and the contracts page says so for each one: the only things that vary are where the call is sent, where the credential goes, and how the name and arguments are wrapped. A caller never has to learn a tool's upstream interface. It learns one shape and asks the Directory for the wrapper it needs. Three rules keep the contracts honest: - **Columns are matched by their header label, never by position.** Adding a way adds a column; nothing else moves. A header the code does not recognise is never guessed at. - **Secrets are names, not values.** A contract writes `[SECRET_NAME]` where a credential goes. The runner substitutes the value only into the outgoing request, never writes it back into the row, and refuses the call if the name is not bound to anything. A Directory can be read by people who must not see the keys, and still be complete. - **Calls into the retired previous build are refused by construction,** including when they hide inside a shell command, a JSON-RPC body or a relay. A catalog that can quietly reach an old system is a catalog with an unlisted door. The index is derived, not written. It is rebuilt from one section file per type only when one of those files changes, and it records the version tag of every section it was built from. Listing the Directory is one small read; opening a row is one targeted read of the section that holds it. ### The catalog problem, measured The industry has discovered the same wall from several directions: a model that loads every tool definition before choosing one spends most of its context on tools it will not use. Anthropic's engineering team describes the common failure plainly: [most MCP clients load all tool definitions upfront directly into context](https://www.anthropic.com/engineering/code-execution-with-mcp). OpenAI's documentation notes that function definitions [count against the model's context limit and are billed as input tokens](https://platform.openai.com/docs/guides/function-calling). Both vendors now ship deferred loading and tool search to cope. The Directory was built with the cheap projection as a first-class read, so the cost can be measured directly on this build: | Projection | Bytes | Tokens at 3.7 bytes per token | |---|---|---| | Every tool with its description (1,582 tools) | 817,371 | about 220,900 | | Every tool name only | 41,663 | about 11,300 | The name-only listing is 19.6 times smaller. The token figures are estimates from the byte counts, not tokenizer output. What a model does with that is the standard two-step: read the names, then open the one row it needs, with its description and its 26 to 30 contracts. The catalog is a database table, so the second step is a lookup, not a prompt. ### Words resolve to rows with a reason attached A catalog of 1,583 tools is useless if a caller has to know the exact name. The resolver takes anyone's words, a person's or a model's, and returns ranked rows, each with the layer that matched and the words that matched it. No embedding and no model is involved. A score is an integer that names its layer, never a probability: | Score | The row matched because | |---|---| | 1000 | its name is the query | | 900 | the query is the start of its name | | 850 | its name is exactly the words of the query | | 800 | every word of the query is a word of its name | | 750 to 772 | a word of the query is an alias that means this row | | 700 | every word of the query is somewhere in the row | | 600 | its name starts with the first word and the rest is in the row | | 300 | a word is one edit or one transposition away | | 200 | a word sounds like part of the name | The alias table maps 118 everyday words to the name patterns of the rows that do that job: "imessage" to the rows beginning `IMSG_`, `IMESSAGE` and `BLOOIO`; "article" to `ARTICLE_`, `WRITE_ARTICLE` and `ARTICLES`. The framing of a question is stripped first ("how do I", "where is my"), so a question word is never treated as a search term. Misspellings are corrected by Damerau-Levenshtein distance with an edit budget scaled to the word's length, then by Soundex. Every alias pattern was checked against the live tool list; an alias cannot name a row that does not exist. ```widget { "type": "json", "title": "Three live queries, with the layer that matched", "ref": "#facts.resolver.examples", "open_depth": 2 } ``` The third query is the deliberately weak end. Nothing in the Directory is about TikTok; the sound-alike layer returns files whose names contain "tests", scored 200, and says that is why. Because the score and reason travel with every row, a caller can refuse anything below a layer it trusts. The plain answer "nothing in the Directory is called, or means, this" appears only when no layer matches at all. **The resolver never invents a row.** That one property is what makes a 1,583-tool catalog safe to hand to a model: a wrong guess is impossible to make silently. The same resolver answers MCP `find`, the Directory's query route, site search and the contracts page. One implementation, four doors. ## Part two: the Ledger ### Exact bytes, stored once The Ledger is one append-only PostgreSQL table. Events are added; nothing is overwritten. Each event carries its metadata (kind, direction, tool, session, task, model, status, URL, cost, tokens) and a pointer to its payload, which is stored separately under its SHA-256 hash. The design follows two ideas that are each decades old and rarely combined in an agent system. The first is **event sourcing**: [all changes to application state are stored as a sequence of events](https://martinfowler.com/eaaDev/EventSourcing.html), so any past state can be rebuilt by replaying the log. The second is **content addressing**, the idea behind Git, where [the key is based on the content](https://en.wikipedia.org/wiki/Content-addressable_storage): identical bytes have one address, and fetching by address proves the bytes are unchanged. What that gives the Ledger: - **Every event carries its source's own stable id, and that id is the primary key.** A retry, a replay, or two collectors sending the same event add nothing: the insert is `INSERT … ON CONFLICT DO NOTHING`, which PostgreSQL defines as [simply avoiding inserting a row](https://www.postgresql.org/docs/current/sql-insert.html). Ids are fixed when the event is created, so a retried write is the same write. - **One batch is one transaction:** the payload blobs, the event rows, the batch's accounting row and the collector's delivered position commit together or not at all. Recording never records itself; the bookkeeping rows are not events. - **Payloads are content-addressed.** Each is stored under its SHA-256, compressed inline in PostgreSQL up to 256 KiB and in object storage above that. If the same tool list is answered a thousand times, its bytes are stored once and referenced by a thousand event rows. The history of occurrences is complete; the storage of identical content is not multiplied. - **The database enforces the shape.** A check constraint ties a payload state of `complete` or `partial` to a stored hash and size, and `none`, `artifact` or `missing` to no hash. A row cannot claim a payload it does not have. - **Text is labelled UTF-8 only when strict decoding gives back the identical bytes.** Anything else is stored as binary and never coerced. The bytes are kept exactly, with three stated exceptions, each written into the code rather than left as a surprise: the values of the build's own secrets found in recorded headers, bodies and URLs are replaced by the secret's name (and the names replaced are kept with the event); request bodies over 8 MiB are not read and are recorded as `missing`; deployed files are recorded by reference, not copied. ### Recording happens at the boundary, not by convention A logging discipline that depends on every feature remembering to log is a discipline with gaps. Here, each request runs inside an invocation context that follows it through every `await`, using AsyncLocalStorage, the mechanism Node.js provides to [associate state and propagate it throughout callbacks and promise chains](https://nodejs.org/api/async_context.html) and which [Cloudflare Workers implement](https://developers.cloudflare.com/workers/runtime-apis/nodejs/asynclocalstorage/). Inside that context: - The request in and the answer out are two events, linked by one id. - A streamed answer is teed. The Ledger's branch reads independently of the client. If the client disconnects, the Ledger keeps reading for up to 25 seconds and stores what arrived, marked `partial`. - Every outbound `fetch` the build makes is two events, request out and response in, with exact bytes. The tool runner and the model door record their own exchanges and switch the generic capture off for their calls, so nothing is stored twice. - The Ledger's own read routes are never recorded. A view that recorded itself would grow every time someone looked at it. - The storage layer's own traffic is never an event. A feature cannot forget to log, because logging is not something a feature does. It is something the boundary does to every feature. ### Durability is bought where someone can pay for it By default the Ledger write runs after the answer has gone out, using the Worker's ability to [perform work without blocking returning a response](https://developers.cloudflare.com/workers/runtime-apis/context/). No user waits on the Ledger. The cost is stated in the code: events a Worker holds in memory when it is killed before the write completes are lost. The build accepts that rather than add a queue. Inbound webhooks are the exception. For those paths the write is awaited before the build answers. If it fails, the sender gets a 503 with `retry-after` and sends again. The rule is general: make the write synchronous exactly where the other side can retry, and nowhere else. ### What the Ledger can answer Because the Ledger is a SQL table, every question about the system's behaviour is a projection, not a feature. The read routes answer one event whole (`/api/ledger/row/`), every event of one request (`/api/ledger/action/`), the raw bytes of one payload exactly as kept (`/api/ledger/blob/`), and counted views by hour, session, turn, kind, agent, model, tool or task. A question in plain words is parsed into the same filter object the screen uses; the SQL is never guessed. Two live readings give the scale. The counts route's one-day snapshot, taken 7 October 2026, held 293,015 events across 30 distinct tools, 8 models and 40 sessions. Grouping the last 30 days by kind, the "AI gateway call" kind alone was 547 events, 1,346,661 tokens and $0.90 of model cost, each figure summed from rows rather than estimated. ## Part three: why the join is the product A Directory alone is a better tool catalog. A Ledger alone is a better log. Joined by the same names and ids, they become something a reviewer can use: - **From a capability to its uses.** Open a tool's row, then ask the Ledger for every event with that tool name. The contract that was supposed to run sits beside the bytes that actually ran. - **From a recorded call back to its contract.** Open a Ledger action, read the tool name, open the row. If the recorded request does not match the contract, something other than the runner made it. - **From a task to everything done under it.** A scoped token can be bound to one task, so every call made with it is filed under that task. The task's page lists them, and the Ledger holds their payloads. That is the subject of [[task-payload-provenance-article|Task provenance]]. - **From a permission to the record of its enforcement.** Every refusal the permission checker makes is itself a Ledger row that names the limit which refused it. The limits are the subject of [[unified-dispatch-macaroon-atoms|Unified dispatch]]. The articles are stored the same way. This page is a JSON object at [/a/directory-ledger-pattern.json](https://miscsubjects.com/a/directory-ledger-pattern.json) with its own hash and a hash of every quoted source; the page, the Markdown and the machine copy are three views of one record. ```widget { "type": "note", "title": "Authority enforced outside the model", "text": "In this system the model does not decide what it may do, and does not write the record of what it did. Permissions are checked by code the model cannot see or change; actions are declared operations the runtime executes; the record is written by the runtime at the boundary. A model that is wrong, manipulated or working against its operator is simply not consulted on any of those three points. That is why the Directory and Ledger are candidate primitives for containing agent actions in development, evaluation and deployment, not only a tidier way to call tools. The full argument, including what is enforced today and what full containment would still need, is in [[authority-outside-the-model|Authority outside the model]]." } ``` ## The rest of the series - [[unified-dispatch-macaroon-atoms|Unified dispatch]]: one runner reaches the terminal, browsers, phone, cloud and services; scoped tokens, built on macaroons, lend out part of that authority with limits anyone can read. - [[atomic-tooling|Atomic tooling]]: one tool per domain with the orchestration inside it, so the model declares what it wants and the runtime does the plumbing. Measured against programmatic tool calling. - [[task-payload-provenance-article|Task provenance]]: a task's page assembled from the checker's records, a recorded run, and the article a task can become. ## Where the design stops - The Ledger's default write is deferred; an event held in memory when a Worker is killed is lost. Inbound webhooks are the only synchronous path. - Bodies over 8 MiB and deployed files are recorded by reference, not as bytes. - The Directory lists what has a row. A capability reached by a path that bypasses the runner leaves no contract behind and no Ledger row. Today the build's own agents still hold the full build key, so their calls are recorded but not limited. - The resolver is lexical. It handles aliases, misspellings and sound-alikes; it does not handle paraphrase it has never seen an alias for. ## What a reader can check from outside The code is private. These are public: - [/a/scoped-access](https://miscsubjects.com/a/scoped-access): written by the build from its own code each time it is opened. It lists every way of calling one tool, the token format, a worked signature chain, and every limit. - [/schemas/jarticle.json](https://miscsubjects.com/schemas/jarticle.json): the format this page is stored in. - [/a/directory-ledger-pattern.json](https://miscsubjects.com/a/directory-ledger-pattern.json): this page as its stored object, with its hash and the hash of every source quote. ## Facts ```json { "measured_at": "2026-10-09T05:10:00Z", "directory": { "rows": 24890, "types": 25, "untyped_rows": 4, "rows_by_type": { "variable": 17525, "article": 3239, "tool": 1583, "flow": 785, "cloudflare": 298, "file": 290, "skill": 267, "lead": 225, "credential": 201, "asset": 194, "task": 143, "method": 27, "kind": 25, "agent": 22, "setting": 17, "mac": 14, "contact": 9, "sheet": 9, "page": 5, "webhook": 3, "build": 1, "folder": 1, "outreach_lab": 1, "outside API": 1, "spreadsheet": 1 }, "tools_answered_by_api_tools": 1582, "call_forms_per_tool": { "always": 26, "with_a_token": 29, "with_a_token_and_an_upstream_service": 30 }, "contract_shapes": [ "{method, url, headers, body}", "{method: \"MCP\", server, headers, tool, arguments}", "{method: \"CLI\", command}", "an array of those, run in order", "{method: \"CODE\", language, code}" ], "secrets_in_contracts": "written as [SECRET_NAME]; the value is substituted only into the outgoing request; an unbound name refuses the call", "index": "derived from one section file per type; rebuilt only when a section changes; records the version tag of every section it was built from", "index_built_at": "2026-10-06T04:54:17.361Z" }, "catalog_projection": { "every_tool_with_description_bytes": 817371, "every_tool_name_only_bytes": 41663, "ratio": 19.6, "tokens_estimate_at_3_7_bytes_per_token": { "with_descriptions": 220911, "names_only": 11260 }, "note": "token counts are estimates from the byte counts at an assumed 3.7 bytes per token, not tokenizer measurements" }, "resolver": { "model_or_embeddings_used": false, "aliases": 118, "layers": [ { "score": 1000, "when": "the name is the query" }, { "score": 900, "when": "the query is the start of the name" }, { "score": 850, "when": "the name is exactly the words of the query" }, { "score": 800, "when": "every word of the query is a word of the name" }, { "score": "750-772", "when": "a word of the query is an alias that means the row" }, { "score": 700, "when": "every word of the query is somewhere in the row" }, { "score": 600, "when": "the name starts with the first word and the rest is in the row" }, { "score": 300, "when": "one edit or one transposition away (Damerau-Levenshtein, budget scaled to word length)" }, { "score": 200, "when": "sounds like part of the name (Soundex)" } ], "examples": [ { "query": "send an imessge", "read_as": [ "imessage" ], "top_row": "IMSG_SEND (tool)", "score": 764, "because": "one of the words means this row" }, { "query": "where is my terminal key", "read_as": [ "terminal", "key" ], "top_row": "TERMINAL_KEY (credential)", "score": 850, "because": "the row is named exactly the words of the query" }, { "query": "tiktok", "read_as": [ "tiktok" ], "top_row": "a file under tests/", "score": 200, "because": "one word of the query sounds like part of the name" } ], "on_no_match": "Nothing in the Directory is called, or means, \"\". Every row is listed at GET /api/directory" }, "ledger": { "table": "one append-only PostgreSQL table, ms.ledger, on Supabase, reached from the Worker through Hyperdrive with query caching off", "primary_key": "event_id: the source's own stable id, fixed when the event is created", "append": "one transaction per batch: payload blobs, event rows (INSERT ... ON CONFLICT DO NOTHING), the batch accounting row, the collector's delivered position", "payloads": { "key": "sha256", "inline_max_bytes": 262144, "inline": "compressed bytea in PostgreSQL", "larger": "object storage under sha256//" }, "payload_states": [ "complete", "partial", "none", "artifact", "missing" ], "exact_bytes_exceptions": [ "values of the build's own secrets in recorded headers, bodies and URLs are replaced by the secret's name, and the names replaced are kept with the event", "request bodies over 8 MiB are recorded as missing", "deployed files are recorded by reference" ], "stream_cut_off_seconds": 25, "write_timing": { "default": "after the answer has gone out; events a Worker holds in memory when it is killed first are lost", "inbound_webhooks": "awaited before answering; on failure the sender gets 503 with retry-after" }, "never_recorded": [ "the Ledger's own read routes", "the storage layer's own traffic", "the batch and position bookkeeping rows" ], "counts_route_snapshot": { "as_of": "2026-10-07T02:40:13.691Z", "window": "1d", "events": 293015, "distinct_tools": 30, "distinct_models": 8, "distinct_sessions": 40 }, "group_by_kind_30d_example": { "kind": "AI gateway call", "events": 547, "tokens": 1346661, "cost_usd": 0.901335 }, "read_routes": [ "GET /api/ledger/row/: one event whole", "GET /api/ledger/action/: every event of one request", "GET /api/ledger/blob/: the raw bytes exactly as kept", "GET /api/ledger/screen, /ask, /state, /counts, /tally, /group, POST /api/ledger/query" ] }, "series": [ "directory-ledger-pattern", "unified-dispatch-macaroon-atoms", "atomic-tooling", "task-payload-provenance-article" ] } ``` ## Sources 1. [Scoped access: the page the build writes from its own code](https://miscsubjects.com/a/scoped-access) — “Every capability of this build runs through one dispatch authority. A scoped token is a link-sized string that carries its own limits: which tools, by which doors, how many times, until when, for which task, inside which folders, for whom.” (sha256 257f6287e22a3fce) 2. [Code execution with MCP](https://www.anthropic.com/engineering/code-execution-with-mcp) — “Most MCP clients load all tool definitions upfront directly into context” (sha256 ed6e8b1978b1cbcf) 3. [OpenAI: Function calling](https://platform.openai.com/docs/guides/function-calling) — “callable function definitions count against the model’s context limit and are billed as input tokens” (sha256 39624dcf42f53a7a) 4. [Claude Developer Platform: Tool search tool](https://docs.claude.com/en/docs/agents-and-tools/tool-use/tool-search-tool) — “Instead of loading all tool definitions into the context window up front, Claude searches your tool catalog” (sha256 6e78ff92e0532f05) 5. [Event Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html) — “Event Sourcing ensures that all changes to application state are stored as a sequence of events.” (sha256 b989fc1be2ee949b) 6. [Event Sourcing: rebuilding state](https://martinfowler.com/eaaDev/EventSourcing.html) — “we can also use the event log to reconstruct past states” (sha256 6a02825a92d76ea0) 7. [Content-addressable storage](https://en.wikipedia.org/wiki/Content-addressable_storage) — “as the key is based on the content of the file, retrieving a document with a given key ensures that the stored file has not been changed” (sha256 3b917211417f0557) 8. [Git Internals: Git Objects](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects) — “Git is a content-addressable filesystem.” (sha256 a8330836fdd165a9) 9. [FIPS 180-4, Secure Hash Standard](https://csrc.nist.gov/pubs/fips/180-4/upd1/final) — “The digests are used to detect whether messages have been changed since the digests were generated.” (sha256 060aed903fca7812) 10. [PostgreSQL: INSERT](https://www.postgresql.org/docs/current/sql-insert.html) — “ON CONFLICT DO NOTHING simply avoids inserting a row as its alternative action.” (sha256 8e9b6273a14edde7) 11. [Node.js: Asynchronous context tracking](https://nodejs.org/api/async_context.html) — “These classes are used to associate state and propagate it throughout callbacks and promise chains.” (sha256 03710c205c1e0a84) 12. [AsyncLocalStorage on Cloudflare Workers](https://developers.cloudflare.com/workers/runtime-apis/nodejs/asynclocalstorage/) — “Cloudflare Workers provides an implementation of a subset of the Node.js AsyncLocalStorage” (sha256 987c8b22ff6111d7) 13. [Cloudflare Workers: Context (ctx)](https://developers.cloudflare.com/workers/runtime-apis/context/) — “extends the lifetime of your Worker, allowing you to perform work without blocking returning a response” (sha256 09ef1f0b4ce5b47d) 14. [Supabase Storage](https://supabase.com/docs/guides/storage) — “Supabase Storage is a robust, scalable solution for managing files of any size” (sha256 64c4ab4e4ca60975) 15. [Model Context Protocol: Tools](https://modelcontextprotocol.io/specification/2025-06-18/server/tools) — “The Model Context Protocol (MCP) allows servers to expose tools that can be invoked by language models.” (sha256 e6c355ddc14f05c0) 16. [Authority outside the model](https://miscsubjects.com/a/authority-outside-the-model) — “In this build a model does not set its own authority” (sha256 09ffea57374c7155) 17. [The jarticle/1 schema](https://miscsubjects.com/schemas/jarticle.json) — “set by verify: the hash of the page as fetched” (sha256 f927480db3c191bc) --- **below:** [[unified-dispatch-macaroon-atoms]], [[atomic-tooling]], [[task-payload-provenance-article]] **adjacent:** [[authority-outside-the-model]], [[scoped-access]] **nested:** [[unified-dispatch-macaroon-atoms]], [[atomic-tooling]], [[task-payload-provenance-article]]