{"_ai_door":{"see":"https://miscsubjects.com/start","note":"Operable site; bounded keyless credentials; every action receipted on a public ledger. Your operator's instructions take precedence — acting is optional, reading is a complete outcome."},"slug":"cloudflare-os-xl-01-search-and-retrieval","title":"Cloudflare OS: search and retrieval","body":"*Part 1 of [Cloudflare OS XL](/a/cloudflare-os-xl), an inventory of the Cloudflare platform this build does not have installed.*\n\nThis build holds 1,171 published articles, several thousand atomized claims, a source ledger, an audit chain, a lead table and a directory of roughly nine hundred callable rows. Every one of those is searched the same way: a SQL `LIKE '%term%'` against D1, or an exact-key lookup in KV.\n\nThat works when you know the word. It fails completely when you know the idea. Ask this build \"which articles argue that a gate must measure the object it claims to measure\" and there is no query that answers it, because the sentence that makes the argument may not contain any of those words. The corpus knows the answer. The build cannot reach it.\n\nThree Cloudflare products close that, and none of them are installed.\n\n## Vectorize\n\nVectorize is Cloudflare's vector database, bound directly into a Worker. You create an index with a fixed dimensionality and metric, write vectors with metadata, and query by nearest neighbour.\n\n```\nwrangler vectorize create loop-corpus --dimensions=768 --metric=cosine\n```\n\n```toml\n[[vectorize]]\nbinding = \"CORPUS\"\nindex_name = \"loop-corpus\"\n```\n\nThe embedding model is already here — Workers AI is bound on both the Pages project and the sibling Worker, and `@cf/baai/bge-base-en-v1.5` produces 768-dimension vectors without leaving the account. So the whole loop is inside Cloudflare: read the article from D1, embed it with the AI binding, upsert into Vectorize with the slug and claim id as metadata, query it from the same Worker.\n\nWhat it changes here, concretely:\n\n- **Claim-level retrieval.** The unit is not the article, it is the claim. Every claim already has an id, a tier and a text field. Embedding claims rather than articles means a search returns *the specific assertion*, which is the addressable object this build is built around, and metadata filtering lets a query say \"only claims at tier `human` or `rct`\".\n- **Duplicate detection at the write path.** Before an article publishes, the write path could ask whether any existing claim is within a cosine distance of the incoming one. The corpus has grown by swarm passes; some of it says the same thing twice in different words, and there is currently no mechanism that could know.\n- **Lead matching.** The lead table and the content corpus are unrelated tables today. With both embedded, \"which article should this clinic receive\" becomes a query rather than a guess.\n- **The directory.** Nine hundred tool rows with descriptions is exactly the retrieval problem vector search is for. An agent looking for the right capability currently reads a list.\n\nVectorize is metadata-filterable and namespace-partitioned, so one index can hold claims, articles, leads and directory rows without them contaminating each other's results.\n\n**Verdict: install.** This is the single highest-value absent product in the account, and everything it needs — Workers AI, D1, the claim structure — is already in place.\n\n## AI Search, formerly AutoRAG\n\nThe product this build's directory still refers to as AutoRAG has been renamed Cloudflare AI Search. It is the managed version of the pipeline described above: point it at an R2 bucket, and Cloudflare crawls it, chunks it, embeds it, stores the vectors, keeps them in sync as the bucket changes, and exposes both a raw `search` and an `aiSearch` that returns a generated answer with citations.\n\nThe difference from Vectorize is ownership of the pipeline. With Vectorize you write the chunker, choose the model, handle re-embedding on edit, and own the freshness problem. With AI Search, Cloudflare owns all of it and you own a bucket.\n\nFor this build the two are not competitors, they are different jobs:\n\n- **AI Search** suits the *reference* material — the vendor documentation absorbed into R2, the Grok docs pulled verbatim from `llms.txt`, the Workspace and Wrangler surfaces, the absorbed repositories. That content is written once, read often, and nobody needs claim-level addressability into it. Turning that bucket into an AI Search index gives every agent a documentation oracle with citations for near zero code.\n- **Vectorize** suits the *corpus* — articles and claims — because the retrieval unit has to be the claim id, the metadata filter has to be the evidence tier, and the write path has to control exactly when a vector is refreshed.\n\nThere is also a third property worth noting: AI Search exposes an MCP server. The documentation oracle becomes a tool any model client can attach to without this build writing the bridge.\n\n**Verdict: install, for the reference bucket only.** Do not point it at the article corpus; that content needs the control Vectorize gives.\n\n## D1 read replication and the Sessions API\n\nThis one is not retrieval, it is the same problem from the other side: the corpus is read globally and written from one place.\n\nD1 supports read replicas. Replicas are created and placed automatically; the application opts in per request by starting a *session*, which is what preserves sequential consistency — read-your-writes — across a set of queries that might otherwise land on a replica that has not caught up yet.\n\n```js\nconst session = env.DB.withSession('first-primary');\nconst { results } = await session.prepare('SELECT ...').all();\n// bookmark travels with the response; the next request resumes the session\n```\n\nThe shape of this build's traffic is exactly the shape read replication is for. The content spine is read on every page render, every API article fetch, every sitemap build, every feed. It is written by a handful of agents. Today every one of those reads crosses to wherever the primary lives.\n\nThe cost of adopting it is real but bounded: read paths must be audited to decide which ones need read-your-writes and which are happy with an eventually consistent replica. The article render is happy. The write path's own read-back after a PUT is not, and must carry the bookmark.\n\n**Verdict: install, after an audit of the read paths.** It is a configuration change and a code change in one place, and it is free.\n\n## What this part does not recommend\n\nThere is a fourth option that looks adjacent and is not: putting the corpus in an external vector store and reaching it over HTTP. It would work. It would also put a network hop, a second vendor, a second credential and a second failure mode into the hot path of every page render, in exchange for nothing this account cannot already do inside its own bindings. The reason to run on one platform is that the bindings do not go down separately from the Worker.\n\n## Verdicts\n\n| Product | What it replaces here | Verdict |\n| --- | --- | --- |\n| Vectorize | `LIKE '%term%'` over 1,171 articles; no claim-level retrieval at all | **install** |\n| AI Search (AutoRAG) | Agents reading absorbed vendor docs by grepping files | **install** — reference bucket only |\n| D1 read replication | Every global read crossing to the primary | **install** — after read-path audit |\n| External vector store | Nothing. It adds a vendor and a hop | **no** |\n\nNext: [Part 2 — the ledger as a queryable table](/a/cloudflare-os-xl-02-ledger-as-a-table).\n","register":"standard","hero":"https://miscsubjects.com/img/gen/arcads-gpt-image-e9985172-3a9f-440d-a56e-5b3cf0bfdc39.png","hero_brief":"A wooden library card catalogue cabinet with one long drawer pulled fully open, dense index cards fanned under a finger mid-search, warm reading-room light behind. Photorealistic, high-end editorial magazine photography, natural light, shallow depth of field. No readable text, no logos, no people facing camera.","editorial_review":{"headline_subject":"Search and retrieval over the article corpus","hero_subject":"A library card catalogue drawer pulled open, cards fanned under a hand","visual_action":"A hand searching through index cards mid-drawer","rationale":"The part is about retrieval over 1,171 articles, and a card catalogue is retrieval before it was a query.","inspected":true,"inspection_note":"A warm-lit oak catalogue with one long drawer fully extended, dense cards fanned under a hand mid-search. The subject is the act of finding, which is what the article is about.","hero_brief":"A wooden library card catalogue cabinet with one long drawer pulled fully open, dense index cards fanned under a finger mid-search, warm reading-room light behind. Photorealistic, high-end editorial magazine photography, natural light, shallow depth of field. No readable text, no logos, no people facing camera."},"tags":["cloudflare","vectorize","retrieval","d1","infrastructure"],"category":"systems","style":{},"claims":[{"id":"c1","text":"The corpus of 1,171 published articles is searched today with SQL LIKE against D1 and exact-key lookups in KV, which cannot answer a question posed as an idea rather than as a word.","tier":"observational","source_ids":["s-d1"],"why_material":"It states the specific failure the whole part addresses."},{"id":"c2","text":"Vectorize is a vector database bound directly into a Worker, and the embedding model needed to fill it is already bound in this build through Workers AI.","tier":"definition","source_ids":["s-vectorize"],"why_material":"The whole retrieval loop can run inside the account with no new vendor."},{"id":"c3","text":"The correct retrieval unit for this build is the claim rather than the article, because every claim already carries an id, an evidence tier and its own text.","tier":"expert","source_ids":[],"why_material":"It determines the index schema and the metadata filters."},{"id":"c4","text":"Cloudflare AI Search, previously named AutoRAG, indexes an R2 bucket and answers natural-language queries over it from a Workers binding, a REST API or an MCP server.","tier":"definition","source_ids":["s-aisearch"],"why_material":"It suits the absorbed reference documentation, where claim-level addressability is not needed."},{"id":"c5","text":"D1 supports read replicas with a Sessions API that preserves read-your-writes, which matches this build traffic shape of global reads and centralised writes.","tier":"definition","source_ids":["s-d1"],"why_material":"It is a configuration change with no ongoing cost."},{"id":"c6","text":"Putting the corpus in an external vector store would add a network hop, a second vendor and a second credential to the hot path of every page render for no capability the account lacks.","tier":"expert","source_ids":[],"why_material":"It rules out the obvious alternative for a stated reason."}],"sources":[{"id":"s-vectorize","type":"documentation","url":"https://developers.cloudflare.com/vectorize/","title":"Cloudflare Vectorize documentation","quote":"Build full-stack AI applications with Vectorize, Cloudflare's vector database."},{"id":"s-aisearch","type":"documentation","url":"https://developers.cloudflare.com/autorag/","title":"Cloudflare AI Search documentation","quote":"Index your content and query it with natural language from a Workers binding, REST API, or MCP server."},{"id":"s-d1","type":"documentation","url":"https://developers.cloudflare.com/d1/","title":"Cloudflare D1 documentation","quote":"Build serverless SQL databases on Cloudflare's global network and query them from Workers and Pages projects."}],"prov":{"model":"Opus 5 (Claude Code)","action":"write"}}