# Cloudflare OS: search and retrieval

slug: cloudflare-os-xl-01-search-and-retrieval · https://miscsubjects.com/a/cloudflare-os-xl-01-search-and-retrieval · category: systems · tags: cloudflare, vectorize, retrieval, d1, infrastructure · updated 2026-08-06T03:28:32.619Z

*Part 1 of [Cloudflare OS XL](/a/cloudflare-os-xl), an inventory of the Cloudflare platform this build does not have installed.*

This build holds 1,171 published articles, several thousand atomized claims, a source ledger, an audit chain, a lead table and a directory of roughly nine hundred callable rows. Every one of those is searched the same way: a SQL `LIKE '%term%'` against D1, or an exact-key lookup in KV.

That works when you know the word. It fails completely when you know the idea. Ask this build "which articles argue that a gate must measure the object it claims to measure" and there is no query that answers it, because the sentence that makes the argument may not contain any of those words. The corpus knows the answer. The build cannot reach it.

Three Cloudflare products close that, and none of them are installed.

## Vectorize

Vectorize is Cloudflare's vector database, bound directly into a Worker. You create an index with a fixed dimensionality and metric, write vectors with metadata, and query by nearest neighbour.

```
wrangler vectorize create loop-corpus --dimensions=768 --metric=cosine
```

```toml
[[vectorize]]
binding = "CORPUS"
index_name = "loop-corpus"
```

The embedding model is already here — Workers AI is bound on both the Pages project and the sibling Worker, and `@cf/baai/bge-base-en-v1.5` produces 768-dimension vectors without leaving the account. So the whole loop is inside Cloudflare: read the article from D1, embed it with the AI binding, upsert into Vectorize with the slug and claim id as metadata, query it from the same Worker.

What it changes here, concretely:

- **Claim-level retrieval.** The unit is not the article, it is the claim. Every claim already has an id, a tier and a text field. Embedding claims rather than articles means a search returns *the specific assertion*, which is the addressable object this build is built around, and metadata filtering lets a query say "only claims at tier `human` or `rct`".
- **Duplicate detection at the write path.** Before an article publishes, the write path could ask whether any existing claim is within a cosine distance of the incoming one. The corpus has grown by swarm passes; some of it says the same thing twice in different words, and there is currently no mechanism that could know.
- **Lead matching.** The lead table and the content corpus are unrelated tables today. With both embedded, "which article should this clinic receive" becomes a query rather than a guess.
- **The directory.** Nine hundred tool rows with descriptions is exactly the retrieval problem vector search is for. An agent looking for the right capability currently reads a list.

Vectorize is metadata-filterable and namespace-partitioned, so one index can hold claims, articles, leads and directory rows without them contaminating each other's results.

**Verdict: install.** This is the single highest-value absent product in the account, and everything it needs — Workers AI, D1, the claim structure — is already in place.

## AI Search, formerly AutoRAG

The product this build's directory still refers to as AutoRAG has been renamed Cloudflare AI Search. It is the managed version of the pipeline described above: point it at an R2 bucket, and Cloudflare crawls it, chunks it, embeds it, stores the vectors, keeps them in sync as the bucket changes, and exposes both a raw `search` and an `aiSearch` that returns a generated answer with citations.

The difference from Vectorize is ownership of the pipeline. With Vectorize you write the chunker, choose the model, handle re-embedding on edit, and own the freshness problem. With AI Search, Cloudflare owns all of it and you own a bucket.

For this build the two are not competitors, they are different jobs:

- **AI Search** suits the *reference* material — the vendor documentation absorbed into R2, the Grok docs pulled verbatim from `llms.txt`, the Workspace and Wrangler surfaces, the absorbed repositories. That content is written once, read often, and nobody needs claim-level addressability into it. Turning that bucket into an AI Search index gives every agent a documentation oracle with citations for near zero code.
- **Vectorize** suits the *corpus* — articles and claims — because the retrieval unit has to be the claim id, the metadata filter has to be the evidence tier, and the write path has to control exactly when a vector is refreshed.

There is also a third property worth noting: AI Search exposes an MCP server. The documentation oracle becomes a tool any model client can attach to without this build writing the bridge.

**Verdict: install, for the reference bucket only.** Do not point it at the article corpus; that content needs the control Vectorize gives.

## D1 read replication and the Sessions API

This one is not retrieval, it is the same problem from the other side: the corpus is read globally and written from one place.

D1 supports read replicas. Replicas are created and placed automatically; the application opts in per request by starting a *session*, which is what preserves sequential consistency — read-your-writes — across a set of queries that might otherwise land on a replica that has not caught up yet.

```js
const session = env.DB.withSession('first-primary');
const { results } = await session.prepare('SELECT ...').all();
// bookmark travels with the response; the next request resumes the session
```

The shape of this build's traffic is exactly the shape read replication is for. The content spine is read on every page render, every API article fetch, every sitemap build, every feed. It is written by a handful of agents. Today every one of those reads crosses to wherever the primary lives.

The cost of adopting it is real but bounded: read paths must be audited to decide which ones need read-your-writes and which are happy with an eventually consistent replica. The article render is happy. The write path's own read-back after a PUT is not, and must carry the bookmark.

**Verdict: install, after an audit of the read paths.** It is a configuration change and a code change in one place, and it is free.

## What this part does not recommend

There is a fourth option that looks adjacent and is not: putting the corpus in an external vector store and reaching it over HTTP. It would work. It would also put a network hop, a second vendor, a second credential and a second failure mode into the hot path of every page render, in exchange for nothing this account cannot already do inside its own bindings. The reason to run on one platform is that the bindings do not go down separately from the Worker.

## Verdicts

| Product | What it replaces here | Verdict |
| --- | --- | --- |
| Vectorize | `LIKE '%term%'` over 1,171 articles; no claim-level retrieval at all | **install** |
| AI Search (AutoRAG) | Agents reading absorbed vendor docs by grepping files | **install** — reference bucket only |
| D1 read replication | Every global read crossing to the primary | **install** — after read-path audit |
| External vector store | Nothing. It adds a vendor and a hop | **no** |

Next: [Part 2 — the ledger as a queryable table](/a/cloudflare-os-xl-02-ledger-as-a-table).


## Sources

1. Cloudflare Vectorize documentation — https://developers.cloudflare.com/vectorize/
2. Cloudflare AI Search documentation — https://developers.cloudflare.com/autorag/
3. Cloudflare D1 documentation — https://developers.cloudflare.com/d1/