Free, Private Embeddings for Your AI Dev Brain — Powered by Ollama

The problem we kept hearing

When we launched the Cachly AI Dev Brain — persistent cross-session memory for AI coding assistants — the most common question was:

Short answer: you shouldn't have to. Memory is infrastructure. You don't pay Stripe every time you write to a database. So we fixed it.

The question is a fair one, though, and it deserves more than a slogan. It is worth being precise about what an embedding model is doing in a memory system at all — because once that is clear, both the privacy argument and the cost argument stop being marketing and start being arithmetic.

Why memory needs an embedding model at all

A Brain accumulates lessons. Hundreds of them, eventually thousands. When your assistant is about to touch a deploy script, the useful question is not “which lesson contains the word deploy” — it is “which lesson is about this kind of situation.”

Keyword search cannot answer that.

The lesson that saves you may talk about a restart loop and a healthcheck without ever using the word you typed. Meanwhile a lesson that mentions “deploy” four times may be about something else entirely — a deploy that went fine, in a service you no longer run.

Ranking by word overlap gets both of those backwards, and it does so confidently.

An embedding model turns each piece of text into a list of numbers positioned so that texts about similar things land near each other. Retrieval then becomes a distance question, which a database can answer quickly and consistently.

This is also the reason embedding calls are frequent rather than occasional. Every stored lesson needs one when it is written, and every recall needs one when it is asked. A memory system that charges per embedding is a memory system that charges you for remembering.

What we built

We now run nomic-embed-text via Ollama directly on our cachly infrastructure (Hetzner, Germany). Every embedding operation for the AI Brain — index_project, smart_recall, learn_from_attempts, remember_context — goes through this local model.

nomic-embed-text: Why we chose it

nomic-embed-text scores on par with OpenAI's best embedding models, is fully open-source, and fits under 300 MB — making it practical to run on existing server infrastructure without a GPU.

One column in that table is a trap worth naming. Dimensions are not quality. A 1536-dimension vector is not twice as good as a 768-dimension one; it is twice as large to store and slower to compare. What matters is the benchmark score, and there the gap is within noise of each other.

The size column matters for a different reason. A model under 300 MB runs on ordinary server CPUs, which is what makes “no API key” economically possible in the first place. A model that needed a GPU would simply move the bill rather than remove it.

How it works under the hood

MCP Tool Call: smart_recall("fix deploy")
       ↓
cachly MCP Server (npm)
       ↓ HTTP
cachly API (Go) — POST /api/v1/semantic/search
       ↓
EmbedHandler.Embed("fix deploy")
       ↓ HTTP (internal Docker network)
Ollama: POST http://ollama:11434/api/embeddings
       ↓
nomic-embed-text → 768-dim float32 vector
       ↓
pgvector: SELECT ... ORDER BY embedding <=> $1 LIMIT 10
       ↓
Top-k lessons returned to your AI assistant

Everything runs on a single Hetzner CPX32 node in Germany. The Ollama container uses max 700 MB RAM and stays idle at ~62 MB between requests.

Those two numbers are the interesting part of the whole setup. A service that idles at 62 MB can share a node with everything else instead of demanding its own. That is what turns “run your own embeddings” from an architecture diagram into a line item that rounds to nothing.

Bring your own model (optional)

If you'd rather use your own embedding provider, set these in your MCP config:

{
  "env": {
    "CACHLY_JWT": "your-jwt",
    "CACHLY_BRAIN_INSTANCE_ID": "your-instance-id",
    "CACHLY_EMBED_PROVIDER": "openai",
    "OPENAI_API_KEY": "sk-..."
  }
}

The Brain falls back automatically to your provider when CACHLY_EMBED_PROVIDER is set. Otherwise it uses our hosted nomic-embed-text instance.

The one decision that is expensive to reverse

Worth saying plainly, because it is the kind of thing that is obvious in hindsight and costly in practice: vectors produced by one model cannot be compared with vectors produced by another.

They are not merely on different scales. They are coordinates in different spaces, built by different training runs. Comparing them is not inaccurate — it is meaningless.

So the practical advice is unglamorous. If you have a policy reason to run embeddings through your own provider, set it up before you store much. If you do not, the default costs you nothing and keeps the text on one machine.

Either choice is defensible. Changing your mind after a year of accumulated lessons is the expensive path, and nothing in the tooling will stop you from taking it by accident.

Restricted and air-gapped environments

The bullet list above claims this works where outbound API calls are blocked. That claim is worth unpacking, because it is the case where hosted embeddings fail outright rather than merely costing money.

In a network that cannot reach a third-party API, a memory system built on hosted embeddings does not degrade gracefully. It stops. Every write and every recall depends on a round trip that will never complete.

A model that runs beside the database has no such dependency. The same 274 MB that made it cheap on a shared node is what makes it viable behind a firewall — it can simply be present.

An embedding service is now something you operate

Honesty about the trade: removing an API key does not remove a dependency. It converts an external one into an internal one.

A hosted provider fails in ways you read about on a status page. A local container fails in ways you find out about yourself, usually because something upstream started returning results that look thin rather than results that look broken.

So the thing worth watching is not uptime. It is whether embedding calls are still succeeding at all — a recall that quietly falls back to keyword matching still returns rows, and rows look like success from every angle except relevance.

What this does not solve

Local embeddings cover the memory layer, and only the memory layer. Your coding assistant still sends your prompts to whichever model you use — Claude, Copilot, Cursor, anything else. That traffic is not affected by any of the above.

What changes is the part that used to be an extra, avoidable disclosure: the text of your lessons, your cached context, and your recall queries. Those now stay put by default, and they were the hardest part to justify sending anywhere in the first place.

It is a narrower claim than “private AI,” and it is one we can actually draw on a diagram.

Narrow claims have a useful property: you can check them. Point a packet capture at the node during a recall and see for yourself which hosts it talks to. A promise that survives that test is worth more than a broad one that cannot be tested at all.

Get started

npx @cachly-dev/mcp-server@latest autopilot

No OPENAI_API_KEY needed. Just your CACHLY_JWT.