cachly← All posts
Agent memory

I ranked AI memories by age. Every floor broke.

Bob Smith wrote an article on agent memory that argues for an append-only log with a projected view on top. In the comments he asked me something I owe a long answer to:

Have you hit the case where a supersede and a later append disagree, and if so, which wins?

Short answer: the explicit one. Being later wins nothing by itself.

The long answer starts with an experiment that failed.

The obvious idea

Agent memory goes stale. A note says "deploys go through script X". Three weeks later script X is gone, and the note is still there, sounding exactly as sure as it did on day one.

So the obvious move is to let age count. Older notes rank lower, fresh ones rise. Most memory designs I've read reach for some version of that. So did I. I changed the ranking in my memory server: the older a lesson, the further down it goes.

What the benchmark said

My server has a benchmark that runs on every pull request. It's a set of questions with known right answers, over two collections: my own, and an external one shaped differently. Each metric has a floor. Dropping below it blocks the merge.

With age in the ranking:

Right answer at rank 1 fell to 76.9 % against a floor of 90 %, mean reciprocal rank to 88.5 % against 94 %, nDCG@5 to 92.0 % against 96 %, and on the external set rank 1 fell to 67.9 % against 78 %.
Right answer at rank 1 fell to 76.9 % against a floor of 90 %, mean reciprocal rank to 88.5 % against 94 %, nDCG@5 to 92.0 % against 96 %, and on the external set rank 1 fell to 67.9 % against 78 %.

Every floor broke. I closed the pull request.

Why

The right answers were often weeks old. That's what useful memory looks like: the deploy trap you hit in July is exactly the thing you need in October. The age penalty pushed those answers under fresh notes that were merely related.

My mistake was using age as a stand-in for "no longer true". They're properties of different things. Age belongs to the note. Whether it still holds belongs to the fact. A lesson can be two days old and wrong, or two years old and right, and a timestamp can't tell those apart.

Two notes. One written two days ago, wrong since yesterday because the script it names was deleted. One written four months ago, still true. Ranking by age asks the clock; a correction that names its target asks the fact.
Two notes. One written two days ago, wrong since yesterday because the script it names was deleted. One written four months ago, still true. Ranking by age asks the clock; a correction that names its target asks the fact.

What decides instead

When two notes disagree, this is what decides it in my server, in plain terms:

So, Bob: an explicit supersede beats a later append. A later append on its own is one more voice.

The uncomfortable count

"Explicit beats implicit" has one catch. Someone has to write the explicit part.

In September I counted my own store. Of 735 lessons, 2 carried a replacement link. And those two pointed at each other.

735 grey dots, two red ones linked to each other: 2 of 735 lessons carry a link to the lesson they replace.
735 grey dots, two red ones linked to each other: 2 of 735 lessons carry a link to the lesson they replace.

Agents correct themselves constantly. They almost never say what they're correcting. When I looked into why, the answer was plain: the agent writing a correction rarely has the old note in front of it. And word overlap won't find the pairs for you. Real correction pairs share about as many words as unrelated lessons do.

So a design that waits for the writer to declare a supersede waits forever.

That's where Bob's other idea comes in: supersede-by-actor. A passing test that contradicts a model's note, or a human review that rejects it. Those actors know exactly which belief they just overturned. They're the ones who should write the link. I have half of it: a failed check already sits on top of the lesson it failed. The link from the check to the corrected version isn't written yet. That's the next piece.

Count it yourself

This works on any agent memory: a CLAUDE.md, a folder of notes, a vector store.

  1. Take your 20 newest entries.
  2. For each one, search for an older entry about the same thing.
  3. Count a: pairs that disagree. Count b: of those, how many say which one is current.

If b is much smaller than a, your memory settles every conflict by whatever the reader happens to see first.

I'd like to see your a and b. Mine, counted across the whole store: 2 explicit links in 735 lessons.

Where this stops

Bob's question came out of a longer thread under Edward Izgorodin's Everything It Remembers Has the Same Authority. If you care about this problem, the comments there are worth the time. Thanks to @bob_smith_eaea75d3ee1cba3 for asking the question this post answers, and to @izgorodin for the thread it grew out of.

Readers discussed this post on dev.to.

cachly.dev

For solo developers, teams and whole companies

Explain it once. Every AI knows it.

cachly keeps what you explain to your AI in every session. Claude Code, Cursor, Copilot and Windsurf read your rules, decisions and solved bugs: for you alone, for your whole team and for every team in your company. Servers in Germany, with SSO and a DPA for companies.

Servers in the EU · one command, restart your editor, done · 1 project free

Four ways to install

Claude Code
/plugin marketplace add cachly-dev/cachly-mcp
/plugin install cachly-brain@cachly

Then turn on auto-update once: /plugin → Marketplaces → cachly → Enable auto-update

Any MCP editor
npx @cachly-dev/mcp-server@latest autopilot