Ted Lasso recommends being a goldfish. It is, he says, the happiest animal alive: it forgets everything in ten seconds and moves on. My agent took that advice. Hermes is a goldfish — every conversation starts from zero, and anything it “knows” about me is either written into the prompt or fetched from somewhere else. And for a while I was one too: it would ask me again about things we had settled a week ago, and I would explain them again, from the beginning. The difference between us is that I could do something about it. So I spent a few weeks giving it a memory. We are goldfish, the agent and I, but only one of us was going to stay that way. This is what I learned about the options, what each of them actually means, and the architecture I ended up with.
The first thing to understand is that “memory” is not one thing. There are at least three different jobs hiding under that word.
The first job is context injection. Hermes has a built-in memory that is literally a markdown file, read into the system prompt of every session. It is always there, it is cheap to reason about, and it costs tokens on every single call, because it rides along whether it is relevant or not. When I audited my token spend, the always-on context was a large part of the input bill: on heavy days the agent was re-paying for the same notes dozens of times. Context injection is a good place for a few durable facts and a bad place for anything that grows.
The second job is recall: storing what was said and pulling the right fragment back when it matters. The third job is synthesis: looking across everything that was said and forming conclusions about the person. These two get conflated constantly, and they have completely different products behind them.
For recall, the name that comes up is Supermemory. It tops the memory benchmarks, it does temporal housekeeping (a fact about “tomorrow” expires, a newer address supersedes an older one), and it injects very few tokens to find a hit. The catch, in the independent test I read, was that it also had the most stale or forbidden retrievals: it remembers everything, including the things you told it to forget. For a chatbot that is a tradeoff. For an assistant that holds your health data and your finances, it is a disqualifier unless you gate it yourself.
For synthesis, the product is Honcho. Honcho does not primarily match chunks; it observes conversations in the background and derives conclusions: this is what the user prefers, this is how they like work done, this is the project they keep coming back to. It models people as entities with a history, and its main interface is a question you ask about someone, answered from what it has reasoned, not just retrieved. I chose it as the memory provider: it runs as a small server on my home network, and Hermes writes observations to it as we talk.
That was the decision. What followed was the part nobody writes about: the privacy work. A memory server is a recorder, and three lessons arrived the hard way.
The first lesson arrived through my lunch. I log expenses, and that week I had been pasting receipts into the chat for the agent to file. The memory layer observed those conversations, as designed, and drew its conclusions: this user ordered chole bhature, this is what it cost. (In my defence, I stay in Delhi, and there is only so long a person can hold out against chole bhature.) Then I discovered that stored context was being auto-injected into every prompt the agent sent to its cloud model. So my lunch order was now riding along in the context of every unrelated question I asked, travelling to a cloud provider each time. The lesson has two halves. What enters chat is what gets remembered, and what gets remembered gets re-sent: the memory’s recall channel is itself a privacy boundary. The fix was structural, not a setting: financial data moved to files and structured jobs the chat never sees, so the memory simply never learns it. I trust that more than any switch, because a rule that says “don’t remember this” can be undone by one careless config edit, and you would only find out after the fact — a pipeline that never carries the data has nothing to undo.
The second lesson arrived when my partner started using the agent. To understand what went wrong you need one piece of background: Hermes runs several profiles, separate configurations, like separate user accounts on one machine. My partner runs her own agent, on her own chat, through her own profile. That profile had never configured memory, because she never asked for it. And the config system has a fallback rule: if a profile does not set something, it inherits from the default profile. Which was mine. So the first time her agent started up, it silently connected to my memory server, read my stored context into her sessions, and began drawing conclusions about me — not her — from her conversations. The fix was one explicit setting on her profile, pointing it at its own empty store. The lesson: configuration inheritance is a convenience right up until it is a leak. If you run multiple users on one agent, check what each profile inherits before they use it, not after.
The third lesson arrived when I asked a simple question: how much does the memory actually know about me? The answer came back in pieces. The memory keys everything on a peer ID, and the same human talking over Matrix, over Telegram, from the command line produces a different peer ID each time, because each platform names you differently. My memory had quietly split into shards: several hundred conclusions on one side, a hundred-odd stranded under a bare numeric identifier on the other, some of it orphaned entirely. The fragmentation is not recoverable — there is no merge endpoint. Re-pointing an alias table stops future splitting, but the divided history stays divided. The lesson: a memory server will not recognize your own identity across platforms unless you tell it to, and the damage is silent. Give your identities an alias table on day one.
None of these are Honcho problems, particularly. They are the problems of any system that records a person. You will hit your own versions of all three.
Then there is the third system, and it is the one people find surprising: GBrain, wired in deliberately for a different purpose than memory.
GBrain is a second brain: markdown pages in a git repository, with a diary of every change kept alongside each page. You can search it the way you would search anything else, by meaning or by keyword. It passed a source-level security review that honestly not much self-hosted software would: digest-pinned images, privilege dropping, a database that never leaves loopback. Its retrieval is fine. That is not why I run it. I run it because it is the only one of the three systems where I can answer “what exactly did we decide, and when, and show me the redline.”
Honcho answers who I am. GBrain answers what was written down. Those are different questions, and the systems do not compete for the same slot. That was the insight that unlocked the architecture: the memory provider can only be one thing, but a second, independent store does not have to replace anything. So Hermes keeps Honcho as its memory provider, and writes the durable record of what happened to GBrain through a small writeback plugin, over MCP. No shared schema, no synchronisation, two tools reached for at different moments.
The deployment had teeth, which deserves its own honesty. GBrain requires a local embedding model, and only one specific one is supported. My local Ollama was down, and my RAM was nearly gone; the embedding model is small, but it runs alongside a Postgres and everything else on that box. The transcript import needed a size cap because my session database had grown past the adapter’s hard limit. None of it was unsolvable, but the “just run the container” story was really an afternoon of prerequisites. One namespace decision is worth making explicit too: I keep work context out of my personal brain, which means separate GBrain namespaces per profile. The walls are the product.
The afternoon of prerequisites was worth it, because the result is what I set out to build: three layers with three different jobs. The small hand-curated file still rides in every prompt, holding the few things that must never be forgotten and nothing that grows. Honcho holds the picture of who I am — preferences, working style, the ongoing narrative. GBrain holds what we actually did — every decision, with its date and its redline. When I ask what we decided about a project two weeks ago, the answer comes from GBrain, cited to a page. When I ask how I like my briefings written, the answer comes from Honcho, drawn from a hundred small observations.
If I could tell my past self three things, they would be these. Pick the synthesis layer first, because it is the one that changes how the agent behaves, not just what it can find. Keep secrets out of chat rather than out of memory, because chat is the intake valve, and what never enters the conversation never needs forgetting. And audit the writeback early, because a plugin that journals your conversations preserves the boilerplate as faithfully as the signal — in my first weeks of logs, most of the recorded bytes were scheduled-job prompts, not conversation. A memory system absorbs your noise along with your signal, and the only way to know which one is winning is to open the record and read it.