Research
Your AI agent forgets everything between sessions. Every single time. We built a 50-question eval framework, tested 5 memory solutions against real production data, and found that Cognee with hook-based architecture delivered a 13x improvement over baseline. Zero hallucinations across every system.
March 2026 | Dry Ground AI Research
The Problem
We run a production AI assistant called Hank on the OpenClaw framework. Hank manages our workspace, tracks projects, handles operational tasks. He's good at his job, with one glaring exception: he wakes up every morning with no idea what happened yesterday.
"What did we work on last week?" Blank stare. "What's the status of the CompanyClaw migration?" Gone. "What voice do I prefer for TTS?" Sometimes he gets that one, sometimes not.
This isn't a minor annoyance. It's a productivity tax on every session. We were re-explaining context, re-stating preferences, re-orienting our own assistant multiple times per day. For a company that builds AI solutions for clients, that's not acceptable.
So we stopped complaining and started measuring.
The Framework
We needed a repeatable, quantifiable way to evaluate memory systems. Not vibes. Not "it feels smarter." Numbers. We built a 50-question evaluation framework across five categories:
Retained user preferences. "What TTS voice does Keith prefer?"
Direct recall. "What is the service ID for the production API?"
Connecting two or more pieces of information. "Which deployment uses the staging Stripe account?"
Time-aware memory. "What was the last infrastructure change made this week?"
Awareness of in-progress work. "What branch is the current deploy targeting?"
Ground truth came from 30 days of actual workspace history: daily memory files, topic files, configuration docs, session logs. Real operational data, not synthetic benchmarks.
Our first draft of this article reported different numbers. Higher numbers. We want to be transparent about why they changed.
The original evaluation used a simpler judge: it checked whether the memory system retrieved relevant text. If the right information appeared somewhere in the retrieval results, we counted it as a win. That approach was generous.
For the final results below, we switched to a full-pipeline test: retrieve context from the memory system, have GPT-4.1-nano answer the question using only that context, then judge the answer against ground truth. This is closer to what actually happens in production. The stricter methodology dropped some scores significantly. That's the point. We'd rather publish accurate numbers than flattering ones.
Results
13x
Improvement over baseline
Cognee at 53% vs QMD at 4%
0%
Hallucination rate
Across every system tested
54%
Best overall score
Cognee + hook-based architecture
$0.01
Cost per eval run
Cheap enough to run continuously
50
Evaluation questions
Across 5 categories
30
Days of ground truth
Real operational data
Every system got the same 50 questions against the same ground truth, judged through the same full pipeline.
| System | Overall | Preferences | Single-Hop | Multi-Hop | Temporal | Active Ctx | Halluc. |
|---|---|---|---|---|---|---|---|
| Cognee + Hooks | 54% | 70% | 60% | 59% | 40% | 39% | 0% |
| Cognee (tool-based) | 53% | 70% | 75% | 59% | 25% | 33% | 0% |
| ByteRover | 12%* | 45% | 0% | 9% | 0% | 5.6% | 0% |
| Postgres/pgvector KG | 6%* | 10% | 0% | 0% | 15% | 5.6% | 0% |
| QMD Baseline | 4% | 10% | 0% | 0% | 10% | 0% | 0% |
| Cipher | DNF | Could not configure reliably | |||||
| Supermemory | Skipped | Cloud-only; data sovereignty concerns | |||||
*ByteRover and KG pgvector were tested with the original retrieval-only methodology. Their scores would likely be lower under the stricter full-pipeline test.
Deep Dive
This is what ships with OpenClaw out of the box. QMD (Quick Memory Documents) stores markdown files that get loaded into context at session start.
Under the full-pipeline test, QMD scored 4%. That's not a typo. The original evaluation gave it 22% because it only checked if grep found relevant text in the files. When we actually asked an LLM to answer questions from QMD's output, it almost never produced a correct answer. The text was there in theory. In practice, the retrieval was too unfocused for an LLM to extract the right answer.
It's a scratchpad, not a memory system. Good to know.
Cognee uses a knowledge graph with an ECL (Extract, Cognify, Load) pipeline. We pointed it at 49 workspace files and let it build its graph. Ingestion took 946 seconds (about 16 minutes), which is a one-time cost.
Under the full pipeline, Cognee scored 53% overall. That's down from the 70% we reported in the retrieval-only test, but it's a more accurate number. 70% measured whether the right text showed up in retrieval results. 53% measures whether an LLM can actually answer from what Cognee returns.
The category breakdown tells the real story: 70% on preferences, 75% on single-hop facts, 59% on multi-hop reasoning. Those are strong. Temporal recall (25%) and active context (33%) are weaker, which makes sense. Time-sensitive information is inherently harder to retrieve from a static knowledge graph.
That 53% vs 4% gap is the headline. Cognee delivered a 13x improvement over baseline.
This is the same Cognee backend, but with a different integration pattern. Instead of the agent deciding when to search and save, hooks handle both automatically.
The before_prompt_build hook searches Cognee and injects relevant context before every turn. The agent_end hook captures each conversation turn and feeds it to Cognee automatically. The agent never has to decide "should I save this?" or "should I search my memory?" It just happens.
The overall score (54%) is nearly identical to tool-based Cognee (53%). That's actually the important finding. The retrieval quality is the same regardless of whether a hook or a tool triggers the search. What changes is consistency. The hook fires every single turn. The tool only fires when the agent thinks to use it.
In production, that consistency matters more than any benchmark score. The agent that always checks its memory will outperform the agent that sometimes forgets to check.
ByteRover uses a hierarchical context tree that summarizes information at different levels of abstraction. Interesting architecture. Disappointing results.
The core problem is that summarization loses detail. When you compress "the service ID is srv-abc123" into a summary about "infrastructure configuration," the specific fact disappears. That's fatal for operational recall.
*Tested with the original retrieval-only methodology. Score would likely be lower under the full pipeline.
This was our existing knowledge graph, a 76-node Postgres database with pgvector embeddings. We built it for relationship mapping between concepts, projects, and people.
It scored 6% overall. Not surprising in hindsight. The graph contains entity relationships ("Keith is CEO of Dry Ground AI"), not operational facts ("we deployed to staging on Tuesday"). Wrong data type entirely for this use case.
*Tested with the original retrieval-only methodology.
Architecture
Supermemory's architecture pattern is the most interesting thing we found in this whole evaluation. The idea: memory capture and recall should be implicit, not explicit. The agent shouldn't have to decide to save or search. It should just happen.
At Dry Ground AI, we build CompanyClaw, our AI platform for enterprise clients. Security and privacy are core tenets of our architecture, not afterthoughts. Our clients trust us with their operational data, their internal processes, their competitive intelligence. Sending that data to a third-party memory service introduces a dependency and a data flow we can't fully control.
Every system we tested runs locally. Our workspace files never leave the machine. Cloud-based memory services may be convenient, but convenience doesn't trump data sovereignty.
We studied Supermemory's architecture and built our own version using Cognee as the backend:
A before_prompt_build hook searches Cognee's knowledge graph and injects relevant context before every turn. The agent never knows it's happening. It just has better context.
An agent_end hook captures each conversation turn and feeds it to Cognee automatically. No manual "save this" commands. Every interaction gets indexed.
Supermemory self-reports 85.9% accuracy on their own benchmark (MemoryBench). Our implementation scored 54% on our benchmark. Different benchmarks, different questions, not directly comparable. But the architecture pattern is validated: hook-based Cognee (54%) matched tool-based Cognee (53%), proving the retrieval quality is identical. The hook architecture's value isn't better retrieval. It's guaranteed retrieval. Every turn, every time.
For anyone building AI agents with persistent memory, this is the pattern to steal. You don't need a cloud subscription. You need two hooks and a decent knowledge graph backend. Same implicit capture/recall pattern as the $20/month cloud service, but fully local and free.
Insights
Cognee at 53% vs QMD at 4%. That's not incremental. That's the difference between a memory system and a text file.
This surprised us. Every system we tested either returned the right answer, said "I don't know," or returned an incomplete answer. None of them fabricated information. That's a meaningful data point for anyone worried about memory systems introducing false context.
Hook-based and tool-based Cognee score nearly identically (54% vs 53%), proving the retrieval quality is the same regardless of trigger mechanism. The hook architecture's real value is consistency: it fires every turn. The tool only fires when the agent remembers to use it.
Cognee's knowledge graph approach outperformed pure vector similarity (pgvector) by a wide margin. The graph captures relationships between facts, which matters when questions require connecting multiple pieces of information.
ByteRover's hierarchical summarization is elegant, but operational memory needs specifics: IDs, dates, exact configurations. Summarization is the enemy of specificity.
Our original methodology was too generous. Checking if retrieval found relevant text is not the same as checking if an LLM can answer from that text. The full-pipeline approach dropped scores across the board, but the results are more reliable and more useful for production planning.
Looking Ahead
Based on these results, we're running the Cognee + Hooks architecture in production:
The before_prompt_build and agent_end hooks are running. Every session automatically captures context and retrieves relevant memories. No manual intervention.
Cognee for deep knowledge graph recall. QMD for fast preference loading (it scored 10% on preferences, so it's not completely useless for that narrow case). Hook-based capture for both.
The eval framework runs at ~$0.01 per evaluation. We'll re-run it as memory systems evolve and as our Cognee graph grows with more production data.
Production Implementation
Benchmark scores only matter if the system is fast enough to use in production. Our initial Cognee implementation added 3.5 to 5 seconds of latency per message. Every turn spawned a new Python subprocess, loaded the Cognee libraries, ran the vector search, traversed the graph, and returned results. That's fine for a benchmark. It's not fine when someone is waiting for a response.
The fix: a persistent HTTP server that keeps Cognee warm in memory. One process, always running, accepting search requests over localhost. No subprocess startup overhead. Vector caches stay hot between queries.
Results after optimization: 1.5 to 2.5 seconds per recall, with warm cache queries hitting as low as 1.2 seconds. That's a 50% reduction in latency without any loss in retrieval quality.
We also discovered that LanceDB (the default vector store) leaks file descriptors across repeated searches. After roughly 15 to 20 queries, the server would exhaust its file handle pool and crash. The production fix: auto-restart after 200 queries, with immediate restart on file descriptor exhaustion. The service manager (launchd on macOS, systemd on Linux) handles recovery automatically. Long term, migrating to Postgres-backed vector storage eliminates this entirely.
None of this showed up in our benchmark. The benchmark ran each query in isolation with a fresh process. Production revealed the real engineering work: managing state across thousands of queries, keeping latency acceptable, and handling the failure modes that only appear under sustained load.
Methodology
Transparency matters. Here's exactly how these numbers were produced.
50 questions, 5 categories, 10 questions each. Categories: preference memory, single-hop facts, multi-hop reasoning, temporal recall, and active context awareness.
30 days of workspace history including daily logs, topic files, configuration docs, and session records. Real operational data from a production AI assistant deployment.
Retrieve from memory system, have GPT-4.1-nano answer the question using only retrieved context, then judge the answer against ground truth. This tests the complete flow from retrieval to comprehension.
OpenClaw framework, all systems running locally. Cost per run: approximately $0.01. Each system received the same corpus of workspace files where applicable.
ByteRover and KG pgvector were tested with an earlier retrieval-only judge. All other systems used the full pipeline described above. The eval framework itself is the real deliverable. Memory systems will keep evolving. Having a cheap, fast, reproducible way to compare them means we can re-run this benchmark whenever a new contender shows up.
Bottom Line
If you're building with AI agents and memory is a pain point, build your own eval first. It takes a day. It costs almost nothing. And it'll save you from picking a memory system based on marketing claims.
The pattern that works: two hooks, a knowledge graph backend, and a benchmark you trust. Everything else is noise.
Benchmarks conducted March 2026 on local infrastructure running OpenClaw. 50-question evaluation framework across 5 categories (preference memory, single-hop facts, multi-hop reasoning, temporal recall, active context). Ground truth sourced from 30 days of production workspace history. Full pipeline methodology: retrieve from memory system, GPT-4.1-nano answers from retrieved context, automated judge scores against ground truth. Systems tested: QMD (OpenClaw built-in), Cognee v0.1 (knowledge graph, ECL pipeline, 49 files ingested), Cognee + hook-based architecture, ByteRover (hierarchical context tree), Postgres/pgvector (76-node knowledge graph), Cipher (MCP-based, DNF). Cost per evaluation run: ~$0.01. Zero hallucinations observed across all systems tested.
We use cookies to improve your experience. Cookie Policy · Privacy Policy