I started building memory systems for myself early this year, and that expanded into other areas of AI infrastructure. I pretty much exclusively engineered solutions to solve my own problems, and the results turned into quality systems that started to hold up more and more. Somewhere in there, when I was trying to improve my agent's memory, I ran more memory benchmarks than any one person should have to sit through, and I realized how easy the numbers are to move. Pick a friendlier k, a smaller pool, a kinder judge model or a stronger answer model, and the same system suddenly looks a lot better. What annoyed me the most was that oftentimes that wasn't disclosed. That, and there's a lot of chest-thumping over numbers that don't really mean that much, yet some still define their worth by the high number.
So I decided I wanted to build a benchmark that didn't have variables to tweak, had no LLM involved in the scoring, and was deterministic at its core. After months of putting it off, I finally finished Parsica Recall. It only asks one thing: when an agent needs a memory, does the memory system put the right record in front of it?
My rules:
- The judge never calls a model. Every score is an id match, a phrase match or a token overlap, so two people running the same inputs get the same bytes. `--verify` runs the whole thing twice and compares the hashes.
- You don't get to pick k. The window comes from the size of the pool, and the chance line (what a random retriever would score) is printed next to every number.
- Every run writes out each question's full window with ids, ranks and scores, so any number can be re-derived.
- It doesn't publish anyone's scores, mine included. The only numbers in the repo are calibration, plain BM25 and an oracle, so you can see where the floor and the ceiling are. BM25 gets 87.58 gold@15 on the current set, so that's the floor to beat. I don't really think comparing benchmark scores is a good indicator that a system is actually useful and practical, so I've never felt the need to advertise numbers for my own systems.
The coolest part of this, and the thing I wanted the most, is that you can build a benchmark out of your own agent's memories, doc2query style. It reads an export of your store and writes the questions an agent would later ask about those records, with the right records pinned by id. That’s the one step that uses a model. It works with Anthropic, OpenAI or Google using your own API key, or any endpoint that speaks OpenAI’s chat format, including a local model, and it gives you a cost estimate first and won’t do a live run without a spending cap. Scoring never touches a model, and the benchmark itself runs fully offline. Fair warning though, building a set sends a sample of your records to whichever API you pick (unless it’s a local model), so read the doc2query guide guide before you point it at anything sensitive.
It should work with pretty much any memory system. You can hand it a results file from anything in any language, write a small Python adapter, or run it read-only against a live store.
pip install parsica-recall
parsica-recall run --adapter bm25 --verify
It's MIT, standard library only, Python 3.11+. It's also a beta, and it stays a beta until it's actually been used and peer reviewed, which is a big part of why I'm posting it here. If the methodology is wrong somewhere, I really want to know. Not in this release yet: the bigger standard tracks, scoring against the sealed half of the questions, and a full-stack mode where an answer model reads the window. But please let me know if you have thoughts or ideas. I would love to collaborate or take feedback. As far as I know, nothing like this really exists, so it's important to me that it eventually gets a green light from other users in the community.
https://github.com/Antaris-Analytics-LLC/Parsica-Recall