ZeroMemory Holds 100% Recall on LongMemEval V2 — Now With Open Llama 70B Instead of GPT-4o
When we first published 100% recall on LongMemEval, the pipeline relied on GPT-4o for memory extraction and consolidation. ZeroMemory V2 reproduces the same 100% recall using an open Llama 70B model instead — removing the closed-model dependency without losing accuracy.
V1 vs V2
| V1 | V2 | |
|---|---|---|
| Extraction model | GPT-4o | Llama 70B (open) |
| LongMemEval recall | 100% | 100% |
| Vendor dependency | Yes (OpenAI) | No |
| Cost per consolidation | Higher | Lower |
Why this matters
A memory system that only works with one closed-source model is a fragile thing to build a product on — pricing changes, rate limits, or deprecations upstream become your outage. Proving the same recall holds with an open model means the consolidation architecture is doing the work, not the specific model's idiosyncrasies.
Architecture
The pipeline is unchanged in shape: raw conversation turns flow into an episodic store, get consolidated into semantic summaries on a schedule, and get scored for retention via RLHF feedback. Only the extraction model underneath swapped out.
What's next
We're now evaluating smaller open models in the same slot to see where the accuracy/cost curve bends — the goal is the cheapest model that still holds 100% on LongMemEval, not the biggest one.
Originally published on the AINative Studio blog.