Benchmark · October 2026
DolphinBench feeds three simulated people’s histories, 13,500 messages over three years, into a memory system, then asks an assistant to complete 600 tasks that depend on remembering them. We ran it on Ruune with the benchmark’s public runner.
77.5%
465 of 600 tests passed. The best official configuration, Mem0 on Hermes, scores 70.7%.
Download the evidence archiveSee the official leaderboard at dolphinbench.ai →17 MB · every transcript, tool call and judge verdict, in DolphinBench’s own format
Six of the 465, chosen for how far back the fact sits and how little the question gives away. Quotes are from the evidence files; the people are DolphinBench’s simulated personas.
“Text Jamie with the booking rule we agreed on during the flight home from Tokyo, applied to the refundable cabin for December 27 through January 2.”
Ruune remembered
“On the flight home we said the obvious thing out loud: protected dates only work if we name them and then treat them as actual commitments.”
The assistant did
Texted Jamie: “For Dec 27–Jan 2, let’s apply our flight-home rule to the refundable cabin: treat those dates as real, protected commitments and route work/coverage around them—not as soft placeholders. If we can’t genuinely protect the dates, we shouldn’t book it.”
Judged against
Supports protecting the named dates and arranging ordinary work or coverage around them. Making the dates contingent on normal workload fails.
4 searches, then read the full note · test morgan/075
“Create a private note that says which Marcus “Friday Marcus call” means and distinguishes him from the Marcus in my work context.”
Ruune remembered
“Friday Marcus call in a sec -- brother Marcus Tanaka, not Marcus Vail lol.”
The assistant did
Wrote the note: “‘Friday Marcus call’ means Marcus Tanaka, my brother—not Marcus Vail, my work colleague.”
Judged against
Names both identities and their relationship to the user, and nothing else.
1 search, then the person page for Marcus Tanaka · test riley/069
“Email Devon why we passed on Drift Capital after evaluating its terms.”
Ruune remembered
“wrote the Drift Capital thing down so i stop re-litigating it. four discovery calls was enough. passed. board observer language was too grabby, the IP carve-out stuff got weird, and i really don’t want a tranche tied to MAU.”
The assistant did
Emailed Devon: after four discovery calls and a review of the terms, the board observer language felt too intrusive, the IP carve-out terms raised concerns, and a tranche tied to MAU wasn’t acceptable.
Judged against
Describes an evaluated-and-passed decision based on overreaching board-observer language, problematic IP carve-outs, and an MAU-linked tranche.
6 searches across mail, recordings and documents · test morgan/118
“Elena Volkov’s CRM record was recreated with only her name. Restore my last recorded follow-up status and the reason for it.”
Ruune remembered
“Elena Volkov sent the no from the VC firm. passed on the round, but not boilerplate at all … her main push was the retention funnel.”
“Elena Volkov is first because she is warm context and gave the useful retention feedback earlier.”
“I’ll reconnect in January if the package is cleaner.”
The assistant did
Set the record to “warm later” with the reason: she passed over the retention funnel but her feedback was thoughtful; keep warm, reconnect when the package can address uneven expansion.
Judged against
Keeps a warm/later follow-up, not an active process. Explains that her pass questioned the retention-funnel shape. Does not invent a new investment commitment.
12 calls: searches, the person page, a timeline of late 2023, and three full sources read end to end · test morgan/104
“Privately thank the two teammates I credited in my old team note about the pricing-page rollback. Send each a separate, concise Slack DM acknowledging their specific contribution.”
Ruune remembered
“put the team note in Slack … gave Owen the nod for the cut and Ines for catching the email QA stuff before it turned into a mess.”
The assistant did
Two DMs. Owen: thanks for pulling the cohort cut for the pricing-page rollback. Ines: thank you for catching the email QA issues early and keeping them from turning into a bigger mess.
Judged against
Six checks: both DMs sent, to the right two people, each crediting that person’s specific contribution, without a broad experiment recap.
6 searches and two full reads of the rollback thread · test riley/157
“Create a short historical timeline document recording the date, time, and timezone when the lead firm formally passed on the 2023 B-round pitch.”
Ruune remembered
“Just got the lead firm’s formal pass email. 12:45pm on Feb 16. So yeah. B-round pitch fell through.”
The assistant did
Saved a timeline document: “February 16, 2023 — 12:45 p.m. PST (UTC−08:00): received the lead firm’s formal pass by email.”
Judged against
Identifies the formal pass as February 16, 2023 at 12:45 PM Pacific, or the equivalent instant, and attributes it to no other event.
4 searches, then read the full entry · test morgan/137
| Memory | Harness | Model | Pass rate | Total cost | Median latency |
|---|---|---|---|---|---|
| Ruune | Custom (tools loop) | GPT Luna | 77.5% | $232 | 36 s |
| Mem0 | Hermes | gpt-5.6-luna | 70.7% | $96 | 38 s |
| Hindsight | Hermes | gpt-5.6-luna | 69.5% | $85 | 55 s |
| Honcho | Hermes | gpt-5.6-luna | 68.5% | $143 | 45 s |
| Built-in | Hermes | gpt-5.6-luna | 65.7% | $61 | 44 s |
| Supermemory | Hermes | gpt-5.6-luna | 59.2% | $346 | 53 s |
| Mem0 | Hermes | MiniMax M3 | 47.8% | $148 | 47 s |
| Honcho | Hermes | MiniMax M3 | 47.3% | $212 | 80 s |
| Hindsight | Hermes | MiniMax M3 | 41.0% | $158 | 144 s |
| Honcho | Claude Code | Claude Sonnet 5 | 35.8% | $1,566 | 38 s |
| Mem0 | Claude Code | Claude Sonnet 5 | 32.3% | $1,831 | 42 s |
| Built-in | Hermes | MiniMax M3 | 26.5% | $107 | 51 s |
| Built-in | Claude Code | Claude Sonnet 5 | 26.3% | $1,133 | 32 s |
| Supermemory | Hermes | MiniMax M3 | 20.0% | $498 | 138 s |
Total cost is memory build plus the 600 tests. Latency is the median time per task.
memory_* tools over MCP, the same tools any MCP client gets. No write tools were offered.The benchmark, dataset and paper are at dolphinbench.ai.