Extractive v19 · adaptive high recall
14.78M total
9,598tokens / question
- Answering
- 11.43M
- Evaluation
- 3.35M
Back to research
Benchmark report 01
August 26, 20268 min readUpdated September 4, 2026Measuring long-term conversational memory on LoCoMo
A transparent end-to-end evaluation of Copepod's ability to retrieve and answer questions from long, multi-session conversations.
Key result
92.34%
latest core-run answer accuracy
1,422 of 1,540 answers were judged correct in the latest published core run.
What this result means
This is an end-to-end result: Copepod selected evidence, an answer model produced the response, and a separate judge evaluated it. It measures the complete system path, not retrieval alone.
How we evaluate
We measure the complete system and state the limits of each result.
We evaluate Copepod as a complete system: an answering model working with a durable memory layer. We do not copy another company's architecture or tune our work to match a competitor's reported score. We build the memory system we believe agents need, then measure whether it improves over time.
Quality and efficiency matter together. We report answer and evaluation tokens alongside accuracy because a memory system should improve useful context without relying on an ever larger prompt.
Each published score includes its protocol, category breakdown and limitations. Only complete benchmark runs support headline claims. Smaller tests and retrieval diagnostics remain supporting evidence.
Core-run history
How results changed across runs
Each point is a complete 1,540-question end-to-end run. Smaller diagnostic tests are not included.
Overall answer accuracy
All runs use the same 1,540 questions. Read protocol changes alongside the chart.
| Run | Overall | Multi-hop | Temporal | Open-domain | Single-hop |
|---|---|---|---|---|---|
Extractive v19 · adaptive high recall Historical baseline | 90.32% 1391 / 1540 | 89.72% 253 / 282 | 88.47% 284 / 321 | 68.75% 66 / 96 | 93.70% 788 / 841 |
Candidate auto · adaptive midpoint August 25, 2026 | 91.43% 1408 / 1540 | 92.20% 260 / 282 | 91.59% 294 / 321 | 65.63% 63 / 96 | 94.05% 791 / 841 |
Temporal-win checkpoint · adaptive midpoint August 26, 2026 | 92.34% 1422 / 1540 | 92.91% 262 / 282 | 93.15% 299 / 321 | 70.83% 68 / 96 | 94.29% 793 / 841 |
Token use
Tokens per question
Totals include full answer generation and evaluation across all 1,540 questions. Category totals are not tokenised separately.
Candidate auto · adaptive midpoint
13.19M total
8,564tokens / question
- Answering
- 9.84M
- Evaluation
- 3.35M
Temporal-win checkpoint · adaptive midpoint
13.73M total
8,918tokens / question
- Answering
- 10.38M
- Evaluation
- 3.35M
Extractive v19 · adaptive high recall
Historical core baseline. Exact frozen artifact paths are unavailable in the current checkout.
Context method: extractive-v19 adaptive-high-recall
Candidate auto · adaptive midpoint
Completed product-visible core run. Its lower open-domain result failed a later explicit gate, so it is reported as an observed core result rather than acceptance evidence.
Context method: candidate-auto-adaptive-midpoint
Temporal-win checkpoint · adaptive midpoint
User-accepted temporal-win checkpoint from a fresh, commit-bound full run. Every LoCoMo category improved versus the historical baseline; the observed delta also includes compiler, context-policy, and stochastic model-call changes.
Context method: candidate-2b38545 auto-adaptive-midpoint; bounded product-visible supporting sessions
Scorecard
What we measured and how to read it
92.34% · 1,422 / 1,540
Answer accuracy
The proportion of benchmark questions whose final answer was judged correct.
How to use this measure
Use this to compare complete question-answering performance when the benchmark version, answer model, judge, and prompt protocol match.
This measure does not show which part of the system caused an incorrect answer.
4 LoCoMo categories
Category breakdown
Multi-hop, temporal, open-domain, and single-hop scores are reported separately for every published core run.
How to use this measure
Use it to see where an improvement occurred instead of treating the overall result as a single undifferentiated number.
Changes to the context policy affect several parts of the system, so category differences do not identify a single cause.
8,918 tokens / question
Token efficiency
The accepted core run used 13.73 million answer-and-judge tokens across all 1,540 questions.
How to use this measure
Use it to compare evaluation cost when the denominator, answer model, judge, and protocol are held constant.
This includes evaluation judging and is not a production inference-cost forecast.
Run-specific
Latency
Time spent executing the evaluated path, reported with runtime and isolation conditions.
How to use this measure
Use it for capacity and experience planning only when hardware, tenancy, load, and timing boundaries are comparable.
Latency from a recovered or shared runtime is not an authoritative performance claim.
Run protocol
- Benchmark
- LoCoMo conversational QA
- Questions scored
- 1,540
- System evaluated
- Copepod retrieval + answer generation
- Answering model
- GPT-5.6 Luna
- Evaluation model
- GPT-5.6 Sol
- Measurement type
- End-to-end QA accuracy
What this supports
- How reliably the evaluated system answers this benchmark's questions across long conversations.
- How the end-to-end result changes when the evaluator changes.
- The exact denominator, models, and evaluation boundary behind the claim.
What this result does not show
- A universal ranking across other datasets, versions, or vendor protocols.
- Retrieval-only quality, oracle-context quality, or an isolated model capability unless separately reported.
- Production latency under every tenant, model, hardware, or concurrent workload.