copepodBenchmark report 03
September 4, 20267 min readA controlled LoCoMo study of an answerability gate that starts compact and adds ranked evidence only when the current packet cannot support the answer.
Key result
10.62%
fewer answer-stage provider tokens
The candidate matched raw top-10 quality at 75 of 81 answers while using 213,767 fewer answer-stage tokens.
What this result means
This is a controlled 81-question LoCoMo diagnostic on one conversation cohort. It compares the same system's answerability-gated context against a raw top-10 reference. It proves a quality-versus-token point for this recorded run; it is not a new 1,540-question LoCoMo score and the policy has not been promoted into the production assembly path.
How we evaluate
We evaluate Copepod as a complete system: an answering model working with a durable memory layer. We do not copy another company's architecture or tune our work to match a competitor's reported score. We build the memory system we believe agents need, then measure whether it improves over time.
Quality and efficiency matter together. We report answer and evaluation tokens alongside accuracy because a memory system should improve useful context without relying on an ever larger prompt.
Each published score includes its protocol, category breakdown and limitations. Only complete benchmark runs support headline claims. Smaller tests and retrieval diagnostics remain supporting evidence.
The design
The candidate begins with a compiler capsule and six ranked raw sources. It extends that raw prefix only when source-grounded provenance and coverage diagnostics expose a direct-fact or temporal answerability gap. Eight of 81 questions widened; 73 kept the compact path.
Paired result
10.62%
fewer answer tokens
31.96%
less rendered context
| Context policy | Correct | Accuracy | Answer tokens | Context units |
|---|---|---|---|---|
Raw top-10 reference All ten ranked raw sources were supplied to the answer model. | 75 / 81 | 92.59% | 2,013,565 | 2,629,533 |
Compact top-six pass The compact starting point saved context but missed two answers recovered by bounded tail evidence. | 73 / 81 | 90.12% | 1,792,114 | 1,731,691 |
Answerability-gated tail The gate widened eight questions only when provenance and coverage exposed a bounded evidence gap. | 75 / 81 | 92.59% | 1,799,798 | 1,789,219 |
Multi-hop
11 / 11
100%
Temporal
26 / 26
100%
Single-hop
38 / 44
86.36%
This run made no new enrichment or embedding provider calls. It reused the existing 589,679-token enrichment ledger. Judge usage was recorded separately as 152,272 evaluation-only tokens; it is not mixed into the answer-stage saving.
Scorecard
92.59% · 75 / 81
The answerability-gated pass matched the raw top-10 reference on the controlled cohort.
How to use this measure
Use it to check whether bounded evidence escalation recovered the quality lost by the compact starting packet.
This denominator covers one 81-question conversation cohort, not the full 1,540-question LoCoMo suite.
1,799,798 · −10.62%
Luna input and output tokens for 81 answer calls, compared with 2,013,565 for raw top ten.
How to use this measure
Use it as the clean provider-cost comparison between these two answer paths.
This is answer-stage usage, not memory-context tokens alone and not a comparison with another vendor's accounting.
1,789,219 units · −31.96%
Exact answer-context size measured in the harness's cme-budget-v1 units, versus 2,629,533 for raw top ten.
How to use this measure
Use it to measure how much memory evidence was supplied before answer generation.
CME budget units are bytes-like rendering units, not provider tokens; the 31.96% reduction must not be described as token savings.
8 widened · 73 unchanged
The policy started with a compact capsule and six raw sources, then extended the ranked prefix for eight questions.
How to use this measure
Use it to understand where the token-efficient version spends extra context instead of applying a fixed larger prompt everywhere.
The gate used provenance and coverage signals, never benchmark IDs, categories, gold answers, retrieval-hit labels, or previous judgments.