copepodBenchmark report 02
August 26, 20266 min readA transparent development timeline on a frozen 12-question diagnostic set, used to improve temporal and knowledge-update memory before spending a full 500-question evaluation.
Key result
75.00%
best fixed-12 diagnostic accuracy
9 of 12 answers were judged correct at the accepted diagnostic checkpoint.
What this result means
This is a development diagnostic, not an official LongMemEval score. We have not run the full 500-question suite. We will do that once retrieval and context compilation are stable enough for a reproducible result.
How we evaluate
We evaluate Copepod as a complete system: an answering model working with a durable memory layer. We do not copy another company's architecture or tune our work to match a competitor's reported score. We build the memory system we believe agents need, then measure whether it improves over time.
Quality and efficiency matter together. We report answer and evaluation tokens alongside accuracy because a memory system should improve useful context without relying on an ever larger prompt.
Each published score includes its protocol, category breakdown and limitations. Only complete benchmark runs support headline claims. Smaller tests and retrieval diagnostics remain supporting evidence.
Why this is a diagnostic
We have not run the full 500-question suite. These 12 questions help us check whether the system is ready for a meaningful full evaluation.
Diagnostic history
All points use the same questions and frozen retrieval input. Each run uses new answer and evaluation calls, so small differences may be random.
| Run | Overall | Temporal | Knowledge updates | Tokens / question |
|---|---|---|---|---|
Frozen raw top-six control ControlRaw retrieved sessions supplied without the bounded compiler capsule. This is the fixed diagnostic control, not a product configuration. | 8 / 12 66.67% | 2 / 6 33.33% | 6 / 6 100.00% | 21,916 262,995 total |
Compiler candidate 23dc6b7 Did not meet the checkReduced prompt cost substantially, but regressed one answer against the frozen control. | 7 / 12 58.33% | 3 / 6 50.00% | 4 / 6 66.67% | 6,424 77,092 total |
Compiler candidate ed70543 Did not meet the checkMatched the control overall and improved the temporal slice, but regressed knowledge-update questions. | 8 / 12 66.67% | 5 / 6 83.33% | 3 / 6 50.00% | 9,892 118,709 total |
Compiler candidate eb1d397 Did not meet the checkNarrowing the fallback evidence did not restore the missing knowledge-update information. | 7 / 12 58.33% | 4 / 6 66.67% | 3 / 6 50.00% | 9,806 117,676 total |
Temporal-win checkpoint 2b38545 CheckpointFirst accepted diagnostic uplift: one more correct answer than the control, with a much smaller context envelope. | 9 / 12 75.00% | 4 / 6 66.67% | 5 / 6 83.33% | 9,726 116,709 total |
Scorecard
75.00% · 9 / 12
The best result on the frozen 12-question temporal and knowledge-update diagnostic.
How to use this measure
Use it to decide whether a compiler revision is ready for broader evaluation.
A 12-question diagnostic has high variance and must never be presented as the official 500-question LongMemEval result.
66.67% · 4 / 6
Questions requiring dates, elapsed time, or changing state in the accepted checkpoint.
How to use this measure
Use it to expose temporal evidence and answer-assembly failures before a full run.
An earlier candidate reached 5/6 temporal but regressed knowledge-update performance, so slices are read together.
83.33% · 5 / 6
Questions requiring the system to retain and apply updated information.
How to use this measure
Use it to test whether compressed context preserves superseding facts.
Answer and evaluation model outputs can vary between runs, so a change cannot be attributed to one cause alone.
9,726 tokens / question
Total answer and judge tokens at the accepted checkpoint, versus 21,916 for the raw top-six control.
How to use this measure
Use it to track whether bounded evidence keeps evaluation cost under control.
Token totals cover this exact 12-question diagnostic protocol and should not be projected directly to production traffic.