memfor
GitHub
Display
Theme
Motion
memfor
Theme
Motion 
git clone https://github.com/r5rana/agentware.git
cd agentware
git checkout 3c25cdd
claude "hi"
git clone https://github.com/r5rana/agentware.git
cd agentware
git checkout 3c25cdd
codex "hi"

This installs agentware, the open-source engine behind memfor. No tagged release yet. These commands pin commit 3c25cdd.

Apache-2.0. No memfor account. Needs macOS or Linux (Windows via WSL), git, Python 3, bash, jq, and Claude Code or Codex.

Send setup steps to my computer Join the Teams waitlist

Benchmark ledger. Every run is appended, never edited. Every number on this site links to its row.

Last LongMemEval-S run: 25 June 2026. Checked 24 October 2026. Runs aren't automated yet

Reproduce this number View the run Download the ledger (JSONL)
Headline receipt Ledger row 86
Measurand
Recall@5 (retrieval)
Definition
For each question, the share of its evidence sessions that appear in the top 5 results, averaged over 470 answerable questions. Retrieval, not answer accuracy.
Dataset
LongMemEval-S (cleaned), Hugging Face commit 98d7416c…d48f, sha256 d6f21ea9…a442. Licence: MIT.
Protocol
Session granularity. 470 answerable questions; the 30 abstention questions are excluded from Recall and reported separately. BM25, top 5, default mode.
Environment
Hardware, OS and Python version: not recorded for this run. The next clean run will record them.
Result
Recall@5 0.9140
nDCG@5 0.8831
MRR 0.9104
measured
Run
Commit 42d58a0, clean tree, recorded 25 June 2026, 15:34:44 UTC. Determinism check passed; 332 tests ran, 0 failed.
Ledger · 137 runs at 3c25cdd, one slab per row
data · One slab per ledger row, oldest at the bottom. Width shows the suite (LongMemEval full, own gold set ¾), hollow slabs are dirty-tree runs, and the lit slab is the run this site cites · benchmarks/history.jsonl at 3c25cdd

Reproduce it from a clean clone.

Run this at commit 42d58a0, the commit that recorded the cited row. The install commands pin 3c25cdd, a later commit where we haven't re-run this benchmark yet (gate G4 records a row there; until then the receipt says so).

Prerequisites (ours, not from GUIDE.md) 
# Use a new terminal window: it points HOME at a scratch folder,
# so your own agentware setup and knowledge base are untouched.
git clone https://github.com/r5rana/agentware.git
cd agentware
git checkout 42d58a0
SCRATCH="$(mktemp -d)"
export HOME="$SCRATCH/home" AGENTWARE_KNOWLEDGE_DIR="$SCRATCH/kb"
mkdir -p "$HOME"
scripts/agentware init --knowledge-dir "$AGENTWARE_KNOWLEDGE_DIR"
KDIR="$(scripts/agentware config --knowledge-dir-only)"
From docs/GUIDE.md at 42d58a0, verbatim 
# 1. Fetch the cleaned variant into your knowledge dir, pinned to an exact commit.
#    KDIR="$(scripts/agentware config --knowledge-dir-only)"
mkdir -p "$KDIR/benchmarks/longmemeval" && cd "$KDIR/benchmarks/longmemeval"
curl -sS -L -o longmemeval_s_cleaned.json \
  "https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/98d7416c24c778c2fee6e6f3006e7a073259d48f/longmemeval_s_cleaned.json"

# 2. Verify the bytes (MUST match, or the comparison is not apples-to-apples).
shasum -a 256 longmemeval_s_cleaned.json
#   → d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442

# 3. Run the benchmark (pure stdlib — nothing else to install for Mode A).
cd -  # back to the agentware repo
scripts/agentware eval --suite longmemeval --strategy bm25 --top-k 5 --as-of 2026-06-25
#   → Recall@5 0.9140 · nDCG@5 0.8831 · MRR 0.9104 · 470 answerable · 30 abstention separate

The dataset is about 277 MB. No model sits in the scoring path, so at 42d58a0 the metrics should match the ledger row to four decimal places; latency varies by machine.

Latency.

p50 10.5 ms per query on this run. Hardware wasn't recorded, so this number isn't comparable across machines. We'll publish a hardware-stamped run.

The ledger.

At the pin, 135 of the 137 rows (suite: null) are own-gold-set runs against the maintainer's private knowledge base, whose size drifted from 181 to 228 entries across those rows.

Showing 137 of 137 runs.

The ledger
Run date Suite Strategy k Queries Recall@k nDCG@k MRR Tree Commit Receipt
Own gold set (private corpus) tag 5 56 0.6518 0.5754 0.5753 dirty tree · not cited 5b9e4b4 Row 1
Own gold set (private corpus) bm25 5 56 0.9821 0.9440 0.9440 dirty tree · not cited 5b9e4b4 Row 2
Own gold set (private corpus) bm25 5 56 0.9821 0.9440 0.9440 dirty tree · not cited 5b9e4b4 Row 3
Own gold set (private corpus) bm25 5 56 0.9732 0.9437 0.9464 dirty tree · not cited 5b9e4b4 Row 4
Own gold set (private corpus) bm25 5 56 0.9732 0.9437 0.9464 dirty tree · not cited 5b9e4b4 Row 5
Own gold set (private corpus) bm25 5 56 0.9732 0.9437 0.9464 dirty tree · not cited 5b9e4b4 Row 6
Own gold set (private corpus) bm25 5 56 0.9732 0.9437 0.9464 dirty tree · not cited 5b9e4b4 Row 7
Own gold set (private corpus) bm25 5 56 0.9732 0.9437 0.9464 dirty tree · not cited 5b9e4b4 Row 8
Own gold set (private corpus) bm25 5 56 0.9732 0.9437 0.9464 dirty tree · not cited 5b9e4b4 Row 9
Own gold set (private corpus) bm25 5 56 0.9911 0.9506 0.9470 dirty tree · not cited 5b9e4b4 Row 10
Own gold set (private corpus) bm25 5 56 0.9911 0.9506 0.9470 dirty tree · not cited 5b9e4b4 Row 11
Own gold set (private corpus) bm25 5 56 0.9911 0.9506 0.9470 dirty tree · not cited 5b9e4b4 Row 12
Own gold set (private corpus) bm25 5 56 0.9911 0.9506 0.9470 dirty tree · not cited 5b9e4b4 Row 13
Own gold set (private corpus) bm25 5 56 0.9911 0.9506 0.9470 dirty tree · not cited 5b9e4b4 Row 14
Own gold set (private corpus) bm25 5 56 0.9911 0.9506 0.9470 dirty tree · not cited 5b9e4b4 Row 15
Own gold set (private corpus) bm25 5 56 0.9911 0.9506 0.9470 dirty tree · not cited 5b9e4b4 Row 16
Own gold set (private corpus) bm25 5 56 0.9911 0.9506 0.9470 dirty tree · not cited 5b9e4b4 Row 17
Own gold set (private corpus) tag 5 56 0.6518 0.5742 0.5738 dirty tree · not cited 5b9e4b4 Row 18
Own gold set (private corpus) tag 5 56 0.6518 0.5742 0.5738 dirty tree · not cited 5b9e4b4 Row 19
Own gold set (private corpus) bm25 5 56 0.9911 0.9506 0.9470 dirty tree · not cited 5b9e4b4 Row 20
Own gold set (private corpus) bm25 5 56 0.9911 0.9470 0.9426 dirty tree · not cited 5b9e4b4 Row 21
Own gold set (private corpus) bm25 5 56 0.9911 0.9470 0.9426 dirty tree · not cited 5b9e4b4 Row 22
Own gold set (private corpus) bm25 5 56 0.9911 0.9470 0.9426 dirty tree · not cited 5b9e4b4 Row 23
Own gold set (private corpus) bm25 5 56 0.9911 0.9470 0.9426 clean tree 2f2c106 Row 24
Own gold set (private corpus) bm25 5 56 0.9911 0.9477 0.9446 dirty tree · not cited 2f2c106 Row 25
Own gold set (private corpus) bm25 5 56 0.9911 0.9477 0.9446 dirty tree · not cited 2f2c106 Row 26
Own gold set (private corpus) bm25 5 56 0.9911 0.9477 0.9446 dirty tree · not cited f5d0535 Row 27
Own gold set (private corpus) bm25 5 56 0.9911 0.9477 0.9446 dirty tree · not cited f5d0535 Row 28
Own gold set (private corpus) bm25 5 56 0.9911 0.9477 0.9446 dirty tree · not cited f5d0535 Row 29
Own gold set (private corpus) bm25 5 56 0.9911 0.9477 0.9446 dirty tree · not cited f5d0535 Row 30
Own gold set (private corpus) bm25 5 56 0.9821 0.9428 0.9426 dirty tree · not cited f5d0535 Row 31
Own gold set (private corpus) bm25 5 56 0.9821 0.9428 0.9426 dirty tree · not cited f5d0535 Row 32
Own gold set (private corpus) bm25 5 56 0.9821 0.9428 0.9426 dirty tree · not cited acdd14f Row 33
Own gold set (private corpus) bm25 5 56 0.9821 0.9428 0.9426 dirty tree · not cited acdd14f Row 34
Own gold set (private corpus) bm25 5 56 0.9911 0.9470 0.9426 dirty tree · not cited acdd14f Row 35
Own gold set (private corpus) bm25 5 56 0.9911 0.9470 0.9426 dirty tree · not cited acdd14f Row 36
Own gold set (private corpus) bm25 5 56 0.9911 0.9470 0.9426 dirty tree · not cited acdd14f Row 37
Own gold set (private corpus) bm25 5 56 0.9911 0.9470 0.9426 dirty tree · not cited acdd14f Row 38
Own gold set (private corpus) bm25 5 56 0.9911 0.9470 0.9426 dirty tree · not cited acdd14f Row 39
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited acdd14f Row 40
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited acdd14f Row 41
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited acdd14f Row 42
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited acdd14f Row 43
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited acdd14f Row 44
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited acdd14f Row 45
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited acdd14f Row 46
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited acdd14f Row 47
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited acdd14f Row 48
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited acdd14f Row 49
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited acdd14f Row 50
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited acdd14f Row 51
Own gold set (private corpus) tag 5 56 0.6518 0.5668 0.5640 dirty tree · not cited acdd14f Row 52
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited acdd14f Row 53
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited 35190d5 Row 54
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited 35190d5 Row 55
Own gold set (private corpus) bm25 5 56 1.0000 0.9513 0.9426 dirty tree · not cited 35190d5 Row 56
Own gold set (private corpus) bm25 5 56 1.0000 0.9513 0.9426 dirty tree · not cited 35190d5 Row 57
Own gold set (private corpus) bm25 5 56 1.0000 0.9513 0.9426 dirty tree · not cited 35190d5 Row 58
Own gold set (private corpus) bm25 5 56 1.0000 0.9513 0.9426 dirty tree · not cited 35190d5 Row 59
Own gold set (private corpus) bm25 5 56 1.0000 0.9513 0.9426 dirty tree · not cited 35190d5 Row 60
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited 35190d5 Row 61
Own gold set (private corpus) bm25 5 56 1.0000 0.9517 0.9426 dirty tree · not cited 35190d5 Row 62
Own gold set (private corpus) bm25 5 56 1.0000 0.9530 0.9440 dirty tree · not cited 42c8822 Row 63
Own gold set (private corpus) bm25 5 56 1.0000 0.9530 0.9440 dirty tree · not cited 42c8822 Row 64
Own gold set (private corpus) tag 5 56 0.6518 0.5579 0.5521 dirty tree · not cited 42c8822 Row 65
Own gold set (private corpus) bm25 5 56 1.0000 0.9530 0.9440 dirty tree · not cited 42c8822 Row 66
Own gold set (private corpus) bm25 5 56 1.0000 0.9530 0.9440 dirty tree · not cited 42c8822 Row 67
Own gold set (private corpus) bm25 5 56 1.0000 0.9530 0.9440 dirty tree · not cited 42c8822 Row 68
Own gold set (private corpus) bm25 5 56 1.0000 0.9530 0.9440 dirty tree · not cited 42c8822 Row 69
Own gold set (private corpus) bm25 5 56 1.0000 0.9530 0.9440 dirty tree · not cited 42c8822 Row 70
LongMemEval-S bm25 5 470 0.9140 0.8831 0.9104 dirty tree · not cited e8cb9c5 Row 71
Own gold set (private corpus) bm25 5 56 0.9732 0.9354 0.9353 dirty tree · not cited e8cb9c5 Row 72
Own gold set (private corpus) bm25 5 56 0.9732 0.9354 0.9353 dirty tree · not cited e8cb9c5 Row 73
Own gold set (private corpus) bm25 5 56 0.9821 0.9389 0.9344 dirty tree · not cited e8cb9c5 Row 74
Own gold set (private corpus) bm25 5 56 0.9821 0.9389 0.9344 dirty tree · not cited e8cb9c5 Row 75
Own gold set (private corpus) bm25 5 56 0.9821 0.9455 0.9433 dirty tree · not cited e8cb9c5 Row 76
Own gold set (private corpus) bm25 5 56 0.9821 0.9455 0.9433 dirty tree · not cited e8cb9c5 Row 77
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9393 dirty tree · not cited e8cb9c5 Row 78
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9393 dirty tree · not cited e8cb9c5 Row 79
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9393 dirty tree · not cited e8cb9c5 Row 80
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9389 dirty tree · not cited e8cb9c5 Row 81
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9389 dirty tree · not cited e8cb9c5 Row 82
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9391 dirty tree · not cited e8cb9c5 Row 83
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9391 dirty tree · not cited e8cb9c5 Row 84
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9391 dirty tree · not cited e8cb9c5 Row 85
LongMemEval-S bm25 5 470 0.9140 0.8831 0.9104 clean tree 42d58a0 Row 86 cited on this site
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9389 dirty tree · not cited 42d58a0 Row 87
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9389 dirty tree · not cited 42d58a0 Row 88
Own gold set (private corpus) tag 5 56 0.6161 0.5244 0.5251 dirty tree · not cited ef10608 Row 89
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9389 dirty tree · not cited ef10608 Row 90
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9389 dirty tree · not cited ef10608 Row 91
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9389 dirty tree · not cited ef10608 Row 92
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9389 dirty tree · not cited ef10608 Row 93
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9389 dirty tree · not cited ef10608 Row 94
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9389 dirty tree · not cited ef10608 Row 95
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9389 dirty tree · not cited ef10608 Row 96
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9395 dirty tree · not cited ef10608 Row 97
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9395 dirty tree · not cited ef10608 Row 98
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9391 dirty tree · not cited d4de4b7 Row 99
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9391 dirty tree · not cited d4de4b7 Row 100
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9391 dirty tree · not cited d4de4b7 Row 101
Own gold set (private corpus) bm25 5 56 0.9554 0.9329 0.9391 clean tree ae27258 Row 102
Own gold set (private corpus) tag 5 56 0.5625 0.4788 0.4858 dirty tree · not cited ae27258 Row 103
Own gold set (private corpus) tag 5 56 0.5625 0.4788 0.4858 dirty tree · not cited ae27258 Row 104
Own gold set (private corpus) tag 5 56 0.5625 0.4788 0.4850 dirty tree · not cited ae27258 Row 105
Own gold set (private corpus) tag 5 56 0.5625 0.4788 0.4850 dirty tree · not cited ae27258 Row 106
Own gold set (private corpus) tag 5 56 0.5625 0.4788 0.4850 dirty tree · not cited ae27258 Row 107
Own gold set (private corpus) tag 5 56 0.5446 0.4719 0.4844 dirty tree · not cited ae27258 Row 108
Own gold set (private corpus) tag 5 56 0.5446 0.4719 0.4844 dirty tree · not cited ae27258 Row 109
Own gold set (private corpus) tag 5 56 0.5446 0.4719 0.4844 dirty tree · not cited ae27258 Row 110
Own gold set (private corpus) bm25 5 56 0.9554 0.9351 0.9425 dirty tree · not cited 7bdbbc6 Row 111
Own gold set (private corpus) bm25 5 56 0.9554 0.9351 0.9425 dirty tree · not cited 7bdbbc6 Row 112
Own gold set (private corpus) tag 5 56 0.5446 0.4719 0.4844 dirty tree · not cited 7bdbbc6 Row 113
Own gold set (private corpus) bm25 5 56 0.9554 0.9351 0.9425 dirty tree · not cited 7bdbbc6 Row 114
Own gold set (private corpus) bm25 5 56 0.9554 0.9351 0.9425 dirty tree · not cited 8084cc0 Row 115
Own gold set (private corpus) bm25 5 56 0.9554 0.9351 0.9425 dirty tree · not cited 8084cc0 Row 116
Own gold set (private corpus) tag 5 56 0.5446 0.4719 0.4844 dirty tree · not cited 8084cc0 Row 117
Own gold set (private corpus) bm25 5 56 0.9554 0.9351 0.9425 dirty tree · not cited a52bf15 Row 118
Own gold set (private corpus) tag 5 56 0.5446 0.4719 0.4844 dirty tree · not cited a52bf15 Row 119
Own gold set (private corpus) tag 5 56 0.5625 0.4854 0.4951 dirty tree · not cited 9f02a47 Row 120
Own gold set (private corpus) tag 5 56 0.5625 0.4854 0.4951 dirty tree · not cited 9f02a47 Row 121
Own gold set (private corpus) bm25 5 56 0.9554 0.9285 0.9333 dirty tree · not cited 9f02a47 Row 122
Own gold set (private corpus) bm25 5 56 0.9554 0.9285 0.9333 dirty tree · not cited 9f02a47 Row 123
Own gold set (private corpus) tag 5 56 0.5625 0.4854 0.4951 dirty tree · not cited 996aa54 Row 124
Own gold set (private corpus) tag 5 56 0.5625 0.4854 0.4951 dirty tree · not cited 996aa54 Row 125
Own gold set (private corpus) bm25 5 56 0.9554 0.9285 0.9333 dirty tree · not cited cf93c14 Row 126
Own gold set (private corpus) tag 5 56 0.5625 0.4854 0.4951 dirty tree · not cited cf93c14 Row 127
Own gold set (private corpus) tag 5 56 0.5625 0.4854 0.4951 dirty tree · not cited 9c5091f Row 128
Own gold set (private corpus) bm25 5 56 0.9554 0.9285 0.9333 dirty tree · not cited 9c5091f Row 129
Own gold set (private corpus) tag 5 56 0.5625 0.4854 0.4951 dirty tree · not cited 9c5091f Row 130
Own gold set (private corpus) bm25 5 56 0.9554 0.9285 0.9333 clean tree 820f682 Row 131
Own gold set (private corpus) tag 5 56 0.5625 0.4854 0.4951 clean tree 820f682 Row 132
Own gold set (private corpus) bm25 5 56 0.9554 0.9285 0.9333 dirty tree · not cited 820f682 Row 133
Own gold set (private corpus) tag 5 56 0.5625 0.4854 0.4946 dirty tree · not cited 820f682 Row 134
Own gold set (private corpus) tag 5 56 0.5625 0.4854 0.4946 clean tree 73068cb Row 135
Own gold set (private corpus) bm25 5 56 0.9554 0.9285 0.9329 clean tree 73068cb Row 136
Own gold set (private corpus) bm25 5 56 0.9554 0.9285 0.9326 clean tree 71edb3f Row 137

Our own gold set (private corpus, not comparable).

0.9554 Recall@5 on 56 questions written against the maintainer's private knowledge base (195 entries at commit 71edb3f). The questions and the corpus are not public, so this number can't be reproduced outside that knowledge base. The questions also share vocabulary with the corpus, which flatters BM25. Don't compare it with anything.

Reported by others: different definitions.

Reported by others: different definitions
System What they report Metric Questions Source
agentmemory 95.2% hybrid, 86.2% BM25-only recall_any@5 500 benchmark/LONGMEMEVAL.md checked 23 October 2026
MemPalace 96.6% raw reportedly recall_any@5 500 benchmarks/README.md checked 23 October 2026

Note: recall_any@5 counts a question as found if any one of its evidence sessions is in the top 5. Our Recall@5 credits each evidence session, so on the same run it can't be higher than recall_any@5. We'll publish our recall_any@5 and re-run both systems through one scorer.

Different metric, not comparable.

  • Zep reports 90.2% LongMemEval answer accuracy (gpt-5.4 as reader and judge).

    getzep.com/research checked 23 October 2026

  • Mem0's site shows 94.4 for LongMemEval without naming the metric.

    mem0.ai/research checked 23 October 2026

  • supermemory reports 97% judged answer accuracy with a Recall@20 retrieval setting.

    supermemory.ai/research/longmembench checked 23 October 2026

  • Hindsight (Vectorize) reports 91.4% on LongMemEval in its paper: the same digits as our Recall@5, but it's answer accuracy with a larger model backbone, not retrieval (https://arxiv.org/abs/2512.12818).

    checked 23 October 2026

Found a problem?

Dispute a number . Corrections are logged in the changelog.

Methodology.

  • Fractional Recall@k = mean over questions of |evidence ∩ top-k| / |evidence|. recall_at_k in scripts/agentware at the pin.
  • nDCG@k with binary relevance.
  • MRR = mean of 1/rank of the first evidence session.
  • Precision@k = mean over questions of |evidence ∩ top-k| / k. It can't reach 1 when a question has fewer than k evidence sessions, so it is shown on receipts only.
  • reliability (a field in every ledger row) is an internal composite health score from 0 to 100, defined in GUIDE.md. It is not a benchmark metric, and it is never shown outside the raw JSON.
  • Abstention questions are excluded from Recall.
  • Determinism: the scorer re-scores a sample inside each run (determinism_ok), and a test asserts identical recall output across runs. We claim identical rankings and metrics, not identical timing.
  • Ledger schema agentware-bench/v1. Fields: ablation, acr_gate, as_of, checks, commit, committed, corpus_fingerprint, corpus_size, dirty, gold_path, metrics, mode, num_queries, reliability, run, schema, strategy, subject, suite, top_k.
  • Methodology changes are logged in the changelog.
Esc