# Syno Benchmark — Evaluation Protocol

*Companion to `syno-litreview-whitepaper.md` §7.*
*Version 2.0-pragmatic — May 2026.*

> **What this version is.** A startup-scale benchmark designed for **2 FTE, ~$3–5k cash budget, 3 weeks wall-clock**, with as much of the grading automated as is defensible. The credibility move is **radical transparency** — every capture, every judge prompt, every source-passage retrieved, every verdict line, every analysis script published under CC-BY. The community can re-grade with their own gold sets, judges, or raters.
>
> **What this version is NOT.** A Cochrane-grade systematic evaluation. The full rigorous protocol — 3-rater IRR pool, 3 reruns per tool, cross-family judge ensemble, institutional library access, 8–14 weeks — is parked as v3.0 in §11 for after this brief is published and we have data on what matters.

> **Pre-registration.** Lock this document on OSF before any system runs. No system runs before the OSF embargo timestamp.

---

## 0. Scope and Honest Limits

This benchmark compares Syno against six AI literature-review tools on twenty diabetes-research questions. The aim is to publish a defensible interim brief that partners can read, not to win a methods-section debate. Where rigor is traded off for execution speed, the trade-off is named in §10 (Limitations) and material enough that we publish all raw data so a critic can re-do the grading their way.

**Tools:** Syno, Elicit, Consensus, Perplexity Deep Research, OpenAI Deep Research, OpenEvidence, Undermind.

**Questions:** 20 diabetes-research questions pre-registered in `diabetes-benchmark-questions.md`. Scope is diabetes only (matches Syno's corpus, §7 of whitepaper).

**Primary endpoint:** per-claim citation faithfulness — for each cited sentence, does the cited paper support the claim along population, numerics, outcome, recency axes.

**Secondary endpoints:** claim recall against canonical-meta-analysis gold sets; synthesis coherence; hallucinated-citation rate; artifact affordances.

**Deferred to v3.0:** run-to-run stability (3 reruns); full human IRR pool with three accredited raters; cross-family judge ensemble across all claims (we use it only on a calibration slice); institutional-library full-text access; external coordinator with results escrow.

---

## 1. Capture — One Run per (Tool, Question)

### 1.1 Run matrix

7 tools × 20 questions × **1 run** = **140 captures.** Reruns are deferred to v3.0. Acknowledged limit: non-determinism (§1.3 bullet 5 of whitepaper) cannot be measured in v2.0 — call this out explicitly in the brief.

### 1.2 Per-tool capture

| Tool | Access | Capture |
|---|---|---|
| Syno | Internal | `litreview_benchmark_20.py` (shipped); writes the full LitReviewResult per question (`final_draft.md`, `prisma_flow.json`, `references.json`, `claim_verdicts_final.json`, `candidate_papers.json`, `result_summary.json`) plus per-round artefacts |
| Elicit | Public API | `systematic_review` endpoint, save raw JSON + markdown |
| Consensus | Web | Playwright headless first; **manual browser capture as fallback** (HTML + PDF + screenshot via "save complete webpage" + a screen-recording for provenance) |
| Perplexity Deep Research | API (`sonar-deep-research`) | `chat.completions` with citations |
| OpenAI Deep Research | API | Full response + `web_search_call` items |
| OpenEvidence | Web | Playwright headless first; **manual browser capture as fallback** with the same provenance bundle |
| Undermind | Web (try API beta; fall back to Playwright; manual if both fail) | JSON dump if API, scrape if Playwright works, else manual |

⚑ **Decision required day 1:** the headless-Playwright path is brittle on Consensus / OpenEvidence / Undermind — they actively block headless browsers (CAPTCHAs, anti-bot, dynamic DOM). For only 20 queries per tool, **manual browser capture is realistic and often faster than fighting anti-bot for two days**. Plan: try Playwright first per tool (one-day timebox); if it doesn't work cleanly, switch to manual capture for the remaining 19 queries.

**Manual capture protocol (per query):** open a fresh incognito session, paste the verbatim prompt, screen-record the entire response (mp4), save the rendered output as "save complete webpage" (HTML + assets), export as PDF, and copy the citations list to a structured JSON. The provenance is identical to scraped capture; the work is honest. Each manual query takes ~5 minutes once the tool's UI is familiar.

**Time budget revised:** W1 D2–D4 for capture (was D2–D3), to accommodate Playwright drift and the manual fallback for the harder tools.

### 1.3 Standardised prompt

Single canonical question string sent verbatim to all 7 tools. No tool-specific scaffolding. Where mode selection is required (Elicit "systematic review" vs "find papers"), most LR-shaped mode is used and recorded.

### 1.4 Per-capture artefacts

```
captures/<tool>/<question_id>/
  prompt_sent.txt
  raw_response.{json|html}
  normalized.md
  citations.json
  metadata.json     (model_version, ts_utc, run_id, wall_s)
  capture/          (html, pdf, screenshots)
```

For Syno specifically, the existing `litreview_benchmark_20.py` produces a strictly larger artefact set (PRISMA, references, candidate papers, per-round verdicts). This asymmetry is acknowledged in the brief — Syno's API exposes structured affordances competitors don't expose externally — and the Phase 4 affordance audit (§4) gives competitors credit for any structural artefact they expose at their UI surface.

---

## 2. Phase 1 — Mechanical Extraction (Fully Automated, ~1 day)

Per capture, extract deterministically:

| Field | How |
|---|---|
| `word_count`, `char_count`, `wall_clock_seconds`, `model_version_string`, `capture_timestamp_utc` | Trivial Python |
| `claim_count` | Sonnet 4.6 structured-output extraction — one row per citation-bearing sentence |
| `n_citations_total`, `n_unique_papers_cited` | Regex + LLM dedup |
| `n_citations_resolved`, `n_citations_unresolved`, `hallucinated_citation_rate` | Citation-resolver pipeline |

**Citation-resolver pipeline** (`benchmark_resolver.py`, ~200 lines):

1. PMID / PMCID / DOI → PubMed E-utilities `esummary` (free, 10/sec with API key)
2. Author-Year string → fuzzy match via E-utilities `esearch` with title + author + year heuristics
3. URL → extract identifier or resolve via OpenAlex
4. Unresolvable → flagged `hallucinated` (intentionally strict lower bound; §8 false-positive audit caveats this)

**Cost:** ~$25 in LLM tokens (Sonnet claim extraction at $0.045/capture × 140), all API calls free.
**Time:** ~1 day to build, ~2 hours to run across 140 captures.

Outputs land in `phase1.sqlite`.

---

## 3. Phase 2 — AI Judge (Single Opus 4.7 after Calibration, ~3 days)

This is the heart of the v2.0 protocol — and where automation does the most work.

### 3.1 Source-text retrieval — Syno's corpus as primary source-of-truth

For every cited paper across all 140 captures, source text is retrieved in a four-step cascade designed to maximise fair full-text grading and explicitly avoid penalising tools for citing papers we cannot verify:

1. **Syno's own litreview server** → if the cited paper's identifier (PMID, PMCID, DOI) is in our indexed open-access diabetes corpus, we already have the full structured text. This is the primary source for an estimated 60–75% of competitor citations (diabetes OA is mostly PMC-OA, which is what Syno indexes).
2. **PMC E-utilities `efetch` XML** → for OA papers not yet in Syno's corpus but available in PubMed Central.
3. **Unpaywall API** → for closed-access papers with legal OA copies elsewhere (~70% hit rate on biomedical).
4. **PubMed `esummary` abstract** → for papers where no full text is legally accessible; flagged `abstract_only_graded=true`.
5. **Still nothing** → flagged `UNVERIFIABLE_DUE_TO_ACCESS=true`; excluded from primary PASS/FAIL denominator.

**Why this is fair to all 7 tools.** Syno only grades against open-access diabetes literature — the scope its corpus actually covers. A competitor citing a paywalled 2024 NEJM trial that web-browsed deep-research products can access is *not penalised* for citing outside our verification universe; their citation is flagged unverifiable and excluded from the denominator. The per-tool unverifiable rate is reported separately in the brief so a reader can see whether one tool is leaning heavily on sources we cannot read (which is its own signal, just not the citation-faithfulness signal).

**Why this neutralises the abstract-only false-negative trap.** The earlier draft of this protocol would have graded a competitor's correct citation of a Table-2 hazard ratio as a FAIL if the abstract didn't contain that exact number. Under the corrected cascade, claims whose granular content is plausibly in the full text but not in the abstract land in the `UNVERIFIABLE_DUE_TO_ACCESS` category, not the FAIL category. See the explicit judge-prompt rule in §3.4.

**Expected coverage.** Conservative estimate: ~75–85% of competitor citations gradable against full text (cases 1+2+3); ~10–15% abstract-only (case 4); ~5–10% unverifiable (case 5). Per-tool breakdown reported in the brief.

**Passage extraction** is one Opus call per (paper, question) — *not* per claim — to amortise cost across same-question claims. Output: 1–3 paragraphs from the paper that contain the claim-relevant evidence. Caches in `source_corpus/<paper_id>__<question_id>.txt`.

**Cost:** ~$1.5k for full-text fetch + passage extraction across ~3,000 unique cited papers.

### 3.2 Judge model — single Opus 4.7 after a 5% calibration

**v2.0 commitment:** primary endpoint is graded by **a single judge: Claude Opus 4.7** with the hardened prompt below.

The cross-family ensemble recommendation from the literature is honoured **only on a 5% calibration sample (~50 claims):** both Opus 4.7 and GPT-5 grade those 50 claims. We compute pairwise Cohen's κ:

- κ ≥ 0.75 → single Opus judge for the remaining 95%; calibration result published in the brief as a methodology contribution in its own right
- κ < 0.75 → fall back to ensemble across all claims; cost rises ~$1.5k, time +1 day

⚑ **Family-bias mitigation we DO commit to:** a separate **anchor sample of ~100 claims is graded by a non-Anthropic, non-OpenAI model (Gemini 2.5 Pro)** as a held-out bias check. Reported per-tool: does the Gemini anchor shift Syno's score systematically vs the Opus judge? If yes, that's a finding we publish. If no, we have a defensible answer to the "Syno is Anthropic so the Anthropic judge favours it" attack.

### 3.3 Per-claim rubric (1–5 ordinal, 4 axes)

For each (claim, cited_paper) pair, the Judge produces:

```json
{
  "claim_id": "DM-07-C04",
  "cited_paper_id": "PMC1234567",
  "evidence_quote": "verbatim sentence from passage closest to supporting claim",
  "missing_elements": ["any claim element NOT in evidence_quote"],
  "axis_population": {"score": 1-5, "rationale": "≤ 40 words"},
  "axis_numerics":   {"score": 1-5, "rationale": "≤ 40 words"},
  "axis_outcome":    {"score": 1-5, "rationale": "≤ 40 words"},
  "axis_recency":    {"score": 1-5, "rationale": "≤ 40 words"},
  "axis_overall":    {"score": 1-5, "rationale": "≤ 60 words"},
  "passage_used": "full passage shown to judge",
  "abstract_only_graded": false,
  "confidence": "high|medium|low"
}
```

Score anchors:

| Score | Meaning |
|---|---|
| 5 | Fully supported by the cited passage; no overreach |
| 4 | Substantially supported with minor wording overreach |
| 3 | Partial support — one element matches, another does not |
| 2 | Paper exists and is topically relevant but does not support the specific claim |
| 1 | Fabricated, reversed, or grossly misrepresented |
| `unverifiable_due_to_access` | Cited paper is real but the granular content the claim references is not in any open-access portion we can read; excluded from PASS/FAIL denominator (§3.1, §7) |

Headline collapse: PASS = 5, PARTIAL = 3–4, FAIL = 1–2, UNVERIFIABLE = own category.

### 3.4 Judge prompt (verbatim, pre-registered)

```
SYSTEM:
You are a rigorous biomedical peer reviewer. Your only job is to
decide whether THIS PASSAGE supports THIS CLAIM. You do not use your
internal training data. If the claim is true in general but THIS
PASSAGE does not contain the evidence, you score ≤ 3 with overall =
"unsupported_by_this_paper".

Step 1: Copy the single sentence from the passage that is closest to
supporting the claim into `evidence_quote`. If no sentence is even
close, leave it empty.

Step 2: List in `missing_elements` every concept, number, population
descriptor, or outcome named in the claim but not in `evidence_quote`.

Step 3: ABSTRACT-ONLY ACCESS RULE. If `abstract_only_graded` is TRUE,
AND the claim references granular content (specific table values,
secondary endpoints, subgroup p-values, exact CIs, methods detail)
that is plausibly in the full text of the paper but absent from the
abstract you were given, do NOT score it 1 or 2. Set `axis_overall`
to `unverifiable_due_to_access` and assign each axis the value
`N/A`. The brief excludes these claims from the primary PASS/FAIL
denominator and reports their rate separately. This rule prevents
penalising a tool that correctly cited content we cannot read.

Step 4: For all other cases, score each axis independently. Your
population score MUST NOT influence your numerics score, and vice
versa.

Step 5: Assign `confidence`:
  - `low` if the passage is shorter than 100 words OR any axis
    required inference beyond literal text.
  - `medium` if all axes were answerable from the passage but with
    some wording inference.
  - `high` if every axis was answerable from explicit text.

Output strict JSON conforming to the schema.

USER:
[Claim]
{claim_text}

[Cited paper passage — verbatim]
{passage_text}

[Abstract-only flag]
{abstract_only_graded}

[Unverifiable flag — paper not accessible in any open form]
{unverifiable_due_to_access}
```

Temperature ≤ 0.2, fixed seed where supported, no conversation history. Each grading call is fully stateless.

### 3.5 Phase 2 sanity checks (the cheap rigor moves we DO commit to)

- **Self-consistency** on 5% of claims (~50): same Judge regrades, verdicts should not flip > 5%
- **Position-invariance** on 5% of claims: randomise passage order, verdicts should be stable
- **Adversarial passage** on 5% of claims: feed deliberately-wrong paper's passage, Judge should mark `unsupported_by_this_paper` rather than fabricate support
- **Cross-family anchor** on 5% of claims (~50, stratified by tool): Gemini 2.5 Pro grades the same items; report Δscore by source family

If any of these fail their thresholds, the Judge prompt is revised and Phase 2 restarts on a fresh 5% sample before the full run resumes.

**Cost (Phase 2 grading + sanity):** ~$2.5k total. ~$1k Opus across ~1,000 claims (single judge), ~$0.5k ensemble on calibration, ~$0.5k Gemini anchor, ~$0.5k sanity checks.

### 3.6 Syno-as-Validator Cross-Check (the "eat our own dog food" signal)

In addition to the generic Opus judge, **Syno's own per-claim validator** grades every claim across all 140 captures. This is the same scope-first, per-axis judgment infrastructure described in WP §2.2 — population, numerics, outcome, recency, with verbatim-quote-before-label commitment.

**Why include it.** The whitepaper's central claim is that scope-first per-axis judgment catches misinterpretation a generic single-label judge misses. The v2.0 benchmark tests this empirically: where Syno's validator and the generic Opus judge agree, the verdict is high-confidence; where they diverge, the divergence pattern itself is the finding. If Syno's validator systematically catches failures Opus misses on the contractor-validated sample, the architecture claim is backed by data. If it doesn't — equally interesting, and we publish it.

**How it runs.** Each (claim, cited_paper, passage) tuple is sent through Syno's validator endpoint with the same passage the Opus judge sees. The validator emits its native per-axis output (population_match, numerics_alignment, outcome_alignment, recency_concern), plus the scope-first commitment fields (relevant section, verbatim quote, missing elements). These are mapped to the same 1–5 scale via a frozen rubric (locked alongside this document).

**Bias acknowledgement.** Syno grading its competitors is the most direct form of vendor-judges-rival you can construct. The protocol does not hide this. The mitigation is that Syno is **one of four grading signals**: Opus judge, Syno validator, Gemini anchor (on 100 claims), contractor (on 100 claims). The brief reports per-tool scores under each of the four independently, plus the agreement matrix among them. Where the four agree, the per-tool ranking is unimpeachable. Where they disagree — especially where Syno disagrees with the other three — we publish the divergence and let the reader judge.

**Cost:** ~$500 in additional LLM cost (Syno's validator is internal infrastructure but each call consumes Sonnet/Gemini tokens).

### 3.7 Review-level scores (one set per capture, Sonnet)

Per capture, Sonnet 4.6 produces:

- **Claim recall vs gold set** (1–5): how many of the 10–15 gold-claim items are addressed?
- **Synthesis coherence** (1–5, SANRA rubric)
- **Evidence-gap acknowledgement** (1–5)
- **Overall usefulness for a clinical-research lead** (1–5)

Cost: ~$100 total.

---

## 4. Phase 3 — Human Spot-Check (Token, Not Pool, ~1 day)

**v2.0 commitment:** 100-claim active-learning human spot-check with one paid contractor.

### 4.1 Sample selection (active learning, not random)

The 100 claims are selected to maximise information:

- **40 claims:** highest Judge confidence — sanity check that PASS-5 verdicts really are correct
- **40 claims:** lowest Judge confidence — exactly where human judgment changes verdicts
- **20 claims:** stratified across tools and abstract-only-graded status — coverage anchor

### 4.2 Single contractor, not three raters

One paid contractor (clinical-research / evidence-synthesis background, ~10 hours @ $50–80/hr = ~$500–800). Recruit via Upwork, FlexJobs, or methodologist network — 1-week lead time, not 4. They get the same prompt structure the AI Judge sees: claim + passage + rubric. They do not see which tool produced the claim.

⚑ **Trade-off:** no Krippendorff's α across 3 raters. We instead report:

- **Cohen's κ** between the contractor and the Judge across all 100 claims
- **Disagreement breakdown:** of the claims the Judge marked PASS, how many did the human mark FAIL? And vice versa.

If κ ≥ 0.70 → headline numbers ship as Judge-graded with human-spot-check audit appended.
If κ < 0.70 → we publish the disagreement and add a paragraph saying "the AI judge and human contractor diverge on X type of claim; partners should treat headline numbers in that category with care."

### 4.3 Gold-claim set construction (the hardest bias)

The 20 questions need a gold-claim list (10–15 expected claims per question) for claim-recall scoring.

**v2.0 commitment:** Sonnet 4.6 auto-extracts a draft gold-claim set from the canonical published meta-analysis / Cochrane review for each question. The contractor audits 5 of the 20 gold-claim sets (random sample, ~3 hours of their 10). Audit findings are reported as a `gold_set_audit_score` in the brief.

⚑ **Acknowledged limit:** dual-rater gold-set construction is not feasible in 2 FTE × 3 weeks. The mitigation: the gold sets are public CC-BY artefacts; any methodologist can re-construct and re-score against ours.

---

## 5. Phase 4 — Artifact-Affordance Audit (~2 hours per tool, self-do)

Per tool, the engineer FTE spends ~30 minutes on 3 sample questions documenting:

| Affordance | Value |
|---|---|
| In-line citations? | yes / no / hover-only |
| Citation format | superscript / parenthetical / footnote |
| Passage anchors (clickable verbatim quotes)? | yes / no / partial |
| Per-claim confidence or verdict surfaced? | yes / no |
| Bibliography / references list? | yes / no |
| PRISMA flow diagram? | yes / no |
| Search-scope disclosure (sources, date filters)? | yes / no |
| Per-paper evidence-quality indicator? | yes / no |
| Explicit abstention / "don't know" honesty? | yes / no |
| Reproducibility / version hash? | yes / no |

Output: comparison table in the brief. ~14 hours total.

---

## 6. Phase 5 — Open-Data Release (the credibility move that replaces external coordinator)

The single move that mitigates "vendor grades own homework" within startup budget: **publish everything, invite re-grading.**

**Released under CC-BY at publication:**

1. All 140 raw captures (HTML, JSON, screenshots) — every word every tool produced
2. The source-text corpus (passages used by the Judge)
3. The complete `benchmark.db` SQLite with every verdict line
4. Every judge prompt verbatim, every transcript
5. The auto-generated gold-claim sets + the contractor's 5-question audit
6. Analysis code as a `replicate.sh` that, given fresh API keys, regenerates every figure
7. A `re-grade-with-your-judge.py` helper that lets anyone swap in their own LLM judge and re-run

**Brief framing:** "Below are our results under a single-judge methodology validated against a 100-claim human-contractor audit and an independent grading pass by Syno's own per-claim validator. The full grading dataset is public CC-BY; we encourage replication with stricter rubrics, additional rater pools, or alternative judge model families. Our v3.0 protocol will incorporate three-rater IRR and three reruns once industry feedback on this v2.0 lands."

### 6.1 "How to Challenge These Conclusions" — Revisable-Conclusions Commitment

The brief closes with a section titled *"How to challenge these conclusions"* that:

1. **Names the load-bearing assumptions.** Single-judge after calibration; auto-built gold sets; one contractor not three; one run per tool; ~25% abstract-only graded. Each assumption is listed with its corresponding sensitivity analysis in §7.4 so a critic can immediately see the impact.

2. **Provides exact replication paths.** For each assumption, the brief points to the specific cell in `benchmark.db` and the specific argument to `re-grade-with-your-judge.py` that lets a critic re-run the analysis under their alternative.

3. **Commits to a 30-day revision window post-publication.** Any credible methodological critique submitted within 30 days of publication that materially changes results triggers a public revision of the brief — with the critic credited by name. After 30 days, revisions still happen but in a v2.1 batch update with a publish cadence (quarterly).

4. **Pre-commits the criteria for "material change."** A critique is material if applying it shifts any per-tool ranking, or shifts any pairwise comparison's effect size by ≥ 5 percentage points, or shifts the headline first-pass failure-rate framing.

This is the operational expression of "we are confident in our methodology but we trust the data more than our priors." It also makes engagement with critics non-optional and structured — instead of a defensive blog response, the brief itself updates with the correction.

This combination — full open data plus a revisable-conclusions commitment — is the v2.0 substitute for an external coordinator. It is honest, defensible, and turns the budget constraint into a transparency feature.

---

## 7. Statistical Analysis (Pre-Committed)

### 7.1 Primary

- Per-tool **PASS rate** (axis_overall ≥ 4) with **Wilson 95% CI**
- Per-tool **FAIL rate** (axis_overall ≤ 2) with Wilson CI
- Pairwise tool comparison using **paired bootstrap** (5000 resamples) on per-question PASS rates
- Effect sizes reported as **Cliff's δ**

### 7.2 Hierarchical sanity model

In addition to the per-claim primary, fit one **mixed-effects logistic model**:

`PASS ~ tool + (1 | question_id)` on the full 1,000-claim sample

Question random intercept accounts for cluster effects. Report whether per-tool rankings agree with the per-claim primary. If they diverge, that's a finding.

### 7.3 Secondary

- Hallucinated-citation rate per tool with Wilson CI
- Claim recall (Sonnet-graded) per tool, mean ± 95% CI
- Synthesis coherence per tool
- Cohen's κ between Judge and contractor on the 100-claim audit

### 7.4 Pre-committed sensitivity analyses

1. Excluding all `abstract_only_graded` claims
2. Excluding claims where the human contractor disagreed with the Judge by ≥ 2 points
3. Restricting to the questions Syno was pre-committed to perform poorly on (the anti-cherry-picking sensitivity from WP §7)
4. Gemini anchor sub-analysis: does the Gemini grader's per-tool ranking match Opus's?
5. **Including `unverifiable_due_to_access` claims in the FAIL bucket** (the worst-case interpretation a critic could apply) — does the per-tool ranking shift? If Syno still leads under this hostile sensitivity, the ranking is robust to the access-asymmetry concern.

### 7.5 Per-tool transparency reporting

Alongside the primary PASS/FAIL/PARTIAL distribution, the brief reports per-tool:

- **`unverifiable_rate`** — what fraction of the tool's citations could not be graded against open text. A tool that leans heavily on paywalled sources will have a high rate; this is itself a signal (positive or negative depending on the reader's priorities) and is shown openly.
- **`syno_corpus_coverage_rate`** — what fraction of the tool's citations were graded directly from Syno's own indexed corpus.
- **`unpaywall_fallback_rate`** — what fraction were graded via legal OA copies outside Syno's corpus.
- **`abstract_only_rate`** — what fraction were graded against abstracts only.

### 7.5 Multiple-comparison correction

21 pairwise primary tests → **Holm-Bonferroni** within the primary family. Family-wise α = 0.05.

---

## 8. Threats to Validity + Mitigations

| Threat | What we do |
|---|---|
| Vendor-grades-own-homework | **Full open-data release (§6)** plus **revisable-conclusions commitment (§6.1)** plus **four independent grading signals (Opus, Syno, Gemini anchor, contractor)** is the v2.0 substitute for an external coordinator. |
| Judge family bias (Opus judges Syno's Sonnet output) | Gemini 2.5 Pro anchor on 100 claims; published delta |
| Syno-grades-rivals (§3.6 cross-check) | Syno is one of four signals, not the primary; agreement matrix among all four published; divergence is reported as a finding, not hidden |
| Single-judge ≠ ensemble | 5% calibration sample with cross-family agreement reported; if κ < 0.75 we upgrade to ensemble on all |
| Gold sets built by Syntactiq | Auto-generated from canonical published reviews, contractor-audited on 5/20; CC-BY published so anyone can replace them |
| Abstract-only false-negative trap (penalising a competitor for citing content we cannot read) | **Explicit `unverifiable_due_to_access` category in §3.3**; excluded from primary PASS/FAIL denominator; per-tool unverifiable rate reported separately; hostile-case sensitivity counts unverifiables as FAIL (§7.4.5) |
| Hallucinated-citation false positives (resolver misses real paper) | Hand-audit 50 claims marked unresolved; report resolver FPR alongside the headline |
| Position / length bias in judge | Sanity checks on 5% (position randomisation; length-stratified regression); published |
| Tool version drift | Captured at runtime, recorded per capture; brief reports versions |
| Capture asymmetry (Syno's API vs competitors' Playwright) | Phase 4 affordance audit gives credit for any structural artefact at the tool's UI |
| Selection bias in 20 questions | §7 of WP explicitly scopes to diabetes; questions locked before lock-day; brief reports rationale per question |

---

## 9. Timeline — 3 Weeks (2 FTE)

| Day | FTE-1 (engineer) | FTE-2 (methodology) |
|---|---|---|
| W1 D1 | Lock prompts (this doc + judge); finalise resolver; smoke-test capture pipeline on 1 tool × 1 question | Lock 20 questions; OSF pre-register; recruit contractor (post-job); auto-build gold-claim sets |
| W1 D2–D4 | Capture all 7 tools × 20 questions (140 runs). Try Playwright per tool; fall back to manual browser capture when anti-bot blocks. | Audit gold-claim sets (~3 hrs); refine judge prompt per smoke |
| W1 D5 | Build source-text cascade: index Syno-corpus matches; PMC-OA; Unpaywall; abstract fallback; flag UNVERIFIABLE for unresolvable | Contractor on-boarding; signed NDA + paid hour for rubric calibration |
| W2 D1 | Phase 1 mechanical extraction (~2 hrs); hand-audit 10% claim-extraction | 50-claim Judge calibration sample (Opus + GPT-5); compute κ; commit to single or ensemble |
| W2 D1–D2 | Phase 2 main Judge run (~1,000 claims) on Opus; ~3hr wall-clock with batching | Resolver FPR audit (50 claims); Phase 2 sanity checks |
| W2 D3 | Gemini anchor on 100 claims (~1 hr) | Phase 4 affordance audit (~14 hrs across 2 days) |
| W2 D4 | Active-learning sample selection (100 claims for human); ship to contractor | Phase 4 continued; start analysis scripts |
| W2 D5 | Contractor grading (10 hrs over W2 D5 + W3 D1) | Statistical analyses; bootstrap CIs; mixed-effects model |
| W3 D1–D2 | Analyses, figure generation, public-dataset prep | Brief drafting (target 4–5k words) |
| W3 D3 | `replicate.sh` + `re-grade-with-your-judge.py` | Brief drafting cont. |
| W3 D4 | Internal review; pre-publication peer-read by an external methodologist (1 day turnaround if booked W1) | Brief finalisation |
| W3 D5 | Public release: brief + dataset + code + judge transcripts | Public release |

**Buffer:** 2 days at the end of W3 for slip. If we slip past that, we publish minus the sensitivity analyses that aren't ready and add them in a v2.1 update within 2 weeks.

---

## 10. Honest Limits (Section the Brief Will Restate)

What the v2.0 benchmark **cannot** demonstrate, plainly stated:

1. **Run-to-run stability.** Single run per (tool, question). Non-determinism (WP §1.3 bullet 5) is asserted in the whitepaper but not measured here.
2. **Three-rater inter-rater reliability.** One contractor, not three. The standard Krippendorff's α statistic is not computed; we report Cohen's κ judge-vs-contractor instead.
3. **External coordinator.** Syntactiq runs the benchmark. The credibility mitigation is open data, not independence.
4. **Institutional library access.** ~20–25% of claims will be abstract-only-graded; impact is reported and sensitivity-analysed.
5. **Full ensemble across all claims.** Single judge after calibration; cross-family agreement reported only on 5% calibration + 100-claim Gemini anchor.
6. **Cochrane-grade methodology.** This is an interim industry benchmark, not a Cochrane review. v3.0 will add the moves above.

These are not hidden in a methods appendix — they're stated in the brief's abstract.

---

## 11. v3.0 — What We'd Add With Real Resources

After v2.0 ships and we have industry feedback on what matters:

- 3 reruns per (tool, question) → stability sub-analysis
- 3-rater IRR pool on 6/20 questions with Krippendorff's α ≥ 0.80
- Cross-family judge ensemble across all claims (Opus + GPT-5 + Gemini 2.5)
- External benchmark coordinator (Cochrane Methods Group, Bond EBHC, McMaster HiRU)
- Adversarial question contribution (5 each from competitors + clinical board + Cochrane priorities; 5 from Syntactiq)
- Institutional library agreement for closed-access full-text
- Dual independent gold-set construction with κ
- GLMM as primary inferential model with full clustering structure
- Multiple-comparison correction across all 336+ tests with FDR

This is the v3.0 stretch — a 14-week, $40k effort that the published v2.0 funds and motivates.

---

## 12. Budget

| Line | $ |
|---|---|
| LLM API (Opus judge ~$1k, Syno-validator cross-check ~$0.5k, Sonnet extraction + review-level ~$0.5k, calibration ensemble ~$0.5k, Gemini anchor ~$0.3k, sanity reruns ~$0.5k, passage extraction ~$1k) | **~$4,300** |
| Source-text retrieval APIs (all free; bandwidth only) | $0 |
| Contractor (10 hrs @ $50–80/hr + 3hrs gold audit) | **~$800** |
| External methodologist peer-read honorarium | **$500** |
| Cloud (Playwright VMs, S3 storage) | $100 |
| OSF registration + GitHub release tag | $0 |
| **Total ex-headcount** | **~$5,700** |

Plus 2 FTE × 3 weeks loaded.

---

## 13. Decisions to Lock on Day 1

1. **Single judge after calibration vs cross-family ensemble on all claims.** Recommendation: single Opus after 5% calibration with κ ≥ 0.75; ensemble as fallback.
2. **Undermind API or Playwright?** Recommendation: Playwright.
3. **Contractor recruiting source.** Recommendation: post on FlexJobs / Upwork day 1; have 2-3 candidates by end of week 1.
4. **External methodologist for peer-read.** Recommendation: pre-book day 1, $500 honorarium, scope to 5-page review.
5. **Library access fallback.** Recommendation: accept abstract-only for ~25% of claims; report sensitivity.

---

## Appendix A — Judge Prompt (Verbatim, Locked)

See §3.4. Locked together with this document at OSF pre-registration.

## Appendix B — Gold-Claim Auto-Build Prompt

*To be locked alongside this document. Constructs a 10–15 item gold-claim list from a canonical meta-analysis or systematic review per question.*

## Appendix C — Open-Data Manifest

Final dataset (CC-BY at publication):

```
benchmark-v2-release/
  README.md
  LICENSE  (CC-BY-4.0)
  protocol-v2.0.md           (this document, frozen)
  questions-v1.0.md          (the 20 questions, frozen)
  captures/                  (140 raw captures, all 7 tools)
  source_corpus/             (passages used by judge)
  benchmark.db               (SQLite, every verdict line)
  judge-prompts/             (every prompt verbatim)
  judge-transcripts/         (every grading call's input + output)
  gold-claim-sets/           (auto-built + audit findings)
  analyses/                  (notebooks + scripts)
  replicate.sh               (regenerate every figure from benchmark.db)
  re-grade-with-your-judge.py  (community re-grading helper)
```
