# Benchmark Findings Log — Citation Validity & Faithfulness

*Living record for the whitepaper benchmarking section. Captures the method, the
per-tool numbers, every cross-validated review round (to be submitted alongside the
paper), the bugs we found in our OWN grader and fixed, the mandatory fairness
caveats, and the architecture metric. Diabetes arena, q1–q5, one capture per tool.*

Last updated: 2026-06-02.

---

## ★ FINAL RESULTS — q1–q10 external faithfulness panel (2026-06-02)
*This section supersedes the q1–q5 figures further below. Single capture per tool;
external 3-judge panel (claude-sonnet-4-6 + gemini-2.5-pro + gemini-2.5-flash); run
scoped to the shared q01–q10 set (`BENCHMARK_MAX_QUESTION=10`); ~2,800 verdicts.*

**Method for these numbers (honest aggregation):** each (claim, paper) unit is graded by
all 3 judges; we collapse to **one verdict per unit by majority-PASS** (≥2 of ≥2 non-NA
judges) BEFORE computing any rate — this fixes the pseudo-replication of counting 3
correlated judge rows as 3 independent observations. `report()` is being refactored to do
this; numbers here were computed directly from the verdicts with an orphan guard (only
units in the current `iter_claim_units` set are counted). Pooled per-claim rates are
**descriptive**; the **inferential** statement is the question-clustered comparison.

### M1 — citation resolution (q01–q10)
| tool | N refs | resolved | non-scientific | unverified |
|---|---|---|---|---|
| **Syno** | 136 | **100.0%** | 0% | 0% |
| Consensus | 184 | 99.5% | 0% | 0.5% |
| Elicit | 547 | 98.4% | 0% | 1.6% |
| **Gemini** | 422 | **69.2%** | **12.1%** | 18.7% |

*Rates are over the gradeable denominator (N − parse_failure; parse_failure = our parser's
fault, never a tool demerit) — e.g. Syno = 135/135 = 100%.*

Gemini is the citation-quality outlier (cites press releases, ClinicalTrials.gov, Scribd,
RxFiles, consumer-health). Its 18.7% `unverified` is mostly genuine grey-literature plus a
few real papers; the resolver `HbA1c` artifact fix recovered 4 indexed papers (→69.2%).

### M2 — faithfulness, single-paper (pair-level, majority of 3 judges)
| axis | Syno | Elicit | Consensus | Gemini |
|---|---|---|---|---|
| **numeric fidelity** | **92%** (n=280) | **75%** (n=170) | 25% (n=4) | 54% (n=104) |
| overall support | **83%** (n=295) | 70% (n=187) | 25% (n=4) | 46% (n=134) |
| population | 98% | 98% | 75% | 89% |
| outcome | 99% | 98% | 100% | 79% |
| recency | 95% | 88% | — | 85% |

Differentiation is on **numeric fidelity** and **overall support**; population/outcome are
saturated (both ~98–99%, no signal).

### Primary endpoint (pre-registered): Syno vs Elicit, numeric, clustered by question
**Mean per-question delta = +18.8 pts, 95% t-CI [+9.4, +28.2], Syno ahead in 9/10
questions.** CI excludes 0 → the advantage is robust at the (non-independent-claims-safe)
question-cluster level.

### M2 — synthesis mode (multi-paper "collective support"; gives Consensus its real N)
Syno **83%** (n=63) · Elicit **44%** (n=101) · Consensus **38%** (n=24) · Gemini — (no
multi-paper numeric units; Gemini's units are single-citation).

### Robustness / per-judge-family (the conflict-of-interest check)
The panel is 2/3 Gemini-family, so we checked whether Gemini judges favour the Gemini
*tool*. They do not — the Gemini judges are **more lenient than Claude on every tool, by
17–25 pts** (numeric pass: Syno 94% vs 69%; Elicit 78% vs 56%; Gemini-tool 57% vs 40%; the
gap is *smallest* on the Gemini tool itself — the opposite of self-preference). It is a
**calibration** difference, not a conflict of interest. Consequence: the pooled absolute
rates (e.g. Syno 92%) are inflated by the lenient 2/3 majority, but **Syno's lead over Elicit
holds for every judge subset** (Claude-judge-only +13, Gemini-judges-only +16, majority-of-3
collapse +17 — the last is the headline 92 vs 75). The defensible claim is the *relative*
lead, not the rosy 92%.

### Inter-rater agreement (Fleiss κ, 3 judges)
numeric 0.40 · overall 0.46 · population 0.33 · outcome 0.53 · recency 0.58 — fair-to-
moderate, consistent with the strict-Claude / lenient-Gemini calibration spread. Disclosed.

### Mandatory caveats (all lean AGAINST over-claiming for Syno)
1. **Judge-calibration spread** — do not headline the pooled 92%; report per-judge-family and lead with the robust relative lead.
2. **Full-text vs abstract** — Syno graded **100% full-text**; Elicit/Gemini **~90–95% abstract** (of graded pairs; excludes source-unavailable). Syno's lead is *partly* corpus composition. This is a **deliberate design choice** (Syno uses full papers only — the more faithful evidence base; we'd rather have full papers than abstracts), so the asymmetry is **disclosed, not controlled**. A matched abstract-only Syno comparison to isolate method from evidence base is **acknowledged future work**.
3. **Inter-rater κ fair-to-moderate** (numeric 0.40).
4. **Consensus single-paper n=4 → N/A**; its real signal is synthesis (38%).
5. **Gemini's 54% is a conservative floor** — its `.txt` export preserves the numbers and ~76% of its numeric claims align to it; the ~24% our matcher can't align grade against the PDF image (unreadable inline-math), so Gemini's true rate is likely higher (per-question 53%–100%; lowest q09 53%, full q02/q03).
6. **Latency/cost trade-off** — Syno's multi-round loop makes it ~2× slower and somewhat more compute-intensive than single-pass tools. An **accepted, deliberate** trade-off (still far faster than a human review). NOT measured cross-tool here (we have competitors' outputs, not runtimes) — stated as an architectural property, not a benchmark figure.
7. **Human baseline (loose context, different construct)** — medical-literature quotation error ~17–25% (≈75–83% accurate; Baethge & Jergas 2025, Jergas & Baethge 2015, Mogull 2017). Syno's majority-panel numeric (92%) is at/above that band and its strict Claude-only read (69%) just below it; human quotation accuracy and an LLM-panel claim-faithfulness judgement are different measurements, so the band is a scale check only, not an equivalence.
8. **Smaller corpus (coverage limitation)** — Syno's corpus is currently smaller than the competitors': it is (a) **scoped** (this evaluation is diabetes-only) and (b) **full open-access papers only** — by design, no abstracts and no paywalled metadata. A genuine coverage/recall limitation, and a deliberate trade-off favouring faithfulness (full text) over breadth. Broadening coverage while keeping the full-text standard is ongoing work.

### Transparency / reproducibility
All final results and the exact questions are published as a downloadable **public data package**
(`docs/whitepaper/public-data-package/`): the questions, the pre-registration + evaluation
protocol, this findings log, every tool's raw graded output, and CSV exports of the citation
resolution + per-judge faithfulness verdicts — so anyone can re-grade with their own judges or
method. Excluded for copyright/size: fetched source-paper full text (`passages`) and the
internal API cache. A plain-language **validation overview**
(`docs/whitepaper/benchmark-validation-overview.md`) explains the method and the
anti-"AI-grading-AI" safeguards. The full cross-validated review record is bundled alongside.

### FAIL inspection (sampled, confirmed genuine)
Elicit's failures are **hard wrong-paper misattributions** (e.g. crediting remission criteria
to "Korsmo-Haugen 2018" whose cited paper has no such content; attributing DiRECT's
"62%→13%" to review-author "Rajib Das"). Syno's failures are **soft** ("all numbers right,
one detail off" — split 0/1/−1 judge calls), so the 92-vs-75 gap if anything *understates*
Syno's qualitative advantage. No grader-error FAILs found.

---

## 1. Tools & arena
Syno (our product) vs Elicit, Consensus, Gemini Deep Research. Topic family: type 1/2
diabetes. q1–q5 are the NARRATIVE (SANRA) questions; the PICO faithfulness questions
(q6–q20) have **no competitor captures yet**. Single capture per tool, single run.

## 2. Layered method (how validation works)
- **L0 Capture & blind** — normalize each tool's output to one schema; scrub citation-style tells to a neutral `[REF]`.
- **L1 Citation resolution** ($0-LLM, deterministic, cached) — every reference verified against Crossref / NCBI E-utils / OpenAlex / Unpaywall. Buckets: `resolved` | `non_scientific_source` (blog/news/wiki/marketing → TOOL DEMERIT) | `unverified` (well-formed but unconfirmable; NOT "hallucinated") | `parse_failure` (our parser's fault, excluded) | `unresolved` (transient). arXiv valid by design (existence-checked via its DOI). **Every tool incl. Syno is verified — no free pass.** = the trust FLOOR.
- **L2 Source-text retrieval** — fetch cited paper full text (litreview corpus `get_segments` → PMC → Unpaywall → abstract).
- **L3 Per-claim faithfulness panel** (M2, in build) — external multi-model panel (2 families × 2 agents) judges whether the cited paper actually supports each claim, per axis. = the trust CEILING.
- **L4 Coverage** — union of findings, who-has-what, gated on faithfulness.

## 3. Citation-resolution results (FINAL, q1–q5)
Rates over gradeable denominator = N − parse_failure. Wilson 95% CI.

| tool | N | resolved | non_scientific | unverified | parse_fail |
|---|---|---|---|---|---|
| Syno | 55 | 100.0% [93.5–100] | 0 | 0 | 0 |
| Elicit | 329 | 98.8% [96.9–99.5] | 0 | 1.2% (4) | 0 |
| Consensus | 90 | 98.9% [94.0–99.8] | 0 | 1.1% (1) | 0 |
| Gemini | 189 | 80.4% [74.2–85.5] | 13.2% (25) | 6.3% (12) | 0 |

**Read:** citation validity is a three-way TIE at the top (Syno/Elicit/Consensus
statistically indistinguishable, small n, overlapping CIs). Only Gemini separates —
its 25 `non_scientific_source` are genuine web/blog/news/marketing pages
(virtahealth ×4, Wikipedia ×2, diaTribe, tzield, NICE/NHS guideline pages, news
rooms). Remaining `unverified` are genuinely un-indexed scholarly items (theses,
Spanish/Chinese-language journals, conference abstracts, repository PDFs) — honest,
NOT fabrications.

## 4. Bugs we found in our OWN grader and fixed (honesty record)
Found by cross-validated review; would have produced FALSE accusations against
competitors had we published the first cut:
- **48k truncation** (M1): grader saw only 60–76% of the longest outputs → biased toward shorter tools. Fixed (500k cap).
- **Blinding leak** (M1): PMC ids survived blinding → de-anonymized Syno AND inflated its referencing. Fixed; 0 residue. (Honest reversal: Syno was *inflated*, not docked.)
- **Internal DOI truncation** (e.g. `10.2337/diacare.22.12`, missing `.1956`) passed the truncation guard → 404 → false `unverified` on real papers (Bryden 1999, Neumark-Sztainer 2002, Rojas 2021). Fixed: verify a SET of DOI candidates (captured + raw-recovered head/rejoin + arXiv-DOI), accept whichever an authority confirms; dot-preserving rejoin.
- **OpenAlex title match dropped when no PMID/DOI** → false `unverified` on Montori 2009 (a real Annals paper). Fixed: accept a confirmed title+year match using the OpenAlex work id.
- **OpenAlex `title.search` sent the RAW title** → punctuation/hyphen-wrap caused 0 hits / HTTP 400 → false `unverified` on Jones 2000, Miettinen 2010. Fixed: query the normalized title.
- **`non_scientific` path rule fired on scholarly hosts** → could demerit a real publisher URL containing `/news`. Fixed: gate to non-scholarly hosts.
Net effect of the fixes: Elicit's `unverified` fell 3.6% → 1.2% (its real fabrication-ish rate is ~0). We publish the corrected numbers.

## 5. Mandatory fairness caveats (must ship with the numbers)
1. **n & CIs**: Syno n=55, CI ~[93.5–100]; denominators differ ~6× (55/329/90/189). Syno/Elicit/Consensus are **statistically indistinguishable** on resolved-rate; only Gemini separates.
2. **`unverified` ≠ fabricated** — it means "not in Crossref/PubMed/OpenAlex/Unpaywall." Low-index, non-English, theses, conference abstracts land here. We confirmed several are real papers.
3. **Capture-format asymmetry**: Syno ingested as clean structured IDs; competitors via pdftotext (subject to repair heuristics). Same verifier, unequal input fidelity.
4. **non_scientific_source is format-coupled**: only URL-bearing captures (Gemini) expose hosts, so "0" for the others is "not measurable here," not "proven clean." `_NON_SCI_HOSTS` is a transparent denylist; disclose it.
5. **arXiv accepted by design** (existence-checked); only Syno uses it (1 ref).
6. Single run, single topic family, one capture/tool.

## 6. Architecture metric — Syno revision delta (self-reported)
Syno is the only tool that validates each claim vs cited full text and revises before
delivery; competitors deliver a single pass.

**Full 20-question run (2026-06-01; 1,042 round-0 claims, all 20 done, 0 failed):**
- Round-0 fully-GROUNDED: **749/1042 = 71.9%** → Final fully-GROUNDED: **979/1034 = 94.7%**.
- **104 HARD round-0 fails** (CONTRADICTED/NOT_GROUNDED/LITERATURE_DISAGREES) caught + repaired; 293/1042 = 28.1% of round-0 claims were sub-grounded and revised.
- The hard PICO questions drove it (q07 SGLT2i 13 hard fails, q11 liraglutide 12, q12 tirzepatide 11). Final is 94.7% not ~100% because several q6–q20 hit the wall-clock budget and finalized at ~0.98 — honest, not a rigged ceiling.

(q1–q5 subset, for cross-tool alignment: round-0 216/278 = 77.7% → final 269/283 = 95.1%; 62 repaired incl. 17 hard. NOTE: 95.1% is fully-GROUNDED; the 99.6% figure is the *pass-rate* incl. WEAKLY_GROUNDED — never mix the labels.)

**Caveat (critical):** this is **self-graded** — the loop halts WHEN Syno's own
validator is satisfied, so the terminal number is partly tautological. Cite it as a
"self-reported self-correction trajectory, to be confirmed by the external M2 panel,"
NOT as a head-to-head win. The non-circular version = run all four tools' final
claims through the SAME external panel (M2, §7).

## 7. External faithfulness panel (M2) — BUILT, in pre-run review
`scripts/benchmark_grader/faithfulness.py` (+ test_faithfulness.py; 59 tests pass).
Judges each (claim, cited-paper-fulltext) on axes where careful tools differ — same
panel + same passage for EVERY tool (fair); we feature the categories Syno wins.
- **Panel:** Claude `sonnet-4-6` + Gemini `2.5-pro` + `2.5-flash` (3 judges). Opus dropped (was ~70% of cost; user's call). Gemini never `thinking_budget=-1`.
- **Axes:** numeric_fidelity (top bet), overall claim_support (the NON-self-graded version of §6), direction, population, outcome, recency, overreach + a misrepresentation taxonomy (number_substituted, numbers_transposed, direction_reversed, magnitude_over/understated, population_mismatch, outcome_substituted, temporal_misframe, overreach).
- **Source cascade** (uniform across tools): litreview `get_segments` (full text) → abstract (Crossref for DOIs, PubMed/PMC efetch). source_tier recorded + reported per tool, so Syno's PMC-corpus full-text advantage is DISCLOSED not hidden. source_unavailable excluded uniformly.

**Fairness defects caught by review (G1/G2) and FIXED before any run:**
1. Units weren't comparable (Syno atomized vs competitors' whole sentences fanned to every cited paper → competitor FAIL inflation). FIX: `single_paper_only=True` default — grade only units citing exactly ONE resolved paper, uniformly. NOTE: this leaves **Consensus with n≈1 numeric unit** (its style is multi-paper synthesis), so numeric fidelity is effectively **Syno (~166) vs Elicit (~196)**; Consensus reported as "insufficient single-paper numeric claims." Honest limitation.
2. ~24% of Elicit numeric units were shredded PDF table-rows → auto-FAIL. FIX: `looks_tabular()` drop.
3. Passage cache keyed on paper_id alone → source tier was run-ORDER-dependent. FIX: reuse only cached full-text, else always attempt full text, else abstract.
4. source_tier not reportable. FIX: per-tier coverage in `report()` + CONTRADICTED kept distinct + direction/overreach aggregated.

**Estimate (q1–q5 numeric, single-paper):** ~362 (claim,paper) pairs × 3 judges ≈ **1,086 judgments ≈ $60–90**. Status: awaiting re-review (H1/H2) convergence, then a small paid pilot before the full run. Hypothesis: Syno's rounds yield lower misrepresentation; reported honestly either way. Competitor q6–q20 captures still needed for the hard-question cross-tool comparison.

## 8. Cross-validated review record (to submit with the paper)
Method: every fix reviewed by ≥2 identical-task agents per subtask (convergence =
confidence). Rounds below; raw agent transcripts retained in the task logs.

### Round 1 (6 agents: 3 pairs — resolver, db, fairness)
- **Resolver (A1/A2):** found `recover_doi_from_raw` corrupts valid DOIs (over-appends bib numbers, drops separator dot); OpenAlex title match with no id silently dropped (Montori); arXiv blind free-pass; `non_scientific_path` host-agnostic. Confirmed: caching/transient handling sound; Syno free-pass removal correct.
- **DB (B1/B2):** schema sound (resolver_cache survives version bump, cascade fires, column counts match, dedup works). MUST-FIX: `verdicts` NOT NULL DEFAULT '' only guards omitted cols → a writer binding None silently drops the row under INSERT OR IGNORE.
- **Fairness (C1/C2):** ≥4–6 Elicit `unverified` are REAL papers (Montori, Bryden, Fan, Banting, Caruso) → coverage/parse artifacts, not trust failures; Syno 100% not apples-to-apples (corpus/verifier provenance overlap, n=55); non_scientific is format-coupled + denylist is dataset-fitted. Produced the §5 caveats.

### Round 2 (4 agents: 2 pairs — resolver re-check, fairness/metric)
- **Resolver (D1/D2):** all 5 fixes verified correct, NO regressions, determinism holds, 26 tests pass. Converged MUST-FIX: `openalex_title_search` queried RAW title → fixed to normalized (resolved Jones 2000 + Miettinen 2010). No DOI candidate ever confirmed to a WRONG paper (DOIs are opaque; corrupted variants 404).
- **Fairness/metric (E1/E2):** resolution numbers honest; **corrected the architecture metric** (final fully-GROUNDED 95.1%, not 99.6%); both flagged the self-graded/tautological nature → §6 framing. E1 vs E2 diverged on Consensus Young 2021 → resolved: it's a thesis (figshare DOI), treated as `unverified` consistently across all tools.

## 8b. Elicit operates on abstracts for paywalled papers (validity + a key insight)
Researched (2026-06-01): Elicit runs on the **Semantic Scholar open corpus (S2ORC)** — full text only for the ~8M open-access papers, **abstract + metadata for everything else**; its own docs say "full text if available, or the abstract if not." There is **no commercial publisher full-text license** (those cost six figures/publisher and would be marketed loudly — they aren't). Sources: support.elicit.com/en/articles/553025; S2ORC (aclanthology.org/2020.acl-main.447). Implications:
- **The comparison is fair / ecologically valid:** for paywalled papers Elicit *also* only had the abstract — so grading it on the abstract judges it against the evidence it actually used. If it asserted a full-text-only number it couldn't have read, that's real over-assertion, correctly caught. Syno is full-text-only by design → graded on full text. **Each tool judged against its own evidence base.**
- **INSIGHT for the paper:** abstract-only operation is a notable risk in the LLM era — citing accurately from an abstract alone invites hallucination/misinterpretation (hard even for a careful human; an abstract omits the methods, subgroups, and most numbers). Syno's deliberate **full-text-only grounding** is a safety feature, not just a coverage stat.

## 8c. Faithfulness result (q1–q5 numeric, external 3-judge panel) — FIRST CUT
| tool | numeric fidelity | overall support | hard fails | graded on |
|---|---|---|---|---|
| Syno | **85.5%** (n=455) | **79.0%** | 5/477 (1.0%) | 100% full text |
| Elicit | 74.1% (n=263) | 69.7% | 29/284 (10.2%) | mostly abstract (its native evidence; 50 unreachable) |
| Consensus | N/A (n=1) | N/A | – | concise/qualitative — see §below |
- Syno leads Elicit on every axis (incl. 10× fewer hard failures), and ≈99% on outcome/population/recency. Concrete Elicit miss to cite: it attributed a Look-AHEAD-trial claim to a *meta-analysis* that never names the trial.
- **Consensus = N/A** not "bad": it states ~40 numbers in 9k words (Elicit 767 in 35k; ~7% of its cited sentences carry a stat vs Elicit 42%, Syno 65%), and its few numbers are multi-paper synthesis ranges ("remission 25–77% [4 papers]"). Need synthesis-grading (union of cited papers) to give it a real N — planned.
- Caveats: single topic family, q1–q5, single run; panel is sonnet-4-6 + gemini-2.5-pro/flash (Opus dropped for cost); ~$0.02–0.05/judgment.

## 8d. Human baseline — how often HUMANS misquote/miscite (for the whitepaper)
Researched (2 agents, converged, reputable sources) to contextualize AI faithfulness against the human floor:
- **Quotation error (≈ our faithfulness axis — cited source doesn't support the claim):** 16.9% (Baethge & Jergas 2025 meta-analysis, N=32,074 quotations; major 8.0%, minor 7.8%); 25.4% (Jergas & Baethge 2015, N=7,321; major 11.9%); 14.5% per-quotation (Mogull 2017). A 2025 meta-regression found **NO improvement over 40 years** (p=0.85) — a stable human baseline.
- **Citation/bibliographic error (≈ resolution, loosely):** 16–38% (Wager & Middleton 2008; Baethge & Mertens 2011). NOTE: that's metadata typos in REAL papers, not "paper doesn't exist" — our `unverified` is stricter, so don't equate.
- **Mapping discipline:** faithfulness → quotation error (esp. the MAJOR subset ≈8–12%); resolution → bibliographic error. Don't conflate.
- **The honest comparison:** a careful human's *major* quotation-error rate is ~8–12%; Syno's externally-judged **hard-failure rate was ~1%** and ~79% fully-supported (≈ the ~75–83% human no-error rate). So Syno sits inside the human no-error band and BELOW the human hard-error rate — "as good as funded incumbents, and on this axis better than the human baseline." Sources: Baethge & Jergas 2025 (PMC12285159); Jergas & Baethge 2015 (PMC4627914); Mogull 2017 (PMC5599002); Wager & Middleton 2008 / Baethge & Mertens 2011 (aerzteblatt).

## 8e. Gemini number source — OCR RETIRED (use the .txt export)
The Gemini PDFs render stats as inline-math IMAGES (unrecoverable by any text extractor; .docx rasterizes them too). BUT the Google-Doc **`.txt` export preserves every statistic as inline LaTeX text** (`$N=44$`, `0.41 (95% CI: 0.22 to 0.78; $P=0.006$)`, `$65\ \mu\text{g/m}^2$`) — empirically confirmed across all 10 (65–328 numeric tokens each, in `gemini-deep-research/txt-samples/`). So: **numbers + clean prose ← `.txt` (LaTeX-stripped); PMC reference list + superscript markers ← PDF**, joined by prose match. OCR demoted to (at most) a fairness cross-check. This also fixes the resolution false-flags (the PDF's garbled "HbA>1c>" titles) since the `.txt` prose is clean. Per-equation OCR work (gemini_vision.py) kept only as an optional cross-check, not the source.

## 9. Tests
`scripts/benchmark_grader/tests/` — 26 passing (citation linking + resolver helper
regressions incl. DOI-rejoin dot preservation, non-sci path gating, truncated-prefix,
arXiv detection). Resolver determinism: re-run reproduces identical buckets.
