# Corrections & Review Record — what we found wrong, and how we caught it

*Shipped with the benchmark for transparency. This is the self-critical companion to the
results: the bugs we found in our OWN grading pipeline (including ones that favoured us),
the adversarial review process that caught them, and the citation/faithfulness failures we
found in the tools themselves. We publish this because a benchmark that grades AI with AI is
only trustworthy if the authors are openly accountable for their own mistakes.*

Last updated: 2026-06-02.

---

## A. Bugs we found and fixed in our OWN grader (self-audit)

Every item below was caught by **redundant, adversarial review** (independent agents given
the identical task; we acted only when their findings converged) and then fixed and
re-verified. We flag the **direction of bias** each bug had — note that several favoured our
own product, and we corrected them anyway.

| # | Issue | Bias direction | Fix |
|---|---|---|---|
| 1 | **Blinding leak** — a citation-style tell let the panel infer the tool | favoured **Syno** (inflated it) | scrubbed all tool tells to a neutral marker; re-graded |
| 2 | **Pseudo-replication** — the 3 panel judges were counted as 3 independent observations, inflating n ~3× and narrowing confidence intervals | affected all tools (over-precise) | collapse the 3 judges to **one majority verdict per (claim, paper)** before any rate/CI; question-clustered CI is the inferential statement |
| 3 | **Panel conflict-of-interest** — 2 of 3 judges are Gemini-family while Gemini is a graded tool | potential self-preference | measured per-judge-family; found the Gemini judges are *uniformly* more lenient (and *least* lenient on the Gemini tool) → calibration, not self-preference; **disclosed**, and we report the Claude-judge result separately |
| 4 | **`looks_tabular` over-drop** — a comma-count rule discarded legitimate comma-rich competitor prose | **hurt competitors** (Elicit/Consensus) | count commas only after stripping citation brackets/author-year groups; add an explicit table-export signature; ~7 legit Consensus claims restored |
| 5 | **Gemini wrong-number injection (3 classes)** — a sentence-fusion in Gemini's text export could graft a number from a neighbouring clause/table cell onto the wrong paper | **hurt Gemini** (false failures) | hardened sentence-splitting (period/percent/bracket/colon/superscript glue), a length-ratio guard, a bare-statistic-row drop, and de-duplication; verified 0 wrong-number units remain across 932 graded units |
| 6 | **Resolver under-credited Gemini** — stacked "chrome" in titles (`Title \| Journal \| Aggregator`) and a split-subscript artifact (`HbA 1c`) broke the citation lookup | **hurt Gemini** | strip all trailing chrome; collapse the `HbA1c` artifact; recovered 4 genuinely-indexed papers (Gemini resolved 67.8% → 69.2%) |
| 7 | **`native` free-pass** — Syno citations could be graded on an unverified placeholder status | potential favour to **Syno** | require the same `resolved` bar for every tool; (no live impact — all Syno refs were already verified) |
| 8 | **Orphan verdicts** — re-extraction could leave stale graded rows that still counted | over-count risk | report restricted to the current claim set (orphan guard) |
| 9 | **Architecture-metric overstatement** — an internal metric was first quoted as 99.6% | rosier for **Syno** | corrected to the honest 95.1% fully-grounded figure (99.6% includes weakly-grounded); never reverted |

**Takeaway:** of the nine, **three (4, 5, 6) had been penalising the competitors** and were
fixed in their favour; **three (1, 7, 9) had favoured us** and were corrected against our own
interest. We consider that balance the point of the exercise.

---

## B. How the review was run

The pipeline was built and audited under a standing **redundant cross-validation** workflow —
never a single agent's judgment on anything judgment-shaped:

- **Planning** — multiple independent agents scoped the method and converged on one plan.
- **Implementation** — built in pairs; high-risk slices doubled up.
- **Review sweeps** — the full pipeline was split into ~6 parts (resolver, faithfulness panel,
  fairness, statistics/pre-registration, Gemini ingestion, findings-log honesty); each part
  reviewed by an **identical-task pair**, run **twice** (an initial sweep and a post-fix
  confirmation sweep).
- **Fix verification** — every grading-affecting fix was gated behind a **skeptical pair**
  asked to prove it unnecessary or wrong; we changed code only on convergent findings.
- **Pre-spend gate** — before the paid grading run, two independent verifiers confirmed
  CLEAR-TO-RUN (0 wrong-number units, fair gates, green test suite).
- **Results audit** — after the run, **three** independent agents recomputed *every* headline
  figure directly from the database; all three returned GO (every number reproduced exactly).

The cross-validated review record is bundled in this folder.

---

## C. What we found wrong with the tools (the benchmark findings)

These are the substance of the results — see the findings log for the figures and caveats.

- **Gemini Deep Research** — the weakest on citation quality: **12.1% non-scientific sources**
  (press releases, ClinicalTrials.gov, Scribd, RxFiles, consumer-health) and **18.7%
  unverifiable** references, and it renders statistics as images that drop out of the text
  layer (we recovered them from its own document export; ~24% still graded on a number-hole
  fallback — a disclosed floor).
- **Elicit** — clean citations (98.4% resolved), but its faithfulness failures are **hard
  misattributions** (e.g. crediting remission criteria to an author whose cited paper contains
  no such content; attributing a trial's numbers to the review that merely cited them). It
  operates mostly on **abstracts**, which we judged it on (its actual evidence base).
- **Consensus** — clean citations (99.5%); writes in a multi-citation synthesis style, so it
  yields too few single-paper numeric claims to grade on that axis (its real signal is the
  synthesis bar).
- **Syno (ours)** — its failures are **soft** ("all numbers correct, one detail off" — judges
  split), not wrong-paper errors. Reported with full caveats; the lead is presented as the
  *relative* result, not the lenient pooled absolute rate.

---

*Figures, confidence intervals, the pre-registration, and the full caveat list live in
`02_findings/BENCHMARK-FINDINGS-LOG.md` and `01_methodology/`. Anyone is welcome to re-grade
from the published questions and results.*
