# How We Validated the Literature-Review Outputs

*A plain-language overview of the benchmark's methodology, written for the
transparency page that ships with the whitepaper. It explains what we measured,
how we tried to be fair, the specific steps we took to keep "AI grading AI" from
quietly biasing the result, and where the limits are. For the exact figures, see
the [Benchmark Findings Log](benchmark-results/BENCHMARK-FINDINGS-LOG.md). For the
frozen analysis plan, see the [pre-registration](benchmark-prereg.md).*

---

## What the benchmark asks

We compared four AI literature-review tools — Syno (our product), Elicit,
Consensus, and Gemini Deep Research — on a set of diabetes research questions. The
central question is **citation faithfulness**: when a tool cites a paper for a
claim, does that paper actually support the claim — its population, its numbers,
its outcome, its recency? Numbers are the sharpest test, so the pre-registered
primary endpoint is **numeric fidelity** on single-paper claims.

## We put every tool through the same pipeline

Fairness here means *identical treatment*, not a soft touch for our own product.
Every tool's output is normalized to one common schema; citation-style "tells" are
scrubbed to a neutral marker so a grader cannot recognize the source by its
formatting. Every reference is checked by the same resolver; every claim is graded
by the same judges with the same prompts, against the same retrieved source text,
under the same inclusion and exclusion rules. Syno gets no special path, and
Syno's references are verified with no free pass.

There is **one structural asymmetry we disclose rather than hide**: Syno's claims
were graded almost entirely against the *full text* of cited papers, while the
competitors' were graded mostly against *abstracts*. This is not an oversight — it
reflects the evidence each tool actually used. Syno is full-text-only by design;
tools like Elicit operate on an abstract for any paper they cannot open in full.
Grading each tool against the evidence base it actually relied on is the
ecologically honest choice, but it means part of Syno's measured advantage comes
from working with richer source material rather than from method alone. A matched
abstract-only comparison is acknowledged future work.

## Yes, we used AI to help grade — the same AI, the same way, for everyone

The faithfulness judgments are produced by large language models acting as judges.
We state this plainly because it would be the first objection a careful reader
raises. The mitigation is not to pretend otherwise; it is to constrain *how* the AI
grades so the result does not collapse into a model rating its own work.

## How we kept "AI grading AI" from biasing the result

Several independent safeguards, none sufficient alone:

- **A deterministic, AI-free floor.** The first layer — does each cited reference
  exist and is it a real scholarly source? — uses no language model at all. Every
  reference is checked against authoritative scholarly databases (Crossref, NCBI,
  OpenAlex, Unpaywall) and bucketed as resolved, non-scientific, or unverifiable.
  This step is cached and reproducible: anyone can re-run it and get the same
  answer. It is the trust floor under everything else.

- **An external, multi-family judge panel.** Faithfulness is graded by a panel of
  three judges drawn from **two different model families** — Anthropic's Claude and
  Google's Gemini. No single model both writes and grades. We collapse the three
  judges to one verdict per claim by majority before computing any rate, and we
  report how often the judges disagree (Fleiss' κ per axis; fair-to-moderate, and
  disclosed).

- **Grading against the real paper, not the tool's say-so.** Judges see the
  actual fetched text of the cited paper and answer a narrow question — does *this
  paper* support *this claim* — drawing on the passage, not on the model's own
  background knowledge or the tool's assertion.

- **Blinding.** The tool's identity is scrubbed before grading. (One residual tell
  we cannot fully remove — Syno's claims tend to be shorter, atomized sentences —
  is disclosed rather than claimed away.)

- **An explicit conflict-of-interest check.** Two of the three judges are
  Gemini-family, and one of the tools is Gemini, so we tested whether the Gemini
  judges favor the Gemini tool. They do not — the Gemini judges are uniformly more
  lenient across *all* tools, a calibration difference, not self-preference. Syno is
  built on Claude, so we report the Claude-judge results separately as well: the
  relative ordering holds whether you read the Claude judge alone, the Gemini judges
  alone, or the pooled panel. The defensible claim is the *relative* lead, not any
  single rosy absolute number.

- **Pre-registration.** The hypothesis, the single primary comparison, the
  exclusion rules, and the statistics were frozen in writing before the grading run,
  so the headline could not be chosen after seeing the data.

- **Adversarial review of our own grader.** We ran redundant review rounds against
  our own grading code and found real bugs in it — a text-truncation bias, a
  blinding leak, several reference-resolver errors that would have produced *false*
  accusations against competitors. We fixed them and publish the corrected numbers,
  along with the record of what was wrong.

- **A human anchor.** Results are read against the known human baseline for
  citation and quotation error in the medical literature (roughly 8–25% depending
  on severity), so "good" and "bad" are measured against how careful humans actually
  perform, not an abstract ideal.

## Anyone is welcome to re-grade

The point is openness. We publish the full final results and the exact questions
used, so a skeptic can re-grade with their own judges, their own rubric, or their
own raters and check whether they reach our conclusions. We would rather be checked
than trusted.

## Honest limitations

This is an interim benchmark, not a Cochrane-grade evaluation, and several limits
all lean *against* over-claiming:

- **Narrow corpus.** A single topic family (diabetes), full-open-access papers
  only, one capture per tool, one run — no run-to-run stability measured.
- **Judge calibration spread.** The two model families disagree by 17–25
  points in strictness (the Gemini judges are the more lenient), so pooled absolute
  rates are inflated by the more lenient majority; this is why we lead with the
  relative lead and report per-judge-family.
- **Small samples in places.** Consensus writes few single-paper numeric claims,
  so its single-paper figure is not reportable and it appears only in synthesis
  mode; reference counts differ several-fold across tools.
- **Disclosed full-text-vs-abstract asymmetry** (above) — method and evidence base
  are not yet disentangled.

We state these in the open, not buried in an appendix. The figures themselves —
soberly, with their caveats — live in the
[Benchmark Findings Log](benchmark-results/BENCHMARK-FINDINGS-LOG.md).
