# Benchmark Public Data Package

This folder is a transparency package for a literature-review benchmark
comparing the **citation faithfulness** of four research tools — **Syno**,
**Elicit**, **Consensus**, and **Gemini Deep Research** — on a set of diabetes
research questions. It is provided so that the results in the accompanying
whitepaper can be independently audited.

It is a copy-and-organize snapshot of the original benchmark artifacts. Nothing
here is a new analysis; the numbers and their caveats live in the findings log
(`02_findings/`), and the raw per-claim grades live in the data export
(`04_data/`).

## What the benchmark measures

Each tool was asked the same diabetes research questions and produced a
literature review with inline citations. We then evaluated two things:

- **M1 — citation resolution:** can each cited reference be resolved to a real,
  indexed scientific paper (vs. grey literature / unresolvable)?
- **M2 — citation faithfulness:** does each claim's cited paper actually support
  the claim, graded on several axes (numeric fidelity, overall support,
  population, outcome, recency) by an external multi-judge panel.

The shared comparison set is **q01–q10** (the questions all four tools answered).
Syno additionally has runs for q11–q20; those extra Syno captures are included
for completeness but are not part of the four-tool comparison. See the
methodology and findings documents for exact scope, judge configuration, and the
honest-aggregation method (majority-vote collapse before any rate is computed).

## Folder layout

```
public-data-package/
  README.md                 # this file
  00_questions/             # the pre-registered benchmark questions
  01_methodology/           # pre-registration + evaluation protocol + plain-language validation overview
  02_findings/              # the findings log (numbers, caveats, grader-bug fixes)
  03_tool_captures/         # each tool's raw output that we graded
    gemini/                 # Gemini Deep Research: PDF + extracted .txt per question
    syno/                   # Syno: graded final narrative (Markdown) per question
    elicit/                 # Elicit: PDF per question
    consensus/              # Consensus: PDF per question
  04_data/                  # sanitized CSV/JSON export of the grading results
  05_reviews/               # corrections record (bugs we found in our own grader) + the cross-validated review log
```

## How to read the data

Start with **`02_findings/BENCHMARK-FINDINGS-LOG.md`** for the headline numbers,
the pre-registered primary endpoint, the per-judge-family robustness checks, the
bugs we found in our own grader (and fixed), and the mandatory fairness caveats
(e.g. `unverified` is not the same as `hallucinated`; all tools were run through
the same pipeline). Treat that document as the source of truth for any figure.

Then go to **`04_data/`** for the underlying grades, one row per observation:

- `references.csv` — citation-resolution results (one row per cited reference:
  author/year/marker, whether it resolved to an indexed paper, and the
  resolution status/source). This is the basis for the M1 numbers.
- `verdicts.csv` — the faithfulness grades (one row per claim×reference×judge),
  with each axis score, the headline verdict, the supporting evidence quote, and
  the judge's rationale. This is the basis for the M2 numbers. Note this is the
  *raw, un-collapsed* judge output — the findings log explains how the three
  correlated judge rows are collapsed to one verdict per unit (majority-PASS)
  before any rate is computed.
- `captures.csv` — lightweight metadata for each tool capture (tool, question,
  format, word count, inline-citation count).
- `tool_outputs.csv` — each tool's own narrative text, kept in a separate file so
  the metadata file stays small.

The raw captures in **`03_tool_captures/`** are the exact tool outputs that were
graded, so any grade in `verdicts.csv` can be traced back to the source text.

### Note on the Syno captures

For Syno, the graded artifact is the final review narrative
(`syno_q01.md` … `syno_q20.md`). The original Syno runs also contain internal
pipeline artifacts (per-round drafts, internal event logs, intermediate
verdicts); those are intentionally **not** included here — only the final graded
narrative is published, which is the text the faithfulness panel actually scored.

## Exclusions (copyright / size)

Two tables from the internal benchmark database are **deliberately excluded** and
are not present anywhere in this package:

- **`passages`** — the fetched full text of the cited *source papers*. This is
  third-party copyrighted content and must not be republished. (Without it you
  cannot re-run the grading locally, but you can fully audit the grades we
  produced via `verdicts.csv` and its evidence quotes.)
- **`resolver_cache`** — an internal API response cache used during citation
  resolution. It has no audit value and is omitted for size/cleanliness.

All other grading data is included.
