# Pre-registration — cross-tool faithfulness benchmark (M2)

*Frozen before the full paid panel run. This document fixes the hypothesis, the single
primary endpoint, the exclusion rules, and the analysis/statistics plan so that the
headline result cannot be chosen after seeing the data. It is the publishability spine
required by the M2 extension plan, Part F. Implementation lives in
`scripts/benchmark_grader/faithfulness.py` (`report()` `_statistics` block,
`report_synthesis()`).*

Status: **analysis frozen before the full paid run.** Any change after the first
full-run numbers are seen must be logged as a post-hoc deviation, with the original
plan kept intact.

---

## 1. Hypothesis

AI literature-review tools differ in **citation faithfulness** — whether a paper a tool
cites actually supports the claim it is cited for — and in particular in **numeric
fidelity**, whether the numbers in a claim (%, n=, confidence intervals, p-values,
doses, durations) match the cited paper exactly.

**H1 (primary):** On single-paper numeric claims, **Syno** has a higher numeric-fidelity
pass rate than **Elicit**.

**H0 (null):** The single-paper numeric-fidelity pass rates of Syno and Elicit are equal.

Direction is pre-specified (H1 is one-sided in intent) but the test statistic is reported
two-sided; the CI on the difference is the headline, not a single p-value.

---

## 2. Primary endpoint (the ONE frozen comparison)

> **Primary endpoint = single-paper `numeric_fidelity` pass rate, Syno vs Elicit.**

- **Unit:** a (claim, cited-paper) pair that cites **exactly one** resolved paper
  (`single_paper_only=True`) and carries a numeric signal (`numeric_only=True`).
- **Pass:** the panel's `numeric_fidelity` axis = PASS (axis value 1). PARTIAL and FAIL
  are not-pass. NA pairs (claim states no numbers) are out of the numeric denominator.
- **Why Syno vs Elicit only:** these are the only two tools with adequate single-paper
  numeric n. Consensus is synthesis-only (Section 5). Gemini's numbers are
  vision-extracted and flagged low-confidence — excluded from the primary endpoint.
- **Estimator:** per-tool Wilson 95% score interval on the pass rate, plus the raw
  two-proportion difference and its (uncorrected) p-value, AND the clustered-by-question
  comparison (Section 6) which is the inference-grade test given the clustering.

Everything else is **secondary / exploratory** (Section 4).

---

## 3. Frozen exclusion rules

Applied identically to every tool (the fairness invariant). These are fixed before the
run and may not be relaxed to chase a result.

1. **Single-paper restriction** for the per-paper numeric metric: only (claim, paper)
   units that cite exactly one resolved paper. Multi-paper claims go to the synthesis
   mode (Section 5), never to the single-paper metric.
2. **`looks_tabular` exclusion:** sentences that are shredded PDF table rows
   (collapsed column headers, broken `et al` wraps, a marker repeated ≥3×) are not
   claims and are dropped — they would auto-FAIL against any paper and unfairly deflate
   sentence-based tools.
3. **`source_unavailable` exclusion:** a (claim, paper) pair whose paper text is
   unavailable (404 / empty segments / no abstract fallback) is marked NOT_ASSESSABLE
   and **excluded from every rate denominator** — for every tool, by the same rule.
4. **Consensus is synthesis-only:** its house style cites multiple papers per sentence,
   so it has ~no single-paper units. It does not enter the single-paper primary
   endpoint; it is reported under synthesis mode with its own CI and a small-n note.
5. **Gemini numeric is low-confidence / excluded from the primary endpoint:** Gemini's
   numbers are recovered by vision/OCR (image-rendered math). Its numeric axis is
   reported BOTH ways (vision-extracted-and-graded vs excluded) as a sensitivity check;
   the headline must not depend on the OCR choice.

---

## 4. Secondary / exploratory analyses

All of the following are **secondary** and reported with multiple-comparison correction
(Section 7); none can become the headline:

- Other faithfulness axes: `outcome`, `population`, `recency` (and the rolled-up
  `overall` headline, reported descriptively).
- Other tools / pairings beyond Syno-vs-Elicit.
- **Synthesis (union) mode** metrics for all tools (Section 5).
- Misrepresentation tag frequencies, overreach rate, direction-axis counts.
- Source-tier coverage per tool (full text vs abstract vs unavailable).

---

## 5. Synthesis (union) mode — separate, never pooled

Multi-paper claims (Consensus's house style; also Syno/Elicit multi-cite claims) are
graded against the **union** of their cited papers with a synthesis-aware prompt
(`_SYNTH_SYSTEM` / `_SYNTH_USER`): the judge decides whether the cited papers
**collectively** support the claim; for a span-range, **each endpoint must be
attributable to at least one cited paper** (a range whose bounds no paper states is a
FAIL — this catches fabricated ranges that merely look synthesized).

- Synthesis verdicts are written under a **separate `synth_prompt_sha()` namespace** and
  are **NEVER pooled** with single-paper verdicts — the "collective support" bar is
  intentionally easier than strict per-paper, so pooling would bias the comparison.
- Reported under separate keys (`synth_numeric_fidelity`, `synth_overall_pass_rate`) via
  `report_synthesis()`.
- **Consensus** gets its own Wilson CI here with an explicit **small-n** disclosure; this
  is its only faithfulness number.

---

## 6. Clustered (by question) analysis plan

Claims are nested within ~10 questions and are **NOT independent** (claims from one
question share a topic, a literature, and shared failure modes). A per-claim Wilson CI
ignores this clustering and understates uncertainty.

Plan, frozen:
- Parse `question_id` from each `claim_pk` (`tool:qNN::…`).
- Aggregate to **one pass rate per question per tool** (the question is the unit of
  analysis).
- Report, for Syno vs Elicit, on each axis: per-question pass rates for both tools, the
  **mean per-question difference**, a **cluster-aware CI** on that mean difference
  (normal-approx interval over the K paired question deltas), and a **sign summary**
  (questions where Syno > Elicit, Elicit > Syno, tie) over shared questions.
- The clustered comparison — not the per-claim Wilson interval — is the inference-grade
  statement about the between-tool difference. Per-claim Wilson CIs are reported as
  descriptive only.

Implemented in `clustered_by_question()`.

---

## 7. Multiple-comparison correction

The full grid is 5 axes × 4 tools × 2 modes — far more than one test. The primary
endpoint (Section 2) is exempt (it is the single pre-registered hypothesis). For the
**secondary axis family** (`outcome`, `population`, `recency`, Syno vs Elicit) we apply:

- **Holm–Bonferroni** step-down (FWER control), and
- **Benjamini–Hochberg** (FDR control),

and report **both corrected and raw** p-values. Implemented in `holm_bonferroni()` and
`benjamini_hochberg()`; raw two-proportion p-values from `two_proportion_p()`. A
secondary finding is only called "significant" if it survives Holm at FWER ≤ 0.05.

---

## 8. Inter-judge reliability

Beyond the existing unanimity rate, report **Fleiss' κ per axis** across the panel
judges (`fleiss_kappa()`, surfaced per tool as `fleiss_kappa_per_axis` and overall in
the statistics block). Expectation, disclosed in advance: **high κ on numeric_fidelity**
(an objective number match) and **lower κ on overreach / overall** (a judgment call).
κ is reported, not gated — low κ on a subjective axis is a finding, not a failure.

---

## 9. Judge blinding

The panel must not be able to infer which tool produced a claim. Frozen checks:

- **Hard tool-name tells:** assert that no tool name (`syno`, `elicit`, `consensus`,
  `gemini`) appears in the claim text or passage text fed to `_judge_one`
  (`judge_blinding_check()` / `_sampled_blinding_check()`); this must be all-zero.
- **Residual stylometric leak (disclosed, not asserted away):** Syno's **atomized**
  claims tend to be short single-fact sentences, while competitors' claims are longer
  narrative sentences. This is a residual tell the panel could in principle exploit and
  is **not** removable by token substitution. We disclose it (and report the mean Syno
  claim word count as a data-backed signal) rather than claim it neutralized — mirroring
  how the SANRA narrative grader (L0) neutralizes citation-format tells via `blind_text`.

---

## 10. Sensitivity checks (report all; headline must survive each)

- With / without the `looks_tabular` exclusion.
- With / without the single-paper restriction.
- Gemini numeric: vision-extracted-and-graded vs excluded.
- Hostile variant: treat `unverifiable` as FAIL.

The primary result (Section 2) must hold in direction across all of these for the
headline claim to stand.

---

## 11. What is frozen vs free

**Frozen now (pre-run):** the hypothesis, the primary endpoint and its tool pair, the
exclusion rules, the clustering unit, the correction methods, the κ and blinding checks,
and the sensitivity-check list.

**Free (data-dependent, reported honestly):** the observed rates, CIs, κ values, and
which secondary findings survive correction — i.e. the results themselves, not the
analysis that produces them.
