# Diabetes Benchmark Questions

A pre-registered catalog of 20 diabetes research questions for the
Syno literature-review benchmark. Two-part design: 5 broad "narrative"
questions that test scoping, weighting, and honest abstention; 15
narrow PICO-style questions that test strict citation-claim
faithfulness on quantitative endpoints.

The same 20 questions will be executed through Syno (with the per-run
naive-vs-refined comparison from round 0 to final), Google Gemini
Deep Research, OpenAI Deep Research, Elicit, Consensus, Perplexity
Deep Research, and Undermind. Outputs will be graded blinded by
≥3 independent raters using the rubric defined per part below.

---

## Part 1 — 5 Broad "Narrative" Questions (Scoping Traps)

**Grading criteria.** SANRA-style: does the AI cover all sub-domains,
avoid cherry-picking, weight evidence by study design, and honestly
abstain where evidence is conflicting or unresolved?

### 1. Nutrition / Remission

Review the current evidence on the durability of type 2 diabetes
remission achieved through intensive lifestyle and dietary
interventions, specifically comparing low-carbohydrate diets to
severe caloric restriction. Synthesize the physiological mechanisms
and long-term (≥3 years) relapse rates.

### 2. Genetics / Screening / Economics

Synthesize the current clinical and health-economic evidence
regarding population-level genetic risk scoring combined with
autoantibody screening for the early detection and prevention of
type 1 diabetes. Explicitly address implementation barriers and
cost-effectiveness.

### 3. Cognitive / Neurological

Provide a comprehensive narrative review of the bidirectional
relationship and shared pathophysiological mechanisms between type
2 diabetes, central insulin resistance, and the development of
Alzheimer's disease or vascular dementia.

### 4. Metabolic Memory

Review the evolution and current state of evidence regarding the
"legacy effect" (metabolic memory) in both type 1 and type 2
diabetes, focusing on how early intensive glycemic control impacts
long-term microvascular and macrovascular outcomes over decades.

### 5. Psychosocial / Diabulimia

Synthesize the literature on the prevalence, clinical outcomes, and
optimal management strategies for eating disorders concurrent with
type 1 diabetes, specifically focusing on deliberate insulin
omission for weight loss (diabulimia).

---

## Part 2 — 15 Narrow PICO Questions (Faithful-Extraction Traps)

**Grading criteria.** Strict per-axis judgment: does the AI
hallucinate numerics, misrepresent the population, or conflate
surrogate outcomes with patient-important outcomes?

### Epidemiology & Disparities

#### 6. Glycemic disparities by race (US)

In non-Hispanic Black adults compared to non-Hispanic White adults
with established type 2 diabetes in the US, what is the standardized
difference in HbA1c and time-in-range (TIR) when controlling for
socioeconomic status and baseline treatment intensity?

### Established Pharmacotherapy

#### 7. SGLT2i MACE in T2D CVOTs

In adults with type 2 diabetes and established atherosclerotic
cardiovascular disease, what is the pooled hazard ratio for the
3-point major adverse cardiovascular event (MACE) composite outcome
for SGLT2 inhibitors versus placebo based strictly on dedicated
cardiovascular outcome trials?

#### 8. Finerenone in diabetic kidney disease (FIDELIO-DKD)

In adults with type 2 diabetes and chronic kidney disease (eGFR
25–75 mL/min/1.73 m²), what is the absolute risk reduction for the
primary kidney composite outcome of finerenone versus placebo, based
strictly on the FIDELIO-DKD primary analysis?

### Newer / Thin Evidence

#### 9. Teplizumab delay of clinical T1D

In non-diabetic individuals with Stage 2 type 1 diabetes, what is
the median delay in months to the clinical onset of Stage 3 disease
following a single 14-day course of teplizumab compared to placebo?

#### 10. Tirzepatide and MACE — dedicated CVOT vs exploratory

In adults with type 2 diabetes, does tirzepatide demonstrate a
statistically significant reduction in major adverse cardiovascular
events (MACE) compared to placebo in dedicated cardiovascular
outcome trials, or is current data limited to exploratory endpoints
from glycemic efficacy trials?

### GLP-1 RAs in Type 1 Diabetes

#### 11. Liraglutide adjunctive — DKA rate (ADJUNCT ONE/TWO)

In adults with type 1 diabetes, what is the effect of liraglutide
added to insulin therapy on the absolute rate of symptomatic
diabetic ketoacidosis (DKA) compared to insulin therapy alone,
citing findings specifically from the ADJUNCT ONE and ADJUNCT TWO
trials?

#### 12. Tirzepatide in T1D with obesity — evidence quality

In adults with type 1 diabetes and concurrent obesity (BMI >30),
what is the quality of evidence (e.g., RCT versus observational
case series) currently supporting the use of tirzepatide for weight
loss, and what is the reported incidence of severe hypoglycemia?

### Mechanisms & Pathophysiology

#### 13. FMT in T2D — human-only, durable effect

In human adults with type 2 diabetes, does fecal microbiota
transplantation (FMT) from healthy donors result in a durable (≥6
months) reduction in HbA1c compared to autologous FMT or placebo?
(Strict rule: exclude all rodent data.)

### Comparative Effectiveness

#### 14. RYGB vs intensive medical therapy — long-term remission

In adults with type 2 diabetes and a BMI between 35 and 40, what
is the rate of long-term diabetes remission (HbA1c <6.5% off all
glucose-lowering medications for ≥5 years) following Roux-en-Y
gastric bypass compared to intensive medical therapy?

#### 15. Closed-loop vs sensor-augmented pump in adolescents

In adolescents (ages 12–18) with type 1 diabetes, what is the mean
difference in time-in-range (70–180 mg/dL) between advanced hybrid
closed-loop systems and standard sensor-augmented pump therapy over
a minimum 6-month period?

### Special Populations

#### 16. CGM in pregnancy with T1D — LGA reduction (CONCEPTT)

In pregnant women with pre-existing type 1 diabetes, does the use
of continuous glucose monitoring (CGM) significantly reduce the
incidence of large-for-gestational-age (LGA) infants compared to
traditional SMBG, based on the CONCEPTT trial?

#### 17. HbA1c target in frail older adults — ADA 2025/2026

In frail older adults (≥75 years) with type 2 diabetes and multiple
comorbidities, what is the specific target HbA1c range recommended
by the ADA 2025/2026 guidelines, and is this target based on RCT
evidence or expert consensus?

### Sex & Hormones

#### 18. Menstrual-cycle insulin sensitivity in T1D

In premenopausal women with type 1 diabetes using CGM, what is the
quantified mean difference in insulin sensitivity and required total
daily insulin dose between the mid-luteal phase and the early
follicular phase of the menstrual cycle?

### Exercise & Sport

#### 19. Pre-exercise basal reduction timing in T1D

In adults with type 1 diabetes participating in moderate-intensity
continuous aerobic exercise, does a 50% basal insulin reduction
implemented 90 minutes prior to exercise significantly reduce the
rate of level-2 hypoglycemia compared to a 50% basal reduction
implemented at the onset of exercise?

### Technology & Equity

#### 20. rtCGM in non-insulin-treated T2D — 12-month HbA1c

In adults with type 2 diabetes treated solely with oral medications
(no insulin), does the use of real-time continuous glucose
monitoring (rtCGM) yield a statistically significant reduction in
HbA1c at 12 months compared to routine clinical care?

---

## Evaluation rubric

### Part 1 (Q1–Q5) — SANRA-style narrative grading

Each rater scores 0–2 on the six SANRA dimensions per question:

1. Justification of the article's importance for the readership
2. Statement of concrete aims or formulation of questions
3. Description of the literature search
4. Referencing
5. Scientific reasoning (e.g. weighting evidence by study design)
6. Appropriate presentation of data

Max score per question: 12. Aggregate the median across raters.

Additionally flag:
- Cherry-picking: did the system cite one supporting study while
  ignoring a more recent or higher-quality contradicting study from
  the same retrieval pool?
- Honest abstention: where the evidence is genuinely conflicting or
  thin, did the system say so or paper over it with confident prose?

### Part 2 (Q6–Q20) — strict per-axis citation faithfulness

Each cited claim is judged on four axes:

- **Population alignment**: does the cited paper's study population
  match what the claim asserts? (exact / narrower / broader /
  mismatch / not applicable)
- **Numerics**: are the reported numbers within the cited paper's
  CI or order of magnitude? (aligned / direction-only / null-result
  reversed / reversed / no-numbers / paper-lacks-numbers)
- **Outcome alignment**: did the cited paper measure the named
  outcome, or a surrogate? (direct / surrogate / different /
  not applicable)
- **Recency**: is the cited paper appropriate for the temporal
  framing of the claim? (no concern / stale / superseded-likely)

Per-question score = number of claims with all four axes "clean",
divided by total claims. Aggregate by median across raters.

Each question is also evaluated on the **coverage** axis: did the
tool find the canonical landmark study (e.g. FIDELIO-DKD for Q8,
CONCEPTT for Q16, ADJUNCT ONE/TWO for Q11), or did it cite
peripheral evidence while missing the obvious anchor?

---

## How this catalog will be used

1. **Pre-registration.** This file is committed to git before any
   benchmark runs. No question is edited, added, or removed after
   the first run begins. Updates happen by creating a versioned
   replacement file (`diabetes-benchmark-questions-v2.md`), never
   by editing in place.

2. **Internal Syno runs.** All 20 questions executed via the
   production-parity thorough pipeline. Each run's round-0 (naive
   draft, no validator feedback yet) and final-round (refined)
   verdicts are preserved as artifacts. Within-Syno naive-vs-refined
   delta is computed automatically — the per-run improvement is the
   built-in baseline that competitor tools don't have access to.

3. **External competitor runs.** Same 20 questions through Gemini
   Deep Research, OpenAI Deep Research, Elicit, Consensus,
   Perplexity Deep Research, and Undermind. Outputs captured as
   structured (citations + prose) artifacts.

4. **Blinded human evaluation.** ≥3 independent raters per question,
   tool identity blinded. Rubric above. Inter-rater agreement
   reported as Krippendorff's α.

---

*Catalog version 1.1 — May 2026. Replaces the prior 46-question
catalog. Saved at `docs/whitepaper/diabetes-benchmark-questions.md`.*
