Scientific findings have a curious habit of gaining certainty as they travel. A cautious association in a research paper becomes a definitive headline, and perhaps the justification for public policy. A new AI-assisted scientific reviewer offers a way to interrupt that journey, distinguishing what the evidence shows from what we want it to mean.
Image by ACSH using AI
It is impossible to look at the news today without encountering stories about concerns with artificial intelligence (AI). However, AI can also help researchers work with large amounts of information and complex analyses, as in the health sciences, where extensive datasets and seemingly endless studies challenge human evaluation of evidence and the ability to draw scientifically sound conclusions.
A recently developed AI-assisted system, the Automated Scientific Paper Reviewer (ASP1), reviews scientific papers in a transparent and auditable manner. [1] ASP1 generates a collection of reports that systematically apply predefined methodological standards, focusing on how well the evidence and analyses support causal conclusions. The system functions not as an independent arbiter of scientific truth, but as an assistant to aid independent evaluation.
Correlation Is Easy. Causation Is Hard.
In health policy, thousands of scientific studies carried out each year attempt to determine whether an external factor is associated with a specific outcome. “Correlation is not causation” is a fundamental scientific principle, central to the challenges facing regulators and policymakers, and often difficult to communicate to the public.
Let’s examine one of the most controversial issues in nutrition:
Is ultra-processed food associated with an increased risk of diseases, such as cardiovascular disease, cancer, or dementia—and, if so, does it cause them?
Determining causality is difficult, in part because many health policy studies are observational: researchers examine data on people’s exposures and health outcomes rather than randomly assigning exposures. Such studies often report measures of associationincluding hazard, risk, and odds ratios.
Hazard ratios (HR) compare the instantaneous rate of an event in one group with that in another. For example, a study might compare the rate of cancer diagnosis among people who eat a lot of ultra-processed food with the rate among those who eat very little. An HR of 1 indicates equal hazards; an HR of 0.5 indicates half the hazard; and an HR of 2 indicates twice the hazard.
However, an HR alone cannot determine whether eating ultra-processed food caused cancer. Well-designed observational studies can sometimes support causal inferences, but statistical association by itself cannot tell us what would happen if we intervened to change an exposure. Even a large association may reflect confounding, selection bias, reverse causation, or other explanations.
A confounder is a factor, observed or unobserved, that influences both the studied “exposure” and the outcome and can distort their apparent relationship. Scientists address confounding through randomized assignment when feasible, careful study design, comparisons of people with similar characteristics, and statistical adjustment. The credibility of a causal conclusion depends on how well these methods and their assumptions address alternative explanations.
From Scientific Findings to Policy Decisions
If eating ultra-processed food causes adverse health effects, policies that change consumption patterns might improve health. But what policies? That requires further causal reasoning: What specific intervention is proposed? How would diets change in response? Would people substitute healthier foods, less healthy alternatives, or something else? The distinction matters because good policy requires evidence not only about associations, but also about the likely consequences of alternative actions.
An AI That Shows Its Work: The Automated Scientific Paper Reviewer (ASP1)
When AI-assisted ASP1 evaluates scientific studies, it starts by systematically extracting and classifying the key causal claims and conclusions presented in the study. It then evaluates reporting quality, study design, statistical analysis, potential biases, and how well it justifies causal interpretations. Unlike an unstructured request to a general-purpose chatbot to review a paper, ASP1 uses specialized modules organized around explicit methodological questions. Human reviewers can inspect, question, and check ASP1’s outputs.
ASP1 does not determine truth; it evaluates the support presented in a document, including a source’s inherent omissions and ambiguities. Its output is an input for expert reviewers, not an authoritative score or verdict. A finding that evidence does not establish causation or intervention effectiveness should not be considered a conclusion that no action is justified; nor does strong causal support, by itself, establish that a particular policy is desirable.
The result is a series of reports, called an evidence bundle, consisting of a summary, reports from the relevant review modules, and an overall synthesis and assessment of support for the paper’s conclusions. The process also produces audience-specific evidence briefs that provide concise answers to questions about the article for stakeholders including scientists, lawyers, policymakers, and reporters.
A Real-World Test: Obesity, Dementia, and Survival
To demonstrate ASP1’s results, we ran a recent study discussed in an ACSH article through ASP1. This study reported a surprising association: people who were overweight or obese when diagnosed with dementia subsequently had lower mortality rates than those with normal weight.
That finding did not establish that obesity protects against death or that gaining weight would improve survival. The study’s authors were clear that it is unknown whether this inverse association between BMI and mortality among patients with dementia is causal. They discuss possible explanations, including the possibility that greater body reserves could mitigate malnutrition and eating difficulties, without establishing that mechanism.
To review ASP1’s results with this paper, you can go here and click Open Evidence Bundle and Open Briefs. But here is a portion of the ASP1 summary:
“The study presents a robust association between higher BMI (overweight or obesity) at incident dementia diagnosis and lower subsequent all-cause mortality, in a large, nationally representative cohort of US older adults. The analytic design is careful and transparent, with extensive adjustment for confounding, assessment of major biases (reverse causality, survivor bias, collider bias), and strong alignment between stated findings and statistical evidence. However, caution is urged throughout: the evidence is associative, not causal. The document explicitly warns against using these findings to support weight gain or maintenance as a survival intervention in dementia. Prognostic (risk stratification) use of BMI at diagnosis is reasonably justified, but any clinical or policy recommendations beyond this prognostic insight would overstep the evidentiary base.” [emphasis added]
The reporter brief generated by ASP1 emphasizes not overstating the evidence in the headlines:
What should a headline say without overstating the evidence?
Study reports correlation between higher body weight and lower mortality after dementia diagnosis.
What should a headline not say?
Do not use headlines that imply causation or recommendations, such as: Obesity protects against death in dementia” or “Gaining weight helps dementia patients live longer”
And the caveats to be observed:
The study is observational and cannot prove that higher body weight causes lower mortality in people with dementia. The authors explicitly warn against using these findings to recommend weight gain or maintenance as a treatment. Other factors, such as illness-related weight loss before diagnosis, may partly explain the results.
The brief prepared for attorneys recommends greater legal wording:
Lawyers should describe the findings as “statistical associations” or “correlations” between higher BMI and lower mortality in people with dementia and explicitly state that causality is not established. For example: “The study reports a statistical association between higher BMI at dementia diagnosis and lower subsequent mortality but does not demonstrate that higher BMI causes improved survival.
Better Science, Not Automated Certainty
While ASP1 largely reinforces the researcher’s appropriately cautious interpretation in this instance, its value lies in systematically checking the reasoning throughout, distinguishing associations from causal effects, and clarifying what a study does and does not justify.
More broadly, this AI-assisted approach addresses two major challenges in modern science: evaluating vast amounts of research without confusing correlation with causation and communicating complex findings clearly to different audiences. AI cannot replace scientific judgment, but it can help make that judgment more systematic, clear, and accessible. Coupled with human expertise and critical review, it offers the prospect of a scientific enterprise that learns faster, corrects its mistakes sooner, and makes better use of accumulating evidence. The goal is to make scientific reasoning more reliable, decisions better informed, and public trust better deserved.
[1] A proof-of-concept version is available here and can be run by uploading a PDF file of a scientific article or risk-analysis document and clicking the ‘Click Generate ASP1 Evidence Bundle’ button
Note: Dr. Cox is the developer of ASP1
Source: www.acsh.org
