United States

&lt;p&gt;Niklas Bobrovitz,&lt;sup&gt;1,2&lt;/sup&gt; Harriet Ware,&lt;sup&gt;2&lt;/sup&gt; Corson Johnstone,&lt;sup&gt;3&lt;/sup&gt; Juliane Kennett,&lt;sup&gt;3&lt;/sup&gt; Stephana Julia Moss,&lt;sup&gt;4&lt;/sup&gt; Liam Whalen-Browne,&lt;sup&gt;3&lt;/sup&gt; Faizan Khan,&lt;sup&gt;5&lt;/sup&gt; Benjamin Fletcher,&lt;sup&gt;2&lt;/sup&gt; Daniel J. Niven,&lt;sup&gt;3,6,7&lt;/sup&gt; Henry T. Stelfox&lt;sup&gt;3,6,8&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;To synthesize evidence on the prevalence and measurement of reproducibility in health sciences research.&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;This was a systematic review with meta-analysis registered a priori.&lt;sup&gt;1&lt;/sup&gt; We searched MEDLINE, Embase, Web of Science, the Cochrane Controlled Trials Registry and Database of Systematic Reviews, and grey literature published from January 1, 2000, to May 31, 2023. We included English-language articles in the health sciences that reported the prevalence of reproducible research or the prevalence of studies dedicated to reproducing research. We included studies published after 2000 to ensure relevance to current practice. Two authors independently screened articles in duplicate. Extractions were completed by one reviewer and verified by a second. The prevalence of reproducible research was defined as&lt;i&gt; &lt;/i&gt;the proportion of research findings or studies for which the methods could be exactly repeated (methodological), the results could be corroborated by a study using the same or similar methods (results), or qualitatively similar conclusions could be made based on a study reproduction (inferential). Risk of bias was assessed using a modified Joanna Briggs Institute checklist. We meta-analyzed estimates using random effects models. To explore between-study heterogeneity, we conducted univariable and multivariable meta-regression using generalized linear mixed models. The level of significance was set at &lt;i&gt;P&lt;/i&gt; &amp;lt; .05.&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;A total of 177 studies published between 2001 and 2023 were included, covering all health sciences domains: biomedical/preclinical (n = 56), clinical (n = 106), health/social care services (n = 2), population health (n = 5), and multiple domains (n = 8). Studies reported on 3 types of reproducibility: methodological (n = 68), results (n = 129), and inferential (n = 5). There were 7 methodological approaches for conducting reproducibility research and 38 metrics to quantify reproducibility. The pooled estimated prevalence of reproducible research was 36.2% (95% CI, 29.7%-43.3%; &lt;i&gt;I&lt;/i&gt;&lt;sup&gt;2&lt;/sup&gt; = 99.8%; n = 180 estimates) with variation by research domain: 30.3% (95% CI, 20.3%-42.6%; n = 59 estimates) for biomedical/preclinical, 38.3% (95% CI, 30.0%-47.4%; n = 106 estimates) for clinical, 66.6% (95% CI, 20.7%-93.9%; n = 3 estimates) for health/social care services, and 36.5% (95% CI, 0.0%-100.0%; n = 3 estimates) for population health. There was a nonsignificant decline over time between 2001 and 2023 of −4.7% (95% CI −10.1 to 1.1%; &lt;i&gt;P&lt;/i&gt; = .11) per year in the odds of reproducibility (&lt;b&gt;Figure 25-1041&lt;/b&gt;). The pooled estimated prevalence of studies dedicated to reproducing research was 1.4% (95% CI, 0.40%-4.10%; &lt;i&gt;I&lt;/i&gt;&lt;sup&gt;2&lt;/sup&gt; = 99.2%; n = 40 estimates). Most estimates were at moderate risk of bias (n = 193), with some at high (n = 25) or low (n = 2) risk. Meta-regression of publication year, research domain, type of reproducibility, metric to quantify reproducibility, and risk of bias explained minimal estimate heterogeneity.&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/58c52ca1e8ab42d080707ad8bf3acf01.png)

&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;We found that an estimated one-third of health sciences research was reproducible and 1 in 100 studies were dedicated to reproducing research. Further investigation is needed to understand heterogeneity in estimates. Standardized strategies for quantifying reproducibility should be developed. Reproducibility studies are needed in health/social care services and population health research.&lt;/p&gt;
&lt;h4&gt;Reference&lt;/h4&gt;
&lt;p&gt;1. Bobrovitz N. The reproducibility of health research. Open Science Framework. 2021. &lt;a href=&quot;https://file:///Users/jamft/Creative%20Cloud%20Files%20Company%20Account%20American%20Medical%20Association%20joe.amft@ama-assn.org%20D656564062986A6B0A495CB8@d81f414362986a64495eb4.e/JAmft%20Shared/CURRENT%20WORK/SPECIAL%20PROJECTS/PRC25/text/osf.io/3fhd9&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;osf.io/3fhd9&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Department of Emergency Medicine, Cumming School of Medicine,&lt;i&gt; &lt;/i&gt;University of Calgary, Canada, njhbobro@ucalgary.ca; &lt;sup&gt;2&lt;/sup&gt;Centre for Health Informatics, Cumming School of Medicine,&lt;i&gt; &lt;/i&gt;University of Calgary, Canada; &lt;sup&gt;3&lt;/sup&gt;Department of Critical Care Medicine, Cumming School of Medicine,&lt;i&gt; &lt;/i&gt;University of Calgary, Canada; &lt;sup&gt;4&lt;/sup&gt;Department of Pediatrics, Division of Infectious Diseases, IWK Health Centre, Dalhousie University, Canada; &lt;sup&gt;5&lt;/sup&gt;Hotchkiss Brain Institute, Cumming School of Medicine, University of Calgary, Canada; &lt;sup&gt;6&lt;/sup&gt;O’Brien Institute for Public Health, University of Calgary, Canada; &lt;sup&gt;7&lt;/sup&gt;Department of Community Health Sciences, Cumming School of Medicine,&lt;i&gt; &lt;/i&gt;University of Calgary, Canada; &lt;sup&gt;8&lt;/sup&gt;Faculty of Medicine and Dentistry, University of Alberta, Canada.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt;
&lt;p&gt; None reported. &lt;/p&gt;
&lt;h4&gt;Funding/Support &lt;/h4&gt;
&lt;p&gt;The work was funded by the Canadian Institutes of Health Research. &lt;/p&gt;
&lt;h4&gt;Role of the Funder/Sponsor&lt;/h4&gt;
&lt;p&gt; The funding body had no role in the design or conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the abstract; or decision to submit the abstract for presentation.&lt;/p&gt;


The 10th Peer Review Congress

[V] Prevalence of Reproducible Health Sciences Research: A Systematic Review and Meta-Analysis

research methods

open science

reproducibility

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Niklas Bobrovitz,<sup>1,2</sup> Harriet Ware,<sup>2</sup> Corson Johnstone,<sup>3</sup> Juliane Kennett,<sup>3</sup> Stephana Julia Moss,<sup>4</sup> Liam Whalen-Browne,<sup>3</sup> Faizan Khan,<sup>5</sup> Benjamin Fletcher,<sup>2</sup> Daniel J. Niven,<sup>3,6,7</sup> Henry T. Stelfox<sup>3,6,8</sup></p>
<h4>Objective </h4>
<p>To synthesize evidence on the prevalence and measurement of reproducibility in health sciences research.</p>
<h4>Design </h4>
<p>This was a systematic review with meta-analysis registered a priori.<sup>1</sup> We searched MEDLINE, Embase, Web of Science, the Cochrane Controlled Trials Registry and Database of Systematic Reviews, and grey literature published from January 1, 2000, to May 31, 2023. We included English-language articles in the health sciences that reported the prevalence of reproducible research or the prevalence of studies dedicated to reproducing research. We included studies published after 2000 to ensure relevance to current practice. Two authors independently screened articles in duplicate. Extractions were completed by one reviewer and verified by a second. The prevalence of reproducible research was defined as<i> </i>the proportion of research findings or studies for which the methods could be exactly repeated (methodological), the results could be corroborated by a study using the same or similar methods (results), or qualitatively similar conclusions could be made based on a study reproduction (inferential). Risk of bias was assessed using a modified Joanna Briggs Institute checklist. We meta-analyzed estimates using random effects models. To explore between-study heterogeneity, we conducted univariable and multivariable meta-regression using generalized linear mixed models. The level of significance was set at <i>P</i> &lt; .05.</p>
<h4>Results </h4>
<p>A total of 177 studies published between 2001 and 2023 were included, covering all health sciences domains: biomedical/preclinical (n = 56), clinical (n = 106), health/social care services (n = 2), population health (n = 5), and multiple domains (n = 8). Studies reported on 3 types of reproducibility: methodological (n = 68), results (n = 129), and inferential (n = 5). There were 7 methodological approaches for conducting reproducibility research and 38 metrics to quantify reproducibility. The pooled estimated prevalence of reproducible research was 36.2% (95% CI, 29.7%-43.3%; <i>I</i><sup>2</sup> = 99.8%; n = 180 estimates) with variation by research domain: 30.3% (95% CI, 20.3%-42.6%; n = 59 estimates) for biomedical/preclinical, 38.3% (95% CI, 30.0%-47.4%; n = 106 estimates) for clinical, 66.6% (95% CI, 20.7%-93.9%; n = 3 estimates) for health/social care services, and 36.5% (95% CI, 0.0%-100.0%; n = 3 estimates) for population health. There was a nonsignificant decline over time between 2001 and 2023 of −4.7% (95% CI −10.1 to 1.1%; <i>P</i> = .11) per year in the odds of reproducibility (<b>Figure 25-1041</b>). The pooled estimated prevalence of studies dedicated to reproducing research was 1.4% (95% CI, 0.40%-4.10%; <i>I</i><sup>2</sup> = 99.2%; n = 40 estimates). Most estimates were at moderate risk of bias (n = 193), with some at high (n = 25) or low (n = 2) risk. Meta-regression of publication year, research domain, type of reproducibility, metric to quantify reproducibility, and risk of bias explained minimal estimate heterogeneity.</p>

![](https://assets.underline.io/markdown_image/1/image/58c52ca1e8ab42d080707ad8bf3acf01.png)

<h4>Conclusions </h4>
<p>We found that an estimated one-third of health sciences research was reproducible and 1 in 100 studies were dedicated to reproducing research. Further investigation is needed to understand heterogeneity in estimates. Standardized strategies for quantifying reproducibility should be developed. Reproducibility studies are needed in health/social care services and population health research.</p>
<h4>Reference</h4>
<p>1. Bobrovitz N. The reproducibility of health research. Open Science Framework. 2021. <a href="https://file:///Users/jamft/Creative%20Cloud%20Files%20Company%20Account%20American%20Medical%20Association%20joe.amft@ama-assn.org%20D656564062986A6B0A495CB8@d81f414362986a64495eb4.e/JAmft%20Shared/CURRENT%20WORK/SPECIAL%20PROJECTS/PRC25/text/osf.io/3fhd9"><span class="Hyperlink CharOverride-6">osf.io/3fhd9</span></a></p>
<sup>1</sup>Department of Emergency Medicine, Cumming School of Medicine,<i> </i>University of Calgary, Canada, njhbobro@ucalgary.ca; <sup>2</sup>Centre for Health Informatics, Cumming School of Medicine,<i> </i>University of Calgary, Canada; <sup>3</sup>Department of Critical Care Medicine, Cumming School of Medicine,<i> </i>University of Calgary, Canada; <sup>4</sup>Department of Pediatrics, Division of Infectious Diseases, IWK Health Centre, Dalhousie University, Canada; <sup>5</sup>Hotchkiss Brain Institute, Cumming School of Medicine, University of Calgary, Canada; <sup>6</sup>O’Brien Institute for Public Health, University of Calgary, Canada; <sup>7</sup>Department of Community Health Sciences, Cumming School of Medicine,<i> </i>University of Calgary, Canada; <sup>8</sup>Faculty of Medicine and Dentistry, University of Alberta, Canada.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported. </p>
<h4>Funding/Support </h4>
<p>The work was funded by the Canadian Institutes of Health Research. </p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funding body had no role in the design or conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the abstract; or decision to submit the abstract for presentation.</p>


poster

<p>Emilie Gunn,<sup>1</sup> Kelly Brooks,<sup>1</sup> Stephanie Valladao<sup>1</sup></p>
<h4>Objective </h4>
<p>Prior research has shown the effects of shortening a review deadline on overall time to review.<sup>1</sup> We wanted to determine whether extending the standard 2-week deadline for peer review in a scholarly journal would have positive outcomes. In particular, we wanted to test the hypothesis that offering a 21-day deadline would increase the reviewer conversion rate, thus reducing the number of reviewers invited before getting 2 reviewers to agree. A secondary end point was the number of days it took reviewers to complete their reviews.</p>
<h4>Design </h4>
<p>In this pilot study, 593 review invitations for 95 manuscripts were sent from <i>JCO Oncology Practice</i> between July 1, 2024, and September 30, 2024, with a 21-day deadline. This was compared with 606 invitations for 116 manuscripts sent between July 1, 2023, and September 30, 2023, with a 14-day deadline. Only Original Reports and Reviews were studied because these article types require at least 2 reviewers. Using the Editorial Manager submission system, editors invited reviewers using the normal process, except that reviewers were offered 21 days instead of the industry-standard 14 to complete their reviews. Reviewers received information regarding the 21-day time frame at initial invitation, at the time of review invitation acceptance, and via reminders 1 week and 2 days prior to the deadline. The goal for each paper was to obtain 2 reviewers.</p>
<h4>Results </h4>
<p>When the time allowed to submit a review was increased to 21 days, the number of reviewers invited per manuscript increased by 0.5 reviewers (10.4%). The time it took reviewers to accept a review invitation increased by 0.1 days (12.5%), measured as the number of days from the date the invitation was sent to the date the reviewer agreed <br />(<b>Table 25-0847</b>). The percentage of late reviews decreased by 5% (from 27% to 22%). The median (IQR) time to submit a review was 19 (11-33) days, compared with 13 (5-18) days during the control period. Additionally, late reviews were submitted an average of 11.1 days late, a 29.1% increase. However, median (IQR) time from submission to first decision decreased by 4.7% to 41 (31-55) days from 43 (30-65.5) days.</p>

![](https://assets.underline.io/markdown_image/1/image/280d4f142fd24710f891e59a1d23acc4.png)

<h4>Conclusions </h4>
<p>Extending the 2-week deadline for peer review did not decrease the number of reviewers invited before getting 2 to agree. When the allowed time was increased, the time taken to review increased as well. Although there were fewer late reviews overall, the reviews submitted past the deadline were later than those submitted under the 14-day deadline. We ultimately recommended that the journal continue with the current 2-week deadline for reviewers. Further research is needed in this area.</p>
<h4>Reference</h4>
<p>1. Cochran A, Guertin L. Affecting editor behavior with data: a case study. <i>Sci Editor</i>. 2014;37(1):117-119.</p>
<sup>1</sup>American Society of Clinical Oncology, Alexandria, VA, US, emilie. gunn@asco.org. <h4>Conflict of Interest Disclosures</h4> None reported. <h4>Acknowledgment</h4> We thank Jeffrey Peppercorn for his help with designing this abstract and Scott Marchese for his statistical expertise.



Results of Testing the Gold Standard 2-Week Reviewer Deadline

<p>Emmanuel A. Zavalis,<sup>1</sup>Love Ahnström,<sup>2</sup><Natasha Olsson,<sup>2</sup>Gustav Nilsonne<sup>2,3</sup></p>
<p><h4>Objective</h4> The objective of this study was to assess lead times and application decisions using the national database of the Swedish Ethical Review Authority since its inception. We also aimed to assess the effect of the COVID-19 ethical review fast-track implemented in spring 2020. Prior studies of ethical review have been limited to single or few institutions.<sup>1,2</sup></p>
<p><h4>Design</h4> This retrospective analysis examined 21,962 administrative records of all ethical review applications submitted in Sweden between 2021 and 2023. The applications were categorized by type (eg, amendments, clinical trial, or processing sensitive data) and sponsor (eg, health care systems, universities, or corporations). Kaplan-Meier curves were constructed for analysis of time to decision. Adherence to statutory deadlines was assessed for amendments (35 days) and drug trials (60 days).<sup>3</sup> Log-rank test was used for statistical testing with α = .05 as a significance threshold.</p>
<p><h4>Results</h4> Following exclusions (eg, incomplete applications, unpaid fees, or withdrawals), 20,440 applications remained: 5866 from 2021, 6926 from 2022, and 7648 from 2023. Submissions were primarily from health care systems (10,456 [51%]) and universities (8681 [42%]), with corporations accounting for 578 applications (3%). Amendment applications constituted the largest category of applications (51% in 2021, 50% in 2022, and 44% in 2023). Applications in the “processing of personal data” category doubled from 736 to 1574 over the study period (<b>Figure 25-0966</b>, A). Of the 20,440 applications, initial approvals accounted for 68% (n = 13,864), further information or change to the project was requested in 24% (n = 5001) of cases, and 1.5% (n = 313) were rejected. Final decisions resulted in approvals for 91% (n = 18,516), with rejections being rare (2%), as seen in <b>Figure 25-0966</b>, B. Geographic variation was observed, with Stockholm having the shortest (34 days) and Lund the longest (43 days) median lead times. COVID-19 studies were associated with expedited processing times (median time to decision, 51 vs 63 days; P &lt; .001). While 27% of amendment applications were without a decision after 35 days, 39% of original applications as well as 58% of drug trials were without a decision after 60 days (<b>Figure 25-0966</b>, C).</p>

![](https://assets.underline.io/markdown_image/1/image/0bbf11d39a2fb72d73a5e55c57749cc8.png)

<p><span class="text-bold"><h4>Conclusions</h4> </span>The Swedish Ethical Review Authority mainly handled amendments. Lead times were longer than statutory deadlines for more than half of drug trials and for a quarter of amendments. The processing times showed geographic variations, and the fast-track for COVID-19 showed a significant association with processing times. Further studies of the qualitative aspects of ethical reviews’ impact on research plans as well as the nature of the applications for amendments would be of interest for a better understanding of the value introduced and the harms prevented by ethical review.</p>
<h4>References</h4>
<p>1. Hall DE, Hanusa BH, Stone RA, Ling BS, Arnold RM. Time required for institutional review board review at one Veterans Affairs medical center. <i>JAMA Surg</i>. 2015;150(2):103. doi:10.1001/jamasurg.2014.956</p>
<p>2. Varley PR, Feske U, Gao S, et al. Time required to review research protocols at 10 Veterans Affairs institutional review boards. <i>J Surg Res</i>. 2016;204(2):481-489. doi:10.1016/j.jss.2016.06.004</p>
<p>3. Sveriges Riksdag. Ordinance (2003:615) on ethical review of research involving humans. Accessed January 18, 2025. <a href="https://www.riksdagen.se/sv/dokument-och-lagar/dokument/svensk-forfattningssamling/forordning-2003615-om-etikprovning-av-forskning_sfs-2003-615/">https://www.riksdagen.se/sv/dokument-och-lagar/dokument/svensk-forfattningssamling/forordning-2003615-om-etikprovning-av-forskning_sfs-2003-615/</a></p>
<sup>1</sup>Department of Physiology and Pharmacology, Karolinska Institutet, Stockholm, Sweden; <sup>2</sup>Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden, gustav.nilsonne@ki.se; <sup>3</sup>Department of Psychology, Stockholm University, Stockholm, Sweden. <h4>Conflict of Interest Disclosures</h4> None reported.



Analysis of Decisions and Lead-Time in Ethical Review Boards in Sweden

<p>Christopher S. Cotton,<sup>1</sup> Abid Alam,<sup>1</sup> Sophie Tosta,<sup>2</sup> Timothy G. Buchman,<sup>3</sup> David M. Maslove<sup>4,5,6</sup></p>
<h4>Objective</h4>
<p> Peer review is a cornerstone of academic publishing, providing a bulwark against the dissemination of flawed or unethical research. Historically, peer review has been carried out by experts on a volunteer basis. Financial compensation for peer reviewers has been discussed but with little empirical evidence to inform the debate.<sup>1</sup> A single experimental study<sup>2</sup> has examined this question, finding that for peer review at an economics journal, offering a $100 gift card to reviewers reduced median review time by 8 days. No experimental study to our knowledge has looked at the effect of paying for peer review at a medical journal. Our objective was to determine the effect of a cash incentive on the rate of fulfillment of peer review requests at a medical journal.</p>
<h4>Design </h4>
<p>We conducted a quasi-randomized experimental study at <i>Critical Care Medicine</i>, a specialty medical journal. We divided the experimental period (September 2023 to March 2024) into alternating 2-week blocks, during which all peer review requests sent out to prospective reviewers included either an offer of $250 to complete the peer review assignment (incentive blocks) or no offer (control blocks). Only original research reports were included in the study, with the monetary incentive applying only to the initial review. The primary outcome was the rate of invitation –to completed review conversion, defined as the ratio of reviews submitted divided by the number of reviewer invitations sent out. Review quality was a secondary outcome.</p>
<h4>Results</h4>
<p> During the study, 131 manuscripts went out for review, with a median (IQR) of 5 (2-8) reviewer invitations per manuscript. Of 715 reviewer invitations, 414 (58%) included an incentive offer. There was no difference in the proportion of invitations that were accepted (control vs incentive, 144 of 301 [48%] vs 218 of 414 [53%]; <i>P </i>= .20), but a difference was seen in the primary outcome, with reviewers in the intervention group having a greater odds of completing the review compared with those in the control group (control vs incentive, 127 of 301 [42%] vs 206 of 414 [50%]; odds ratio, 1.36; 95% CI. 1.01-1.84). A Cox proportional hazards model showed that incentivized invitations were fulfilled slightly faster (median [IQR] time, 11 [6-14] days vs 12 [7-14] days; hazard ratio, 1.30; 95% CI, 1.04-1.62). There was no difference in the quality of reviewer reports, as adjudicated by the handling editors (control vs incentive, median [IQR] score on a 100-point scale, 75 [70-85] vs 75 [66-90]).</p>

![](https://assets.underline.io/markdown_image/1/image/7a9a7744d70d458b4a46f19a2b4e1870.png)

<h4>Conclusions</h4>
<p> Our findings suggest that cash payments can incentivize peer review at a medical journal, potentially providing a method to help journals expedite peer review in situations like public health emergencies. Important policy questions remain around cost-benefit considerations; extrapolating from the rate of reviews required during the study period, we estimate that offering $250 for each review would cost <i>Critical Care Medicine </i>nearly $150,000 annually. Other considerations include sources of funding for paid reviews, the effect of nonmonetary incentives, and the potential unintended consequences of paid reviews.</p>
<h4>References</h4>
<p>1. Cheah PY, Piasecki J. Should peer reviewers be paid to review academic papers? <i>Lancet</i>. 2022;399(10335):P1601. doi: 10.1016/S0140-6736(21)02804-X</p>
<p>2. Chetty R, Saez E, Sándor L. What policies increase prosocial behavior? an experiment with referees at the Journal of Public Economics. <i>J Econ Perspect</i>. 2104;28(3):169-188. doi: 10.1257/jep.28.3.169</p>
<sup>1</sup>Department of Economics, Queen’s University, Kingston, ON, Canada; <sup>2</sup>Society of Critical Care Medicine, Mount Prospect, IL, US; <sup>3</sup>Emory Critical Care Center, Emory University School of Medicine, Atlanta, GA, US; <sup>4</sup>Department of Critical Care Medicine, Queen’s University, Kingston, ON, Canada, david.maslove@queensu.ca; <sup>5</sup>Department of Medicine, Queen’s University, Kingston, ON, Canada, <sup>6</sup>Kingston Health Sciences Centre, Kingston, ON, Canada. <h4>Conflict of Interest Disclosures</h4> David M. Maslove was an associate editor at <i>Critical Care Medicine</i> at the time the study was conducted. Timothy G. Buchman was the editor-in-chief at </i>Critical Care Medicine</i> at the time the study was conducted. Sophie Tosta is the managing editor of <i>Critical Care Medicine</i>. <h4>Funding/Support</h4> This study was supported by a grant from the Government of Canada’s New Frontiers in Research Fund (grant No. NFRFR-2021-00335). David M. Maslove is supported by a New Clinician Scientist Award from the Southeastern Ontario Academic Medical Association. Christopher S. Cotton receives research funding as the Jarislowsky-Deutsch Chair in Economic & Financial Policy at Queen’s University. <h4>Role of Funder/Sponsor</h4> No funder had a role in the design, analysis, or interpretation of the study. <h4>Acknowledgment</h4> The authors wish to thank the associate editors and senior editors at <i>Critical Care Medicine</i>, as well as the editorial assistants at the journal, and the staff at the Society of Critical Care Medicine for facilitating the study.



Monetary Incentives for Peer Review at a Medical Journal: A Quasi-Randomized Experimental Study

<p>Sara Schroter,<sup>1,2</sup> Rebecca Harmston,<sup>3</sup> Emma Doble,<sup>1</sup> Sophie Cook,<sup>1</sup> Amy Price<sup>4,5</sup></p>
<h4>Objective</h4>
<p> Calls for patient partners to be paid for their contributions to the health sector are growing.<sup>1</sup> <i>The BMJ</i> invites patients and the public, as well as academic and clinical peer reviewers, to review manuscripts.<sup>2</sup> On November 28, 2024, <i>BMJ</i> announced it would remunerate its patient and public reviewers from 2025.<sup>3</sup>We surveyed these reviewers to capture their perspectives on the introduction of remuneration and overall experience of reviewing.</p>
<h4>Design</h4>
<p> Survey of current patient and public reviewers (those who had completed a review in the previous 3 years) on SurveyMonkey on December 11, 2024. We asked reviewers about their experiences of reviewing and their perspectives on the introduction of remuneration (£50 or an online subscription to a BMJ journal was provided for each completed review).</p>
<h4>Results</h4>
<p>We received a response from 183 of 267 invited reviewers (69%). Independent <i>t</i> tests showed that respondents, compared with nonresponders, received more invitations (mean: 14.0 vs 10.3; <i>P </i>= .01) and completed more reviews (mean: 7.7 vs 3.7; <i>P</i> &lt; .001), but there was no difference in 5-point review quality score (mean: 3.8 vs 3.8; <i>P </i>= .81). The majority of respondents were based in the UK and the US. Respondents were positive about their experience of reviewing: 84% (n = 154) reported it as a very good or good experience. The primary motivation for 93% of reviewers (n = 170) was contributing patient perspectives to the research process. Regarding remuneration, 87 (48%) and 58 (32%) indicated they would be more likely to review a manuscript if they were offered £50 or a 12-month online subscription to one of our journals, respectively. However, approximately one-third said these incentives would not make them more likely to review (60 [33%] payment; 71 [39%] journal subscription), and 30 (16%) and 48 (26%), respectively, were unsure. Half of the participants (n = 93 [51%]) thought offering the choice of these incentives would help attract a more diverse sample of reviewers, 69 (38%) were unsure, and 15 (8%) did not. Views on remuneration were divergent (<b>Table 25-0908</b>) with some seeing it as a recognition of value and others considering it unnecessary. While 107 (58%) had no concerns about introducing payment for patient and public reviewers, 37 (20%) did have concerns and 33 (18%) were unsure. Concerns included potential conflicts of interest, tax implications and effort of declaring, impact on means-tested benefits, effect on review quality and integrity, people becoming reviewers for the wrong reasons, and high foreign exchange banking fees. Others viewed the remuneration as a modest token unlikely to influence quality. Respondents emphasized the importance of optional incentives to accommodate individual preferences and suggested monitoring the quality of impact after implementation of remuneration.</p>

![](https://assets.underline.io/markdown_image/1/image/e8c666f657a7ccf5964c0a603485b888.png)

<h4>Conclusions</h4>
<p> <i>BMJ</i>’s patient and public reviewers hold diverse views on reviewer remuneration, highlighting the importance of providing flexible, optional incentive choices to accommodate varying individual needs, values, and preferences.</p>
<h4>References </h4>
<p>1. Asante K. Remuneration of African patient partners is an important tool for health justice. <i>BMJ</i>. 2024;387:q2675. doi:10.1136/bmj.q2675</p>
<p>2. Schroter S, Price A, Flemyng E, et al.<i> </i>Perspectives on involvement in the peer-review process: surveys of patient and public reviewers at two journals. <i>BMJ Open.</i> 2018;8(9):e023357. doi:10.1136/bmjopen-2018-023357</p>
<p>3. Doble E, Schroter S, Price A, et al. <i>The BMJ</i> will remunerate patient and public reviewers. <i>BMJ</i>. 2024;387:q2581. doi:10.1136/bmj.q2581</p>

<sup>1</sup>BMJ, BMA House, Tavistock Square, London, UK, sschroter
@bmj.com; <sup>2</sup>Faculty of Public Health and Policy, London School of Hygiene and Tropical Medicine, London, UK; <sup>3</sup>Patient and public reviewer, BMJ, London, UK; <sup>4</sup>Dartmouth Health, NH, US; <sup>5</sup>Colorado School of Public Health, CO, US.
<h4>Conflict of Interest Disclosures</h4> Sara Schroter and Sophie Cook are employed by BMJ, Emma Doble is a freelance patient editor for BMJ, Amy Price was a freelance patient editor for BMJ, and Rebecca Harmston is a patient and public reviewer for BMJ.
<h4>Acknowledgment</h4> We thank Layla Abdulbaki, Shahad Al Mashjari, Shahd Rihawi, Alaa Almuhaini, Hygieia Jacob, Munya Alshaalani, Aliaa Bhahzad, Manvi Upadhaya, Zainab Almutawa, Sarah Khan, Adriana Goracci, and Adrian Yee for piloting the questionnaire, providing feedback on the clarity of the questions, and reviewing the work for readability and relevance. We appreciate all of BMJ’s patient and public reviewers who participated in the study.


Exploring Views on Remuneration for Review: A Survey of BMJ' s Patient and Public Reviewers

<p>Simine Vazire<sup>1</sup></p>
<h4>Importance</h4>
<p> Scientific institutions, including journals, should work to earn trust from the scientific community and from the public. With prominent threats to trust in science around the world, it is especially important to make clear why scientific institutions are worthy of trust. Too often, journals’ reputations are unearned—based on flawed metrics, such as impact factors—or simply the inertia of prestige. But journal prestige can and should be earned.</p>
<h4>Observations</h4>
<p> There are many things journals can do to give the community concrete, verifiable indicators of their priorities. First, journals can thoroughly vet the articles they publish to ensure they are accurate, well-calibrated, and transparently reported. Second, journals can invest in postpublication critique and correction so that when they inevitably publish some things that are wrong or miscalibrated, they correct themselves. Third, journals can encourage audits of their published articles, such as those conducted by the Institute for Replication. </p>
<h4>Conclusions</h4>
<p> Scientific journals will sometimes publish research that turns out to be wrong. That is not a reason to distrust journals or the peer review process. But journals do have an obligation to take reasonable steps to vet research before it is published and to make those efforts visible and verifiable to readers. Journals that invest in and facilitate both prepublication and postpublication quality checks, error detection, and correction are the ones that deserve the most trust and the most prestige.</p>

<sup>1</sup>Melbourne School of Psychological Sciences, University of Melbourne, Parkville, Australia, simine@gmail.com.
<h4>Conflict of Interest Disclosures</h4> None reported.


Journal Prestige Can and Should Be Earned

<p>Isaac Kohane<sup>1,2</sup></p>
<h4>Importance </h4>
<p>The full cycle from manuscript creation through review and then distribution is currently a heavily human-intensive process. If the continually increasing capabilities of artificial intelligence (AI) can replace much of that human effort, the economics, quality, culture, and reproducibility of scientific publishing will be dramatically affected internationally.</p>
<h4>Observations </h4>
<p>AI is entering into the reviewing process in a few journals. This is going to accelerate rapidly. It will perform higher-quality reviews than a larger majority of the current batch of reviewers. Authorship, including what used to be considered as intellectual contributions, will be broadly extended to AI. This may have as much impact on the consequences for institutional promotion as for the publication process. Reviews for reproducibility and accuracy are highly likely to become ubiquitous comprehensive processes rather than the artisanal passion project of a few. Gamesmanship of citation networks as currently practiced will become easily unraveled.</p>
<h4>Conclusions</h4>
<p> AI is going to affect all aspects of the scientific process. Because it impinges on roles for which we attributed respect, financial reward, and trust, the current publishing process is going to be transformed within the decade. Who will lead the transformation depends on a few critical decisions we make in the next 1 to 2 years.</p>

<sup>1</sup>Department of Biomedical Informatics, Harvard Medical School, Boston, MA, US, isaac_kohane@hms.harvard.edu; <sup>2</sup><i>NEJM AI</i>, Waltham, MA, US.
<h4>Conflict of Interest Disclosures</h4> None reported.


A Singular Disruption of Scientific Publishing-AI Proliferation and Blurred Responsibilities of Authors, Reviewers, and Editors

<p>Nicholas G. Evans,<sup>1</sup> Samuel Angelli-Nichols,<sup>1</sup> Emma Chang-Rabley,<sup>2</sup> Yara Omar,<sup>3</sup> Rachel Nas,<sup>4</sup> Mikaela Finnegan,<sup>4</sup> Rocco Casagrande,<sup>4</sup> Emily E. Ricotta<sup>5</sup></p>
<h4>Objective</h4>
<p> To understand the role of preprint copies of scientific papers in public health policy guidance during the COVID-19 pandemic and the potential effects of changes in published versions of record (postprints) on that guidance.<sup>1,2</sup></p>
<h4>Design</h4>
<p> We extracted preprint citations from the Department of Homeland Security Master Question List from February 12, 2020, to February 9, 2021, and the National Institute for Occupational Safety and Health weekly COVID-19 Report from November 6, 2020, to September 17, 2021. Text of the abstract from each preprint was parsed and compared with its postprint using a neural conditional random field model for sentence alignment.<sup>3</sup> Identified sentence-level differences were subject to review and adjudication by the research team. Substantive changes were categorized as either nonsignificant numerical changes, contextual updates, or significant changes. Significant changes were compared against the text of the policy guidance to determine whether the postprint result would have resulted in changed guidance and whether guidance was updated after publication.</p>
<h4>Results</h4>
<p> Our algorithm had a sensitivity of approximately 100% in detecting significant sentence-level changes when compared against a human adjudicator. Of 600 preprints extracted, 486 (81.0%) had associated postprints published during the study period. Our sample had a higher publication rate than global estimates of COVID-19 preprints (21.1%; odds ratio, 3.53; 95% CI, 3.10-4.02). Over time, guidance was slightly more likely to incorporate new postprints than preprints (odds ratio, 1.02; 95% CI, 1.01-1.03). Our model flagged 10,483 sentence-level changes across 464 preprints for review. Of these 464 papers, 329 (70.9%) required adjudication for the significance of their changes and 105 (22.6%) contained changes our team found could potentially change policy guidance. Significant changes included, among others, conclusions about the effectiveness of clinical interventions or nonpharmaceutical interventions to slow the spread of COVID-19, basic epidemiological properties, and the relative efficacy of vaccine candidates against variants of concern. Of 105 significant papers, 44 (41.9%) impacted policy guidance. Guidance that was impacted by significant changes tended to concern therapeutics and public health measures and, less often, changes resulting from the emergence of variants. The remainder involved guidance that lacked granularity of public health guidance relative to the kind of change in the paper’s history. This was followed by a tendency toward conservative public health guidance when presented with competing evidence sources, which meant a single paper’s change did not alter a recommendation. Only 9 papers containing policy-relevant changes were updated.</p>
<h4>Conclusions </h4>
<p>Preprints can provide critical early information to policymakers but can generate unclear or misleading advice in key areas that may persist if not subject to clear and timely updates. While preprints in our sample tended to be of publishable quality, more work is needed to establish how to ensure this quality in future outbreak response.</p>
<h4>References</h4>
<p>1. Nelson L, Ye H, Schwenn A, Lee S, Arabi S, Hutchins BI. Robustness of evidence reported in preprints during peer review. <i>Lancet Global Health.</i> 2022;10(11):e1684-e1687. doi:10.1016/S2214-109X(22)00368-0</p>
<p>2. Collins, Alexander R. Reproducibility of COVID-19 pre-prints. <i>Scientometrics.</i> 2022;127(8):4655-4673. doi:10.1007/s11192-022-04418-2</p>
<p>3. Jiang, Maddela M, Lan W, Zhong Y, Xu W. Neural CRF model for sentence alignment in text simplification. <i>arXiv</i>. Preprint posted online May 5, 2020. doi:10.48550/arXiv.2005.02324</p>

<sup>1</sup>University of Massachusetts Lowell, Lowell, MA, US, nicholas_evans@uml.edu; <sup>2</sup>Emory University, Atlanta, GA, US; <sup>3</sup>Boston University, Boston, Massachusetts, US; <sup>4</sup>Deloitte, Rosslyn, VA, US; <sup>5</sup>Uniformed Services University of the Health Sciences, Bethesda, MD, US.
<h4>Conflict of Interest Disclosures</h4> None reported.
<h4>Funding/Support</h4> The initial funding of this work was provided through the Intramural Division of the National Institute for Allergy and Infectious Diseases. Nicholas G. Evans received funding from the Greenwall Foundation and from the US Air Force Office of Scientific Research, which provided time for the conduct of this research.
<h4>Role of Funder/Sponsor</h4> The funder had no role in the design, conduct, or reporting of this work.
<h4>Disclaimer</h4> Emily E. Ricotta is an employee of the US Department of Defense. This work does not represent the views of her organization or of the US government.
<h4>Additional Information</h4> Emily E. Ricotta is a co–corresponding author (emily.ricotta@usuhs.edu).


Natural Language Processing to Assess the Role of Preprints in COVID-19 Policy Guidance

<p>Lan Jiang,<sup>1</sup> Xiangji Ying,<sup>2</sup> Mengfei Lan,<sup>1</sup> Andrew W. Brown,<sup>3</sup> Colby J. Vorland,<sup>4</sup> Evan Mayo-Wilson,<sup>2</sup> Halil Kilicoglu<sup>1</sup></p>
<h4>Objective</h4>
<p> SPIRIT<sup>1</sup> and CONSORT<sup>2</sup> guidelines recommend the minimum information to report in randomized clinical trial (RCT) protocols and results reports. Natural language processing (NLP) offers a promising approach for assessing whether manuscripts include the recommended information, the method used, and what was found. We annotated RCTs for adherence to SPIRIT 2013 and CONSORT 2010 and used large language models (LLMs) for automated assessment.</p>
<h4>Design</h4>
<p> We annotated 100 protocol/results pairs (200 articles). We included parallel RCTs registered on ClinicalTrials.gov and published in PubMed Central<sup>3</sup> from January 2011 to August 2022. For inclusion criteria and search strategy, see <a href="https://osf.io/nefa9">https://osf.io/nefa9</a>. Annotators rated 119 questions (85 check 1, 34 check all that apply), each related to a specific SPIRIT (n = 95) or CONSORT (n = 71) item or subitem. Questions assessed whether recommended information was reported and what specifically the trial did or found. For example, “Does the manuscript report any results for non-systematically assessed harms?” (Yes, No, or Cannot tell) was check 1; “If the study investigates a drug intervention or the intervention contains a drug component, does the manuscript mention the dosing schedule of the drug?” (Dose, Frequency, How long the drug will be taken, None reported, Not applicable, or No text to assess) was check all that apply. Two experts independently rated and adjudicated the protocols and results articles for 25 trials; the rest were rated by 1 expert. Protocol/results pairs were split 70:10:20 into training, validation, and test sets, respectively. To assess what the RCTs reported, we prompted GPT-4o with each question and its corresponding response options to elicit responses for every question for each article. We assessed model performance on the test set using F1 score, the harmonic mean of positive predictive value, and sensitivity (range: 0-1, higher is better). The mean F1 score measured overall performance across all questions. We compared results to a baseline that selected the most common response for each question in the training set.</p>
<h4>Results</h4>
<p> There was no information for some questions in most articles. For example, only 4% reported informed consent materials. Even commonly reported items often lacked critical details (eg, 36% did not specify procedures for monitoring participant adherence to intervention). Interannotator agreement was 0.94 (Cohen κ). F1 score comparison for the GPT-4o model and the baseline is shown in <b>Table 25-1019</b> (0.856 vs 0.764). The scores ranged from 0.495 to 1 for GPT-4o and from 0.287 to 1 for baseline. Our approach performed comparably on questions related to both the SPIRIT<sup>1</sup> and CONSORT<sup>2</sup> guidelines, with slightly better results for SPIRIT.</p>

![](https://assets.underline.io/markdown_image/1/image/28e2d290e4cc153af61b5dfc3ebf912b.png)

<h4>Conclusions</h4>
<p> LLMs hold promise for evaluating RCT adherence to reporting standards. Our questions might help pinpoint missing items and identify RCT characteristics. LLM-based tools could support peer review workflows, and further user engagement could help identify the SPIRIT and CONSORT items users consider highest priority for systematic assessment.</p>
<h4>References</h4>
<p>1. Chan AW, Tetzlaff JM, Altman DG, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. <i>Ann Intern Med.</i> 2013;158(3):200-207. doi:10.7326/0003-4819-158-3-201302050-00583</p>
<p>2. Schulz KF, Altman DG, Moher D; CONSORT Group. (2010). CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. <i>BMJ</i>. 2010;340:c332. doi:10.1136/bmj.c332</p>
<p>3. Kilicoglu H, Rosemblat G, Hoang L, et al. Toward assessing clinical trial publications for reporting transparency. <i>J Biomed Inform</i>. 2021;116:103717. doi:10.1016/j.jbi.2021.103717</p>

<sup>1</sup>School of Information Sciences, University of Illinois, Urbana-Champaign, Urbana-Champaign, IL, US, lanj3@illinois.edu; <sup>2</sup>Gillings School of Global Public Health, University of North Carolina, Chapel Hill, Chapel Hill, NC, US; <sup>3</sup>University of Arkansas for Medical Sciences and Arkansas Children’s Research Institute, Little Rock, AK, US; <sup>4</sup>School of Public Health, Indiana University, Bloomington, IN, US.
<h4>Conflict of Interest Disclosures</h4> None reported.
<h4>Funding/Support</h4> This study was supported by the US National Library of Medicine of the National Institutes of Health under award number R01LM014079.
<h4>Role of the Funder/Sponsor</h4> The funder had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.
<h4>Additional Information</h4> Halil Kilicoglu is a co–corresponding author (halil@illinois.edu). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.


Leveraging Large Language Models for Assessing the Adherence of Randomized Controlled Trial Publications to Reporting Guidelines

<p>Fangwen Zhou,<sup>1</sup> Muhammad Afzal,<sup>2</sup> Rick Parrish,<sup>1</sup> Ashirbani Saha,<sup>3</sup> R. Brian Haynes,<sup>1</sup> Alfonso Iorio,<sup>1,4</sup> Cynthia Lokker<sup>1</sup></p>
<h4>Objective</h4>
<p> Deep learning models for classifying published clinical literature to support critical appraisal have garnered wide attention.<sup>1</sup> To automate the critical appraisal workflow for McMaster’s Premium Literature Service (PLUS), a gold-standard database manually labeled by experts was used to develop well-performing models to classify the rigor of randomized controlled trials (RCTs).<sup>2</sup> However, due to the complexity of language models, the lack of transparency remains an important concern. This study explored Shapley Additive Explanations (SHAP)<sup>3</sup> to better understand how a deep learning model makes classification decisions.</p>
<h4>Design </h4>
<p>Details regarding the development and evaluation of the rigor classifiers are published elsewhere.<sup>2</sup> Briefly, classifiers were trained with titles and abstracts of original RCTs, that met or did not meet PLUS criteria for rigor (randomization, ≥10 participants per group, ≥80% follow-up, clinically important outcomes, and preplanned subgroup analyses, if applicable). Models were trained on 53,219 PLUS articles from 2003 to 2023, randomly split 80:10:10 into train, validate, and test sets. Articles in Clinical Hedges, a similar database that preceded PLUS, and PLUS articles from 2024 were used for external testing. A top-performing BioLinkBERT model, with an area under the receiver operating characteristic curve of 94%, was selected for this study. The SHAP partition explainer determined important tokens (words/subwords) for articles from the validate and test datasets, Clinical Hedges, and PLUS 2024. Tokens were combined into words for ease of interpretation, and their SHAP values were summed. The mean SHAP values, which indicate the aggregated average marginal contribution to model output over all samples, for the most impactful words with 100 or more occurrences were examined.</p>
<h4>Results </h4>
<p>Overall, 6,207,935 words were analyzed, of which 49,387 were unique. <b>Figure 25-0848</b> shows the 15 most impactful unique words for positive (rigorous) and negative (nonrigorous) classes. Terms such as <i>noninferiority</i> positively influenced rigor predictions with a mean SHAP value of 0.00904 (95% CI, 0.00844-0.00965), whereas <i>nonrandomized</i> (mean SHAP value, −0.02510; 95% CI, −0.03204 to −0.01816) had negative impacts. These results generally align with the manual appraisal criteria. However, tokens apparently unrelated to the criteria, such as tiotropium (mean SHAP value, 0.00571; 95% CI, 0.00508-0.00634), also had sizable impacts, indicating that they may have been correlated with higher rigor during manual appraisal. These patterns were learned and applied by the model, indicating a certain degree of overfitting.</p>

![](https://assets.underline.io/markdown_image/1/image/83c754796d3484f54b7a4aacbdad1110.png)

<h4>Conclusions </h4>
<p>This study demonstrates that SHAP helps understand which features influence a deep learning model’s rigor assessment of an RCT. Identifying influential features increases confidence in the model assessments and generalizability and reduces unease about the black-box nature of these models. Future work should explore SHAP with other critical appraisal tools and datasets and potentially integrate it into machine learning systems to improve user trust and model accountability in evidence synthesis and critical appraisal workflows.</p>
<h4>References</h4>
<p>1. Lokker C, Bagheri E, Abdelkader W, et al. Deep learning to refine the identification of high-quality clinical research articles from the biomedical literature: performance evaluation. <i>J Biomed Inform</i>. 2023;142:104384. doi:10.1016/j.jbi.2023.104384</p>
<p>2. Zhou F, Parrish R, Afzal M, et al. Benchmarking domain-specific pretrained language models to identify the best model for methodological rigor in clinical studies. <i>J Biomed Inform</i>. 2025;166:104825. doi:10.1016/j.jbi.2025.104825</p>
<p>3. Lundberg S, Lee SI. A unified approach to interpreting model predictions. <i>arXiv</i>. Preprint posted online May 22, 2017. doi:10.48550/arXiv.1705.07874</p>

<sup>1</sup>Health Information Research Unit, Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada, lokkerc@mcmaster.ca; <sup>2</sup>Faculty of Computing, Engineering and the Built Environment, Birmingham City University, Birmingham, UK; <sup>3</sup>Department of Oncology, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; <sup>4</sup>Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada.
<h4>Conflict of Interest Disclosures</h4> McMaster University, a nonprofit public academic institution, has contracts through the Health Information Research Unit under the supervision of Alfonso Iorio and R. Brian Haynes. These contracts involve professional and commercial publishers to provide newly published studies, which are critically appraised for research methodology and assessed for clinical relevance as part of the McMaster Premium Literature Service (McMaster PLUS). Cynthia Lokker and Rick Parrish receive partial compensation, and R. Brian Haynes is remunerated for supervisory responsibilities and royalties. Ashirbani Saha, Fangwen Zhou, and Muhammad Afzal are not affiliated with McMaster PLUS.
<h4>Funding/Support</h4> Fangwen Zhou was funded through the Mitacs Business Strategy Internship grant (IT42947) with matching funds from EBSCO Canada.
<h4>Role of the Funder/Sponsor</h4> The funders were not involved in the design and conduct of the study, collection, management, analysis, and interpretation of the data, preparation, review, or approval of the abstract, and decision to submit the abstract for presentation.
<h4>Acknowledgment</h4> We thank the Digital Research Alliance of Canada for providing the computational resources.


Understanding How a Language Model Assesses the Quality of Randomized Controlled Trials: Applying Shapley Additive Explanations to Encoder Transformer Classification Models

<p>Xiangji Ying,<sup>1</sup> Colby J. Vorland,<sup>2</sup> Kiran Ninan,<sup>1</sup> Jean-Pierre Oberste,<sup>1</sup> Andrew W. Brown,<sup>3</sup> Riaz Qureshi,<sup>4</sup> Sirui Zhang,<sup>5</sup> Nicholas J. DeVito,<sup>6</sup> Matthew Page,<sup>7</sup> Ian J. Saldanha,<sup>8</sup> Halil Kilicoglu,<sup>9</sup> Evan Mayo-Wilson<sup>1</sup></p>
<h4>Objective </h4>
<p>Registries, such as ClinicalTrials.gov, and guidelines, including SPIRIT 2025 (Standard Protocol Items: Recommendations for Interventional Trials) and CONSORT 2025 (Consolidated Standards of Reporting Trials), define clinical trial outcomes using 5 elements: domain, measurement, metric, aggregation method, and time point. Changes between prospective registration and results reporting can introduce bias. Readers can manually compare trial documents, but doing so is resource intensive. Automated methods could facilitate checking and improve peer review. We used a chatbot to define outcomes registered on ClinicalTrials.gov, identify changes between prospective registrations and registry results, and describe those changes.</p>
<h4>Design </h4>
<p>We conducted a cross-sectional study using prospectively registered, completed randomized clinical trials from ClinicalTrials.gov (January 2000-January 2024). Building on the 5-element framework, we developed rules and categories to define outcomes and outcome changes. We used GPT-4o, o1, and o3 mini (Open AI) and developed and optimized prompts using a training set of 225 trials (2221 outcomes). We validated performance on 150 trials (1459 outcomes). We divided the task into 16 structured subtasks in the following steps: (1) defining outcomes in both versions, (2) matching outcomes, and (3) detecting changes in outcome elements. We provided cosine and Jaccard distances to help the chatbot match and compare outcomes. We evaluated all outcomes using the 3 chatbot models in January 2025. We selected the answer given by at least 2 of the models. Two human raters independently evaluated the chatbot results. We present preliminary findings based on unreconciled ratings of changes (final findings will be presented at the conference).</p>
<h4>Results </h4>
<p>The 150 validation trials reported a mean (SD) of 6.9 (5.0) outcomes in prospective registrations and 8.4 (6.0) outcomes in final versions. The final versions included 832 outcomes matched with prospective registrations and 428 additional outcomes and omitted 199 outcomes from prospective versions. We achieved 99.8% (95% CI, 99.4%-99.9%) accuracy in matching outcomes between versions (<b>Table 25-1142</b>). Accuracy for detecting changes was 87.9% (95% CI, 85.5%-90.1%) for measurement, 88.1% (95% CI, 85.7%-90.2%) for metric, 87.2% (95% CI, 84.8%-89.4%) for aggregation method, 94.5% (95% CI, 92.7%-95.9%) for cutoff, and 94.8% (95% CI, 93.1%-96.2%) for time point. On average, the language learning models completed the entire process (ie, matching, defining, comparing) in approximately 2 minutes per trial. It took humans 27 minutes per trial to evaluate the chatbot’s response.</p>

![](https://assets.underline.io/markdown_image/1/image/8c0ccf28450d3d85a18008523d330a86.png)

<h4>Conclusions </h4>
<p>A prompt-based approach was highly accurate in defining clinical trial outcomes and identifying outcome changes in ClinicalTrials.gov. This approach could be expanded to identify changes between registrations and manuscripts. Although it did not achieve perfect accuracy, our prompt-based approach could help editors and peer reviewers detect likely discrepancies that warrant further review. </p>

<sup>1</sup>Department of Epidemiology, University of North Carolina Gillings School of Global Public Health, Chapel Hill, NC, US, evan.mayo-wilson@unc.edu; <sup>2</sup>Department of Epidemiology and Biostatistics, Indiana University School of Public Health–Bloomington, IN, US; <sup>3</sup>Department of Biostatistics, University of Arkansas for Medical Sciences; Arkansas Children’s Research Institute, Little Rock, AR, US; <sup>4</sup>Department of Ophthalmology, School of Medicine; Department of Epidemiology, School of Public Health, University of Colorado Anschutz Medical Campus, Denver, CO, US; <sup>5</sup>Department of Epidemiology, School of Public Health, Brown University, Providence, RI, US; <sup>6</sup>The Bennett Institute for Applied Data Science, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK; <sup>7</sup>Methods in Evidence Synthesis Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia; <sup>8</sup>Departments of Epidemiology (Primary) and Health Policy and Management (Joint), Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, US; <sup>9</sup>School of Information Sciences, University of Illinois at Urbana-Champaign, IL, US.

<h4>Conflict of Interest Disclosures</h4> None reported.
<h4>Funding/Support</h4> National Library of Medicine, National Institutes of Health (R01LM014079).
<h4>Role of Funder/Sponsor</h4> The funder played no roles in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation. The views expressed do not necessarily reflect the views of the National Institutes of Health.
<h4>Additional Information</h4> Our protocol is publicly available and we will share prompts, code, and datasets on the Open Science Framework (https://osf.io/2tyh3/).


Next from The 10th Peer Review Congress

Results of Testing the Gold Standard 2-Week Reviewer Deadline

Similar lecture

Factors Associated with the Reproducibility of Health Sciences Research: A Systematic Review and Evidence and Gap Map

Stay up to date with the latest Underline news!

PRODUCT

COMPANY

RESOURCES