09. Science and Clinical Data — Integrity Screening

The Application of Distribution Tests

Digit-distribution tests serve as a diagnostic tool for researchers investigating the integrity of published scientific literature. Andreas Diekmann noted in his 2007 study, "Not the First Digit! Using Benford's Law to Detect Fraudulent Scientific Data," that these methods provide a unique lens for identifying fabrication. His work demonstrated that while faked first digits might mimic the expected Benford’s Law decline, the second, third, and fourth digits often reveal significant deviations. This research highlights that a screening process must look beyond the most prominent digit to gain a clearer understanding of the underlying numerical structure. By examining these deeper distribution patterns, scientists can better evaluate whether the reported data aligns with the statistical expectations of natural phenomena. The utility of this instrument lies in its ability to highlight anomalies that warrant closer inspection by the broader scientific community.

Screening Clinical Trial Results

John B. Carlisle provided a rigorous demonstration of these principles in his 2012 paper, "The Analysis of 168 Randomised Controlled Trials to Test Data Integrity." He examined 168 randomised controlled trials by a single author, covering the period between 1991 and 2011, to determine if the reported statistics were consistent with chance. The results were striking: 28 of the 33 tested variables, representing 85% of the data, showed distributions that were statistically inconsistent with the expected patterns. This multi-method statistical screen, which incorporated digit-based tests, served as the primary trigger for a deeper scrutiny of the author’s work. The subsequent investigation led to a significant mass-retraction case, proving the effectiveness of these screening tools in maintaining the standards of clinical research. The methodology serves as a warning that aggregated data should behave in predictable ways, and departures from these norms can signal a need for institutional review.

The Limits of Statistical Suspicion

There is no serious dispute found regarding the findings presented in the papers authored by either Andreas Diekmann or John B. Carlisle. Both researchers explicitly frame their statistical results as a trigger for further scrutiny rather than as a definitive verdict of fraud. The danger remains that a statistic strong enough to convince a reader of an anomaly is also strong enough to unfairly convict the innocent.

Example: A smoke detector indicates the presence of fire but does not identify the specific source of the heat or the intent of the person who lit the match.

Verdict. A positive result on a Benford’s Law screen represents a call for a manual audit, not a final judgment of academic misconduct.

Overturned if: A replication of the clinical trial data shows that the reported distributions were actually consistent with the underlying trial design and variables.

Sources. 2007 · Andreas Diekmann, "Not the First Digit! Using Benford's Law to Detect Fraudulent Scientific Data" — Journal of Applied Statistics 34(3) · 2012 · John B. Carlisle, "The Analysis of 168 Randomised Controlled Trials to Test Data Integrity" — Anaesthesia, Wiley