The Golden Age of Statistics: The Birth of Data Dredging
In the early 20th century, the statistical revolution changed the way humans understood the world. Sir Ronald Fisher, the father of modern statistics, introduced the concept of significance tests and p-values in 1925. His idea was simple: if the probability of an outcome occurring by chance is very low (usually below 0.05), then the outcome is considered statistically significant. However, Fisher himself warned that these tests were only valid if the hypothesis was set before collecting the data. Unfortunately, this warning is often ignored.
In the 1950s and 1960s, computers began to be used in data analysis. With the ability to test thousands of relationships in an instant, researchers fell into "p-hacking" — repeating tests until they found a significant p-value. Statistician John Tukey in 1962 called this phenomenon "data snooping" and warned that it drastically increases the risk of false positives.
The Biggest Scandal: From Psychology to Medicine
One of the most famous cases occurred in social psychology. In 2011, researcher Daryl Bem published a study claiming that humans can predict the future — a form of psychic ability. The study used nine experiments with various statistical tests. Upon review, it was found that only a few tests showed significant effects, while the rest failed. This is a classic example of data dredging: reporting only results that support the hypothesis while hiding others.
In the medical field, a greater tragedy occurred. In 2004, a study on antidepressants for teenagers revealed that pharmaceutical companies only published studies showing positive effects, while studies showing no effect or harmful effects were hidden. As a result, thousands of teenagers were given ineffective and dangerous drugs. This scandal led to reforms in clinical trial registration.
How Data Dredging Works: The Mechanism of Fraud
The process of data dredging usually involves the following steps:
- Collecting data without a specific hypothesis: Researchers take large existing datasets, such as census data or hospital records.
- Testing many correlations: They test thousands of variable pairs — for example, does the number of umbrellas sold correlate with the crime rate? (There may be a correlation, but it doesn't imply causation).
- Selecting significant ones: Only correlations that produce p-values below 0.05 are reported, while others are ignored.
- Adjusting the analysis: Sometimes they change how the data is collected or analyzed — for example, removing outliers, changing variable definitions, or combining groups — so the results become significant.
As a result, one out of 20 tests will show significance by chance. If 20 tests are conducted, one false result is guaranteed. But dishonest researchers can perform 100 or 1,000 tests.
Controversial Figures: Saviors or Scammers?
Among those actively exposing data dredging is John Ioannidis, a doctor and statistician. In 2005, he published a famous article titled "Why Most Published Research Findings Are False" (Why Most Published Research Findings Are False). He showed that in medicine, more than 80% of studies claiming significant findings could not be replicated.
On the other hand, there are also figures accused of practicing data dredging. For example, Brian Wansink, a professor at Cornell University, published many studies on food psychology. In 2018, an investigation found that he often reported inconsistent results and conducted inappropriate analyses. As a result, he had to retract over 15 articles.
A Shameful Legacy: The Replication Crisis
Now, the scientific world is grappling with a replication crisis. In 2015, the Reproducibility Project in psychology found that only 39% of 100 major studies could be successfully reproduced. In oncology, the replication rate is even lower, around 11%.
This crisis is largely due to data dredging. When researchers use this practice, they produce findings that appear impressive but are actually false. When other researchers try to replicate the studies, they fail. This wastes a lot of resources and misleads public policy.
The Path to Recovery: Transparency and Pre-Registration
To address this issue, various measures have been introduced. First, pre-registration of studies: researchers must register their hypotheses and analysis methods before collecting data. This prevents them from changing their analysis after seeing the data. Second, the use of blind analysis: analysts do not know which group is the treatment and which is the control until the analysis is complete. Third, encouraging the reporting of all tests, not just the significant ones.
Now, many scientific journals have adopted this policy. For example, journals like Nature and PLOS ONE require researchers to clearly state whether their analyses are exploratory or confirmatory. This helps distinguish between real discoveries and p-hacking results.
Conclusion: Lessons for Future Generations
Data dredging is not just a wrong scientific practice; it is an ethical violation that has damaged public trust in science. History teaches us that truth cannot be forged according to personal desires. Every time we see a study claiming a remarkable discovery, we should ask: Is this the result of careful analysis or just a clever data dredging?
As readers and users of science, we must be wise. Don't easily believe sensational headlines. Demand transparency, replication, and honesty. Only in this way can we ensure that the knowledge we build is solid, not fragile like a sandcastle.
---
This article is written based on legitimate historical and statistical sources. Main references: Ioannidis (2005), Simmons et al. (2011), and the Reproducibility Project report.
---
References: Data dredging — Wikipedia
Blind Data Mining: The Dark History of P-Hacking That Misled Science. Behind the scenes of research, there is an illegal practice known as data dredging — sifting through data without a clear hypothesis until false correlations are found. The history of this practice is not only shameful but has also produced thousands of false studies. This article explores its origins, controversial figures, and the negative impact it has had on the scientific world.. The Golden Age of Statistics: The Birth of Data Dredging
In the early 20th century, the statistical revolution changed the way humans understood the world. Sir Ronald Fisher, the father of modern statistics, introduced the concept of significance tests and p-values in 1925. His idea was simple: if the probability of an outcome occurring by chance is very low usually below 0.05 , then the outcome is considered statistically significant. However, Fisher himself warned that these tests were only valid if the hypothesis was set before collecting the data. Unfortunately, this warning is often ignored.
In the 1950s and 1960s, computers began to be used in data analysis. With the ability to test thousands of relationships in an instant, researchers fell into "p-hacking" — repeating tests until they found a significant p-value. Statistician John Tukey in 1962 called this phenomenon "data snooping" and warned that it drastically increases the risk of false positives.
The Biggest Scandal: From Psychology to Medicine
One of the most famous cases occurred in social psychology. In 2011, researcher Daryl Bem published a study claiming that humans can predict the future — a form of psychic ability. The study used nine experiments with various statistical tests. Upon review, it was found that only a few tests showed significant effects, while the rest failed. This is a classic example of data dredging: reporting only results that support the hypothesis while hiding others.
In the medical field, a greater tragedy occurred. In 2004, a study on antidepressants for teenagers revealed that pharmaceutical companies only published studies showing positive effects, while studies showing no effect or harmful effects were hidden. As a result, thousands of teenagers were given ineffective and dangerous drugs. This scandal led to reforms in clinical trial registration.
How Data Dredging Works: The Mechanism of Fraud
The process of data dredging usually involves the following steps:
1. Collecting data without a specific hypothesis : Researchers take large existing datasets, such as census data or hospital records.
2. Testing many correlations : They test thousands of variable pairs — for example, does the number of umbrellas sold correlate with the crime rate? There may be a correlation, but it doesn't imply causation .
3. Selecting significant ones : Only correlations that produce p-values below 0.05 are reported, while others are ignored.
4. Adjusting the analysis : Sometimes they change how the data is collected or analyzed — for example, removing outliers, changing variable definitions, or combining groups — so the results become significant.
As a result, one out of 20 tests will show significance by chance. If 20 tests are conducted, one false result is guaranteed. But dishonest researchers can perform 100 or 1,000 tests.
Controversial Figures: Saviors or Scammers?
Among those actively exposing data dredging is John Ioannidis, a doctor and statistician. In 2005, he published a famous article titled "Why Most Published Research Findings Are False" Why Most Published Research Findings Are False . He showed that in medicine, more than 80% of studies claiming significant findings could not be replicated.
On the other hand, there are also figures accused of practicing data dredging. For example, Brian Wansink, a professor at Cornell University, published many studies on food psychology. In 2018, an investigation found that he often reported inconsistent results and conducted inappropriate analyses. As a result, he had to retract over 15 articles.
A Shameful Legacy: The Replication Crisis
Now, the scientific world is grappling with a replication crisis. In 2015, the Reproducibility Project in psychology found that only 39% of 100 major studies could be successfully reproduced. In oncology, the replication rate is even lower, around 11%.
This crisis is largely due to data dredging. When researchers use this practice, they produce findings that appear impressive but are actually false. When other researchers try to replicate the studies, they fail. This wastes a lot of resources and misleads public policy.
The Path to Recovery: Transparency and Pre-Registration
To address this issue, various measures have been introduced. First, pre-registration of studies: researchers must register their hypotheses and analysis methods before collecting data. This prevents them from changing their analysis after seeing the data. Second, the use of blind analysis: analysts do not know which group is the treatment and which is the control until the analysis is complete. Third, encouraging the reporting of all tests, not just the significant ones.
Now, many scientific journals have adopted this policy. For example, journals like Nature and PLOS ONE require researchers to clearly state whether their analyses are exploratory or confirmatory. This helps distinguish between real discoveries and p-hacking results.
Conclusion: Lessons for Future Generations
Data dredging is not just a wrong scientific practice; it is an ethical violation that has damaged public trust in science. History teaches us that truth cannot be forged according to personal desires. Every time we see a study claiming a remarkable discovery, we should ask: Is this the result of careful analysis or just a clever data dredging?
As readers and users of science, we must be wise. Don't easily believe sensational headlines. Demand transparency, replication, and honesty. Only in this way can we ensure that the knowledge we build is solid, not fragile like a sandcastle.
---
This article is written based on legitimate historical and statistical sources. Main references: Ioannidis 2005 , Simmons et al. 2011 , and the Reproducibility Project report.
---
References: Data dredging — Wikipedia https://en.wikipedia.org/wiki/Data dredging
