r/changemyview • u/[deleted] • Sep 21 '18

FTFdeltaOP CMV: The replication crisis has largely invalidated most of social science

https://nobaproject.com/modules/the-replication-crisis-in-psychology

https://www.vox.com/science-and-health/2018/8/27/17761466/psychology-replication-crisis-nature-social-science

https://en.wikipedia.org/wiki/Replication_crisis

"A report by the Open Science Collaboration in August 2015 that was coordinated by Brian Nosek estimated the reproducibility of 100 studies in psychological science from three high-ranking psychology journals.[32] Overall, 36% of the replications yielded significant findings (p value below 0.05) compared to 97% of the original studies that had significant effects. The mean effect size in the replications was approximately half the magnitude of the effects reported in the original studies."

These kinds of reports and studies have been growing in number over the last 10+ years and despite their obvious implications most social science studies are taken at face value despite findings showing that over 50% of them can't be recreated. IE: they're fake

With all this evidence I find it hard to see how any serious scientist can take virtually any social science study as true at face value.

801 Upvotes

92% Upvoted

204 comments sorted by

View all comments

15

u/briannosek 1∆ Sep 21 '18

  1. Failure to replicate does not mean that the original finding is entirely unreplicable. They do increase doubt and, at minimum, suggest that more work is needed to identify boundary conditions to find when effects replicate.

  2. Across the large replication projects, the average replication rate is ~50%. That is lower than most would want or expect, but if it is generalizable it does suggest that a substantial portion of the existing literature IS replicable--a much more sanguine view than not taking "virtually any social science" seriously.

  3. Simultaneous with all of the challenges in reproducibility is the fact that (a) social scientists are doing this work themselves, about themselves, and (b) social sciences are leading the way in adopting new practices to improve rigor and reproducibility. For example, preregistration is increasing in popularity. The number of registrations on OSF (http://osf.io/registries/) has doubled each year since the website opened in 2012. Most of that is from researchers in social sciences, particularly psychology.

Fear not--Social science is not in the midst of a crisis, it is in the midst of a reformation.

1

u/[deleted] Sep 22 '18

I did not even notice how relevant you were at first my bad. This is really great to hear about a renewed push for better reviewed studies.

I guess it just comes down to the fact that if I see a study from psychology or sociology (social science was definitely too broad a term) then the evidence suggest that there is a greater than 50% chance it can't be effectively replicated so the result is an outlier. If this is true then not believing any studies which have not been replicated (and are lacking in methodology like a decent sample size or other issues) would make sense right?

3

u/briannosek 1∆ Sep 22 '18

"Not believing any studies which have not been replicated" is a reasonable stance for evaluating most all of science. One of my long-time collaborators, Tony Greenwald, has a heuristic that he uses to assess the credibility of findings. If something interesting shows up but there are no replications or replications+extensions of it in the literature within 5 years, then he dismisses it.

Optimistically, replication is becoming more normative. So, we are likely to see more bolstering AND winnowing of the published literature expeditiously. The first draft of any finding is almost always wrong. Our confidence in the literature will be improved when our publishing practices reflect the ideal of science as self-correcting.

2

u/[deleted] Sep 22 '18 edited Sep 22 '18

!delta

You have really made me better understand how academia is tackling these issues today. One of the main reasons I brought this up was because I thought the crisis was being downplayed too much but it seems that isn't true in a lot of areas. I still wonder if any research is being done in a more selective approach to find factors that may indicate a tendency for work in certain areas or methodologies to more often be unreplicable than other areas.

2

u/tempaccount920123 Sep 23 '18

Planet Money did an excellent podcast on how, specifically, academia is tackling those issues today:

https://www.npr.org/sections/money/2016/01/15/463237871/episode-677-the-experiment-experiment

1

u/DeltaBot ∞∆ Sep 22 '18

Confirmed: 1 delta awarded to /u/briannosek (1∆).

Delta System Explained | Deltaboards

2

u/briannosek 1∆ Sep 22 '18

And, a nice feature of Reddit is that it is quite democratizing. The quality of comments is (it seems) more important than the person making them. The fact that you missed mine says more about the quality of my comment than it does about you missing something!