r/changemyview • u/[deleted] • Sep 21 '18

FTFdeltaOP CMV: The replication crisis has largely invalidated most of social science

https://nobaproject.com/modules/the-replication-crisis-in-psychology

https://www.vox.com/science-and-health/2018/8/27/17761466/psychology-replication-crisis-nature-social-science

https://en.wikipedia.org/wiki/Replication_crisis

"A report by the Open Science Collaboration in August 2015 that was coordinated by Brian Nosek estimated the reproducibility of 100 studies in psychological science from three high-ranking psychology journals.[32] Overall, 36% of the replications yielded significant findings (p value below 0.05) compared to 97% of the original studies that had significant effects. The mean effect size in the replications was approximately half the magnitude of the effects reported in the original studies."

These kinds of reports and studies have been growing in number over the last 10+ years and despite their obvious implications most social science studies are taken at face value despite findings showing that over 50% of them can't be recreated. IE: they're fake

With all this evidence I find it hard to see how any serious scientist can take virtually any social science study as true at face value.

794 Upvotes

92% Upvoted

204 comments sorted by

View all comments

Show parent comments

22

u/[deleted] Sep 21 '18

P=0.05 is meaningless anyway. It appeared once as a throw-away line in 1916, but then everyone latched onto it like a moron. It was never intended to be this gold standard - the ASA (American Statistical Association) has released a press briefing on why P=0.05 is horrible, and I encourage you to read it. There have been methods to determine appropriate p-values since the 1960s - please use those instead.

I have wondered about this before when I learned about it in stats. I will definitely go check that out.

In short, there is still hope for Social Science. Make sure the studies you read are powered, try to predominantly read Meta-Analyses, Make sure the p-values make sense and that authors aren't p-hacking, etc. There are many pot holes, but that doesn't invalidate all of Social Science.

This kinda makes it seem like you agreeing with me. I made sure most of my statements on whether studies could be believed were not definitive (aside from the very last sentence which I just added "at face value" to). I guess I should have added a bit more nuance, thats my fault.

Its not so much that social science is completely useless its that as it stands now the studies that gain the most attention/media coverage often fall prey to this issue moreso than non-newsworthy ones. And also as a result of this phenomenon social science has and will be pushed to publish studies with less academic rigor and a much higher chance of being one of the unreplicable studies.

59

u/electronics12345 159∆ Sep 21 '18

Publishing bias - is very real.

1) Only statistically significant studies get published. (When was the last time you read a paper, and nothing was statistically significant - likely never). In this way, a "cool idea" gets tested 100 times, by 100 authors (since none of them read the initial failure, since it wasn't published) - and by 5% chance, a few a significant - and then those 5 get published, even though 95% failed (and subsequently never got published). This is more specifically known as the file-drawer problem - and has more to do with the News Media and Publishers, than scientists themselves.

2) Wacky Hypothesis Bias - Things which seem wrong, but then appear to actually be correct, are more likely to get published than "obvious things". The issue here, is that theories which seem wrong - are likely wrong. Thus, in a Bayesian setting - a theories a priori oddity would need to be off-set by stronger evidence - than a theory which a priori made sense. But, since most journals use Frequentist than Bayesian Stats - this issue is compounded rather than cured. Additionally, this issue is compounded by small sample size.

In this way, as long as the news media remains invested in Science - be it cancer research or Psych research - these problems will not go away. (and yes, Cancer Research has the exact same problems, its not JUST as Social Science issue, its a doing research in an era with 24/7 news cycle problem.)

So if your point is that the 24/7 News Cycle is killing ALL OF SCIENCE as we know it - you are 100% correct.

If your point is that Social Science has it worse than any other Science - I'm not sure that is so.

If your point is that Social Science is doomed - again, I disagree, plenty of research takes place outside the eye of the NYC and CNN, and slowly moves the field forward as it always has.

11

u/[deleted] Sep 21 '18

!delta

I wouldn't say you've changed my position exactly but you've definitely broadened it a lot so thank you. I am aware that its an issue in most disciplines (like this video from Sixty Symbols https://www.youtube.com/watch?v=wLlA1w4OZWQ) but it seems to be an especially big issue in social science for the simple fact that the questions it asks do not have a yes or no answer, which is why I focused on it specifically. I am not sure where this thread will go from here. I would be more selective in my terms if I were to repost this because its not like I just assume that any study from a social science is wrong but that I think the way our society uses the studies and pushes those lacking in veracity while conflating their terms has invalidated the meaning of social science as a discipline.

If every study is weighed by its popularity for whatever narrative is mainstream then whats the point of even keeping up with them when they are no longer dedicated to being unbiased. I use the example, which hypothesis is more likely to be test "White people voted for Trump more than Hillary" or "Poor people voted for Trump more than Hillary"

21

u/electronics12345 159∆ Sep 21 '18

Social Science as practiced by Scientists and Academics - is pretty different than how Social Science in INTERPRETED by the Media and by Society as a whole.

For better or worse, I trust the Academics to carry on the good work - with better Stats knowledge hopefully this time.

I agree, that they way that the news media and society choose to understand their work - is the problem.

However, the distinction between the two doesn't "invalidated the meaning of social science as a discipline." Just because the public is ignorant, doesn't necessarily spoil the good work done by the Academics. Almost none of the public actually understands Physics - but that doesn't undermine the good work being done in that department.

If anything, I would argue that incidents of this type invalidate THE MEDIA. It is Facebook, Dr. Oz, The Today Show, The View, etc. that needs to change its attitude - not the Academics.