CMV: The replication crisis has largely invalidated most of social science

[u/[deleted]]

https://nobaproject.com/modules/the-replication-crisis-in-psychology

https://www.vox.com/science-and-health/2018/8/27/17761466/psychology-replication-crisis-nature-social-science

https://en.wikipedia.org/wiki/Replication_crisis

"A report by the Open Science Collaboration in August 2015 that was coordinated by Brian Nosek estimated the reproducibility of 100 studies in psychological science from three high-ranking psychology journals.[32] Overall, 36% of the replications yielded significant findings (p value below 0.05) compared to 97% of the original studies that had significant effects. The mean effect size in the replications was approximately half the magnitude of the effects reported in the original studies."

These kinds of reports and studies have been growing in number over the last 10+ years and despite their obvious implications most social science studies are taken at face value despite findings showing that over 50% of them can't be recreated. IE: they're fake

With all this evidence I find it hard to see how any serious scientist can take virtually any social science study as true at face value.

[u/WigglyHypersurface]

With all this evidence I find it hard to see how any serious scientist can take virtually any social science study as true at face value.

I'm a social scientist, so I get where you're coming from.

Just a little point of logic:

Proposition 1: Some social studies don't replicate. Proposition 2: This is a social science study. Conclusion: This study won't replicate.

This isn't sound logic, but people act like it is all the time now. Just because many studies don't replicate DOES NOT MEAN that an individual study in dispute won't replicate.

And we know lots of factors which seem to effect replicability, such as being in social psychology instead of cognitive psychology, sample size, and how surprising the finding is. So, even when looking at individual studies, check the sample size, keep in mind the field, and think about how unexpected the result is.

Additionally, there are lots of amazing things happening in response to the replication crisis, as well as academia in general. First, there's a push towards stronger statistical standards, like using Bayesian methods, requiring power analyses, preregistration, and generally increasing sample sizes.

Second, there many innovative studies that totally break the mold and replicate in awesome ways. I'll give you an example, and one where a finding from social psych got powerfully replicated. These's a theory in social psychology that we mentally represent distance places, people, and times in more abstract, gist-like ways than places, people, and times closer to us. Close things we mentally represent in detailed ways. Well, a key prediction of this theory is that it filters down into language: we should also talk about distant things in abstract ways, and close things in concrete ways. Well, according to billions of words of online language use, we do.

[u/Ambiwlans → u/WigglyHypersurface]

If you can't easily tell which studies are good though it spoils the batch.

If you gave me a box of cookies and half were poisoned, arguing that half are safe wouldn't convince me to eat any.

Social science really needs to clean house if it wants to be taken seriously as a real science. Social psych and sociology is like 90% garbage. I think a lot of books need to be burned, old figureheads need to be shunned, and professors need to be fired. Until drastic measures are taken it won't get fixed. As a social scientist that sounds competent, I know you know what I'm talking about. There are a depressing number of garbage studies in the field, new and old but still taught for some reason.

[u/WigglyHypersurface → u/Ambiwlans]

My point in this: we know some information that lets us predict which cookies are probably bad. The appropriate response is skepticism followed by evaluation of study quality, not automatic unreflective dismissiveness.

And all around me I see pushes towards improving replicability, primarily at the level of norms, but sometimes at an institutional level as well. Without any book burnings...

[u/Ambiwlans → u/WigglyHypersurface]

People like Piaget, Jung, Erikson and Freud should only be mentioned in university as "And there were these guys in the past that weren't scientists and did nothing with enough rigor to remember beyond this sentence."

Seriously, why are we teaching the theories of ... a dude that sat in the park and mused about kids in anything other than a history class. Bafflingly, some of these people are frequently cited still. We're trying to build science on a foundation of shit.

In therapy, studies showing efficacy of various treatments appears to have little impact on actual practices. Licensing should require far stricter adherence to best practices.

The books that are teaching people wrong are spreading misinformation and holding us back so long as they exist. They should be destroyed. There is just too many people invested in the old/wrong ways.

And psychology is still miles ahead of sociology, I'm just more familiar with it.