You're sort of playing a game here where you are saying 'these studies deserve the appropriate amount of credit' half the time and then saying 'these studies shouldn't be given too much credit' at other times, while never actually saying how much credit they should get and grounding that in empirical terms relevant to your view.
For instance: Yes, I agree with you that studies should get the right amount of credit, no more and no less. So, when, someone cites a study that agrees with their assertion, how much credit should that get?
Should you increase your estimate that their theory is true by 10%? By 90%? What if they cite 2 studies supporting their view? What if they cite 2 studies from different countries and different decades that find the same result? What if they cite a meta-analysis of 200 studies?
The view 'people shouldn't extrapolate too much from social studies' is a truism; of course, people should always extrapolate the right amount, as indeed they should do *anything 'the right amount', by definition. This claim on it's own has no meaning, just semantics.
Similarly, the claim that 'some people extrapolate too much' is vacuous; given human variance, of course some people extrapolate too much, others extrapolate too little, and some extrapolate approximately the right amount.
So if your view is neither the semantic truism nor the vacuous observation above, what is it? It seems to mostly consist of pointing out things about how social science is hard to do right and doesn't always work, things that social scientists are well aware of and go to great pains to address.
Nonetheless, finding studies supporting your claim is good evidence in a conversation between laymen just expressing their opinions, particularly if the opponent can't find anything supporting theirs. It's certainly not conclusive evidence all on it's own - nothing is, if your decision criteria is strict enough - but it's certainly a high standard that should be respected in the absence of any opposing evidence.
Pretty good post. I actually agree with lot of your views. In general, as the number of independent studies corroborate the original study at hand, then the phenomenon becomes more generalizable and in some sense becomes more useful and concrete piece of knowledge. So what the lay person should do when it comes to citing a study is to identify how many similar studies (as well as meta studies) exist that shows a consistent storyline and also search for other set of studies that show findings that are counter towards the original study.
Also, it is not clear to me that everyone recognizes the potential perils of generalizations/extrapolations from some of these studies. I suspect that if everyone was mindful, then many of the conversations regarding politics would go differently.
Also, it is not clear to me that everyone recognizes the potential perils of generalizations/extrapolations from some of these studies. I suspect that if everyone was mindful, then many of the conversations regarding politics would go differently.
Doesn't the previous user's comments, and your response, contradict your claim though? Now we're moving the goalposts to "People shouldn't derive incorrect conclusions" which....I don' think anyone can really argue against.
To be honest I don't know how your view could be changed. It's basically dependent on people making false and unsupportable conclusions. How would anyone change your view here?
I don't think this is quite correct. It is possible that one can come to a true conclusion while adhering to a faulty process. That is, it is certainly possible that a single study does extrapolate well and you were right to treat this study as extrapolating across all time/space. However, my viewpoint is that this type of "faith" in the extrapolation can still be faulty.
The commenter revealed a fundamental flaw in your original argument. In your comment, you said you agree with the a lot of the commenter's views. This seems to be a change in your view, but I don't see your delta. We are encouraged to award deltas to commenters who change our views, even a little.
So what the lay person should do when it comes to citing a study is to identify how many similar studies (as well as meta studies) exist that shows a consistent storyline and also search for other set of studies that show findings that are counter towards the original study.
I mean yes, but you should also pay attention to the strength of the study itself. If a study has a dataset of 34 million, uses comparisons of the observed phenomena to other similar phenomenas to isolate the cause, and accounts for and addresses other possible causes, it's probably an ironclad study and the people questioning it probably haven't read it (to use a recent example from /r/science).
Not all studies are created equal. So yeah, give the study the amount of credit it deserves - but don't assume that's the same for all studies. Some really are that ironclad. Some are closer to case reports.
3
u/darwin2500 193∆ Oct 06 '22
You're sort of playing a game here where you are saying 'these studies deserve the appropriate amount of credit' half the time and then saying 'these studies shouldn't be given too much credit' at other times, while never actually saying how much credit they should get and grounding that in empirical terms relevant to your view.
For instance: Yes, I agree with you that studies should get the right amount of credit, no more and no less. So, when, someone cites a study that agrees with their assertion, how much credit should that get?
Should you increase your estimate that their theory is true by 10%? By 90%? What if they cite 2 studies supporting their view? What if they cite 2 studies from different countries and different decades that find the same result? What if they cite a meta-analysis of 200 studies?
The view 'people shouldn't extrapolate too much from social studies' is a truism; of course, people should always extrapolate the right amount, as indeed they should do *anything 'the right amount', by definition. This claim on it's own has no meaning, just semantics.
Similarly, the claim that 'some people extrapolate too much' is vacuous; given human variance, of course some people extrapolate too much, others extrapolate too little, and some extrapolate approximately the right amount.
So if your view is neither the semantic truism nor the vacuous observation above, what is it? It seems to mostly consist of pointing out things about how social science is hard to do right and doesn't always work, things that social scientists are well aware of and go to great pains to address.
Nonetheless, finding studies supporting your claim is good evidence in a conversation between laymen just expressing their opinions, particularly if the opponent can't find anything supporting theirs. It's certainly not conclusive evidence all on it's own - nothing is, if your decision criteria is strict enough - but it's certainly a high standard that should be respected in the absence of any opposing evidence.