There was a piece published recently that claimed to be an academic grievance study, 'exposing' corruption in sociological fields of publication with gathered some attention in The Economist and The Atlantic for example. The full essay can be found here. I'm not going to discuss the intent of the piece here: these sort of things always involve a certain abuse of faith from peer reviewers, and the claim from one of the masturbation/rape reviewers is relevant here in that an inexperienced reviewer was attempting to do the right thing in providing constructive criticism, although I do accept that structural factors could pressure genuine reviewers in a negative direction as the authors of the hoax would claim.
The major problem I wish to discuss is the appalling manner in which this hoax was conducted.
1. Misreporting
Lindsay et al. make repeated reference to the fact that they had 7/20 papers accepted for publication. However, when we examine the actual project notes that are cited in their original article, it emerges that they in fact made 48 new submission, and have mangled their denominator by excluding submissions where the paper was rejected before resubmitting them. There is therefore a severe upward bias in their measure of success- the success rate plunges from 35% to 14%.
2. Lack of a control group
The lack of a control group within this study limits our ability to actually draw much inference about the success rate of the submissions. The authors claim that they are using the overall submission success rate as a 'meta-control' but this is clearly inadequate- there is an obvious and substantive difference between papers that are designed to be manipulated to maximise the probability of success without the baggage of being based on real data, and the median work of real scholarship. It's also a leap to the idea that Lindsay et al. were matched in terms of ability, writing quality to the median submitter of these journals; if Lindsay et al. had submitted some real work we could have differenced out this confounder. But they didn't, so we can't, and a potential bias remains.
3. Methodological inconsistency
Lindasy et al. changed their method throughout the course of the study. This is certainly a major red flag in a lot of experimental social science for the reason that Linday et al. state in their article: they changed their method because their initial attempts were not getting the results that they desired. If we look at their initial claim that it would be relatively easy to get a paper published regardless if it simply sounded right, this seems to have been totally falsified. None of their initial papers went for full peer review. It was only after they changed their approach that they met with any success whatsoever, reporting
4. Subjective interpretationThis shift in success rate followed a commitment to understand the field in greater depth that initiated in late November 2017 and progressed through April 2018, by which time we felt we had become sufficiently competent.
Lindsay et al. claim that much of their data should have raised alarm bells because it was highly implausible. They do not however, present statistics such that this can be verified and we have only their word that these figures should be obviously incorrect. Indeed, the specific paper which they highly as having implausible statistics ("Dog Park") was the paper for which other academics began requesting field work so concerned were they about the paper. The paper has been retracted and as such I cannot see any figures claimed, but if this were a true scholarly work, there would need to be some more empirical backing of the claim that such figures were implausible, rather than the simple assurance of a party with vested interest.
In light of all of these limitations, I'm not sure that we should take the results of this particular grievance study conducted too seriously. This is not to claim that there are not issues in scientific publishing, and in social science publishing, of course there are areas that need to be improved. However, this attempt was not rigorous, and distracts attention from actual scholarly analysis with deeply concerning findings and could falsely lead us to the conclusion that problems are exaggerated or limited to very niche fields, whereas I suspect they are probably more common although less headline grabbing.
Addendum: As of 12/01/2019, one of the authors has failed an internal review for this study, for failing to seek ethical approval for the deception of peer reviewers.
