Modern science works in the following manner. You do the research. You write a paper. You publish the paper. For historical reasons, “publishing the paper” typically means “submit it to a committee of your peers and get their seal of approval”.
We can rightfully be concerned about the unwanted biases that this committee of peers might have. For example, maybe these peers have unconscious biases against women?
It is not unreasonable. We have evidence that peer review is strongly biased in favour of authors from prestigious institutions. For example, Peter and Ceci took already accepted articles from authors at prestigious institutions, and they resubmitted them as authors from lesser institutions: they encountered an 89% rejection rate.
reviewers were more likely to recommend acceptance when the prestigious authors’ names and institutions were visible (single-blind review) than when they were redacted (double-blind review) (87% vs 68%) and also gave higher ratings for the methods and other categories.
That, by itself, is not as worrying as it seems. It is a fact that Stanford researchers are better, on average, than researchers from that school you never heard from. Everything else being equal, it makes sense to put more trust in work from Stanford.
But gender is something else. We have no rational reason to trust the work done by men more than the work done by women.
So is peer review sexist? Webb et al. in Does double-blind review benefit female authors? write:
We found a significant interaction between gender and time (P < 0.0001), reflecting the higher female authorship post-2001 than pre-2001, but there was no significant interaction between gender and review type (…) Budden and colleagues call for the ecological and evolutionary community to revisit the issue of peer review in light of their study of BE, and their claim that double-blind review benefits female authors (…) This is despite the fact that the only evidence they supply is an increase in female first authorship in a single journal (an increase also seen in other journals that did not switch to double-blind review) rather than anything more compelling, such as a demonstration that the ratio of accepted to submitted manuscripts had increased for females relative to males after the introduction of double-blind review.
The preponderance of evidence, including the best and largest studies, indicates no discrimination in reviewing women’s manuscripts
(…) recent meta-analysis suggests that claims of gender bias in peer review “are no longer valid”. For example, if there is gender bias in review, we would expect double-blind conditions to increase acceptance rates for female authors. However, this is not the case. Nor are manuscripts by female authors disproportionately rejected at single-blind review journals. Even when the quality of submissions is controlled for, manuscripts authored by women do not appear to be rejected at a higher rate than those authored by men. Meta-analyses and large-scale studies of grant outcomes found no gender differences after adjusting for factors such as discipline, country, institution, experience, and past research output.
the influence of author gender on bidding or reviewing behavior is not statistically significant.
We found no evidence that White male principal investigators received evaluations that were any better than those of principal investigators from the other social categories, and this conclusion was robust to a wide array of model specifications. Supplemental analyses suggest that any bias that is present is likely below the threshold of pragmatic importance.
Li and Agha found in Big names or big ideas that grant reviews are fair:
We find that better peer-review scores are consistently associated with better research outcomes and that this relationship persists even when we include detailed controls for an investigator’s publication history, grant history, institutional affiliations, career stage, and degree types
Our results indicate that female principal investigators (PIs) receive a bonus of 10% on scores, in relation to their male colleagues.
So it seems that we have good news. Peer review is not, statistically speaking, sexist.