‘Be constructive but unvaryingly succinct’
Before writing this piece, I consulted widely with colleagues about why they undertake peer review. The answers were pretty much as I expected. Senior colleagues feel that it is an essential requirement of being a scientist to engage in the process – “a moral obligation”, as one put it. Many spend a significant amount of time at the weekend or in the evenings reviewing, since it is “a duty above and beyond the day job”.
They are certainly aware of the flaws of peer review – unconscious bias, conflicts of interest, personal loathing, lack of specialist knowledge, ignorance, not being fully up to date on the topic, and so on. Despite all this, my colleagues still see peer review as the “gold standard of science”. One – and he is by no means the first to do so – paraphrased Churchill’s thoughts on democracy: “Peer review is the worst form of assessment except for all those other forms that have been tried from time to time.”
Interestingly, my junior colleagues are more pragmatic. I quote directly from one succinct colleague: “I review because it is the only way to prevent crap from being published that I would then have to wade through before I [can] get my own work published.” Early career scientists are also much more selective in what they will review, choosing manuscripts from the top journals and only in fields of direct relevance, with the primary aim of “finding out what is going on”. Another difference between senior and junior colleagues is that the latter are much more likely to take rejection personally, and to get upset. Whether this is because age results in the accumulation of wisdom, the denudation of the emotions or the acquisition of alternative forms of gratification remains unclear to me.
Henry Oldenburg, the founding editor of The Philosophical Transactions of the Royal Society of London, is credited with inventing pre-publication peer review in 1665. By the 1830s, all Royal Society publications were subject to some form of external peer review. However, it was not until the mid-20th century that refereeing became the norm for most journals; indeed, Nature instituted formal peer review only in 1967; previously, editors had relied on their own expertise, supplemented with internal discussion.
But editors retain immense power. They decide whether a manuscript should be sent for peer review, to whom the manuscript will be sent, and whether the reviewers’ advice is acted upon or ignored. A senior colleague makes the point that “the key thing about getting a paper published these days is to make the abstract sufficiently idiot proof to get the [section] editor to send it out for peer review”. And while my colleagues express frustration with referees, there is a grudging trust that, on balance, they do a good job. By contrast, there is little faith in editors’ ability to act as qualified “gatekeepers” of published science.
In my own case, I am sometimes frustrated by the manuscripts I am asked to review. Lack of novelty, unclear language, unspecified methodological details, weak experimental design, muddled statistics and obscure data presentation can all elicit this response. If the manuscript’s conclusions are faith-based and not data-based, I also get very disappointed, resenting that my evening’s work has not taught me something new. And I get especially irritated when important data are deliberately buried in the supplementary data section, or when there are obvious technical mistakes. I once reviewed a mouse paper that reported levels of a hormone that a mouse is genetically incapable of making – that experience required a very large, calming whisky.
My responses in such cases are, hopefully, always constructive but are unvaryingly succinct – perhaps even blunt. And I don’t really discriminate between “top” and “mainstream” journals. Good science is good science.
But on the rare occasions when the work is truly novel and brilliantly designed, I am transported to a new plane of happiness, and I gush shamefully in my written response, even if a few more experiments or tweaks are required. I might even admit coyly, in conversation with the author, that I was a referee!
Shortly after my election to one of the UK’s national academies, I joined its election committee. I witnessed for the first time the multiple layers of peer review that eventually lead to the appointment of a new fellow. In view of the intense competition, scrutiny and detailed discussion, I remarked to another committee member that I wondered how the bloody hell I had ever been elected. The individual turned his head, and with a complete absence of humour, answered: “Well, in my case, I wondered why it had taken so long”.
On balance, peer review is an invaluable force for good, but it clearly does not select for humility!