REVIEWING PEER REVIEW

Prepublication scrutiny of papers isn’t a panacea, but is best safeguard we have

Science can be defined as the process of gaining knowledge about the workings of the world through experiments and observation. Indeed, the word itself derives from “scientia,” the Latin term for knowledge.

Sometimes simple observation is enough to impart knowledge. No extra experimentation is needed to determine that poison ivy can cause blisters, that chickens come from eggs or that the sun can burn the skin. But when it comes to launching satellites into orbit, developing new drugs, answering questions about diet or determining the toxicity of chemicals, knowledge can only come from well-designed experiments. Reproducing experiments and extending them is the essence of science, which means that the sharing of results by researchers is integral to its pursuit. Exactly how to do this can present a thorny problem.

Like in every other area of life, human foibles can enter the picture. Shoddy research, overzealous conclusions based upon sketchy data, catering to vested interests and outright fraud can all conspire to place hurdles in the race toward accumulating sound knowledge. The traditional way of minimizing these obstacles is “peer-review” of studies before they are published.

Peer-review involves submitting a research paper to the editor of a journal who then sends it out to two or three experts in the field who are asked to act as “referees.” Their task, for which they are not paid, is to evaluate whether the paper merits publication. The authors are unaware of the identity of the referees but the referees know who the authors are. Often there is back-and-forth correspondence through the editor with the referees asking for clarification, changes, or occasionally more work to be done. In the end, the paper is accepted or rejected. Some prestigious journals, like the New England Journal of Medicine, have high rejection rates, while at the other end of the spectrum there are journals that will accept almost any work.

The most highly regarded journals require a subscription, but there is a trend toward “open access” journals that can be accessed by anyone. These vary greatly in quality, with some being termed “predatory” journals since they will publish anything for a fee. A number of academics have highlighted this problem by submitting totally nonsensical papers to such journals. A classic example is a ridiculous paper submitted to the supposedly peerreviewed journal Cogent Social Sciences with jumbled, meaningless jargon arguing that “penises shouldn’t be thought of as male genital organs but as damaging social constructions.” Absurdly titled, The Conceptual Penis as a Social Construct, the paper was published.

There are other issues. Peerreview is a notoriously slow process often involving months of back-and-forth correspondence, meaning that the information may even be out of date by the time it is published. Also, since reviewers by necessity work in the same field as authors, there may be a temptation to delay papers or even to appropriate ideas or data in cases where the criterion for career advancement is “publish or perish.” There is sometimes also the question of peer objectivity. For example, when it comes to journals that deal with the dubious discipline of homoeopathy, the reviewers are likely to be homeopaths who are unlikely to be objective critics.

Then there is the problem of fraud. Since referees cannot repeat the work they are asked to evaluate, they have to assume that the information submitted is correct. Unfortunately, there are crooked researchers who for various reasons will attempt to get fraudulent data published.

A further concern is that a great deal of published research cannot be duplicated. Why this should be the case isn’t clear. Perhaps it is inadequate description of methodology, or selective reporting of results.

Another issue is that negative results, or studies that just attempt to replicate others, are often shunned by referees.

Einstein was perhaps the most famous critic of peer-review. In 1936 after receiving the comments of a reviewer he arrogantly wrote to the editor: “Dear Sir, we had sent you our manuscript for publication and had not authorized you to show it to specialists before it is printed. I see no reason to address the — in any case erroneous — comments of your anonymous expert. On the basis of this incident I prefer to publish the paper elsewhere.”

Various suggestions have been advanced to replace traditional peer review. Some critics say that the names of the referees should be disclosed to the authors, others say that the reviews should be double blind, meaning that neither the referees nor the authors should know each others’ identities. There are also suggestions that the referees’ reports and all correspondence should be published. Another idea is to have authors submit their hypotheses, research methods and protocols to a journal before any results are submitted. The journal will then decide whether the work is worth publishing, irrespective of the results. Once the final paper is submitted, it will be published whether the results are negative or not. A truly radical idea is peer review by endorsement. Researchers would be asked to submit a paper

Peer-review involves submitting a research paper to the editor of a journal who then sends it out to two or three experts in the field who are asked to act as ‘referees.’

along with reports from two peers who cannot be at the same institution and have not worked with the authors.

With about five peer-reviewed papers being published every minute, some excellent, some dismal and the vast majority mediocre, it is a monumental task to separate the wheat from the chaff. It is possible (I know because I have done it) to put together a very convincing presentation on a controversial topic from either side by selectively referring to the literature, be it about bisphenol A, GMOs, meat consumption or chemicals in cosmetics.

Even the highest quality research may not resolve controversies, but there is a need for increased awareness of the proliferation of “predatory” or “pay for play” journals that often publish substandard papers. It is important to realize that “peerreviewed” is not equivalent to “proven.”

While peer-review is not a vaccine against sloppy or fraudulent work, it is still the best safeguard against dodgy science.