Peer review – reviewed.

We’re entering a time when the process of scientific peer review will likely undergo a careful scrutiny from all sides. For example, the UK Parliament has just launched one such inquiry. This is serious business, and merits considered thought, from all quarters, for an extended period. We want to get this right.

Why? Because peer review is a fundamental component of the hygiene of science.

By hygiene we mean the whole of those habits and ways of doing business that keep science healthy – that distinguish science from unfounded opinion and conjecture, from wishful thinking, from fantasy and delusion, from lies and falsehoods. These activities would include, for example, the development of verifiable hypotheses, the test of those hypotheses by experiment(s), the evaluation and articulation of the uncertainties in the work and conclusions, the publication of the findings in peer-reviewed literature for all to critique and evaluate, efforts to duplicate the findings in independent work, explorations of the implications of those findings and tests of those implications – and on and on.

What is peer review? Just what the name implies. It means this: before a paper is published, experts in the field are given an opportunity to examine the paper thoroughly, to check it for content, for errors in logic, for the consistency of its conclusions with data and experiment, for any mis-representations, for originality, for appropriate acknowledgment of relevant prior and contemporary work, etc. The list is extensive, and probably not completely captured here. But you get the flavor.

Note that experts in the field may often be in competition with the author or authors of the work in question: for funding, for scientific priority, for prestige and recognition. The stakes are high! Nevertheless, they’re expected to put these considerations aside. Because the stakes are high, reviewers are usually but not always guaranteed anonymity.

Sounds complicated, and tenuous at best. How can this possibly work? Here’s an analogy that might help – one that may be familiar to many readers.

When I was younger I played a lot of basketball. Only a small fraction of these games were actually run by unbiased third parties, a.k.a. referees. Virtually all my basketball was of the pick-up variety – street games.

We called our own fouls and kept our own score.

This worked well. At the time I didn’t think it in any way remarkable. In hindsight it seems amazing.

Most of the violations were called by the person fouled: “foul!” Maybe 95% of the time, these calls were accepted without argument. Many times, the foul would first be called by the person committing the foul: “I fouled him! Their ball!” without waiting for protest from the other side. Sometimes the foul would be called by another player. Occasionally, a call would be overruled by others, usually to their own sides’ detriment. And whether the score was close or not, whether it was near the end of the game or not, play would swiftly move on. There was very little argument. We all wanted to play ball!

Peer review is very much – though not entirely – like that. It’s pretty easy to see both the advantages and limitations of this approach. The situation is not unlike what Winston Churchill once famously said, in reference to the political arena: “It has been said that democracy is the worst form of government – except all the others that have been tried.”