I have had the literary misfortune of reading The Lost Symbol by Dan Brown, and I've idled my mind with the film National Treasure, and I have wasted precious sleeping hours touring the lunatic fringe of the internet, and also, regrettably, I have occasionally watched Fox News. If you have done any of these things, you too have been duly warned that Washington, D.C., is the seat of colossal conspiracies and secretive organisations with dark motives and inordinate influence. So, when I received an invitation – from an organisation with acronym formed from the letters N, and an S, and an A – to participate in a meeting in that city, a meeting which is not publicly advertised, to which you cannot just sign up, which has a strict communications embargo, and included speakers from the White House and the US Department of Energy, how could I refuse? Surely this would be the ante-room of masonic Bilderberg-level privilege in the new world order, and a reason to consider getting a white cat and practice my deep supervillain laughter.
Alas, no. There were no black helicopters involved. The occasion was the NAS Journal Summit, and it was just as nefarious as you would imagine a bunch of academics and journal managers discussing publishing is. Which, come to think of it, can be imagined to be pretty nefarious nowadays. But I promise it was not. Instead, it was a well-organized opportunity to hear many short but sharp talks on various current topics in academic publishing from many angles, with a program designed to leave more time for questions and commentary from the audience than conferences usually do. I liked that concept, and it worked quite well.
However, there was one comment on my talk that I did not have time to address before a coffee-break beckoned. As the issue raised is quite important, and although the commenter probably does not traverse these domains, I will execute my parry and counter here.
I had just spoken about the importance of improving and guarding the quality of peer reviewing through peer-review-of-peer-review. One of the men in black draw a sabre at that, by stating that there has always been peer-review-of-peer-review, and it is this thing called Editing.
Well. En garde, prêt, allez.
I would certainly agree that within and for a given journal, an editor should be the ultimate judge and arbiter of the quality and truthfulness of peer reviews. Indeed, diligent and skilful execution of that responsibility should be required, enforced, and praised. Prise de fer.
There are however two serious problems with equating editorial responsibility with peer-review-of-peer-review. By itself, in both execution and in outcome, it does not achieve what peer-review-of-peer-review is designed to do.
First, the editorial form of peer review on peer reviewers largely takes place pre-request. That is, based on accumulated knowledge and available data, the editor assigns a person – a peer – he or she trusts to be qualified and adept in providing judgement on the manuscript. In itself, there is nothing wrong in this, and Peerage of Science encourages editors (and everyone else) using the system to actively send out Referrals to solicit good reviewers to a process.
Of course it would be nice if, in the traditional system too, that first step in editorial peer review of peer reviewers was then always later followed by careful comparison and judgement of all the texts.
But I would need to see very solid data to be convinced that many editors in the real world, when receiving negative peer reviews, actually then read the entire manuscript and then carefully compare, contrast and judge the arguments in peer reviews relative to the research presented. Particularly if they are not full-time editors paid to do that, and if they have to continuously handle a large volume of submissions, most of which they must reject (i.e. are editors in high-profile journals). Knowing how busy we academics imagine ourselves to be, and how much pre-assigned weight any comment by someone at the top of scientific hierarchy carries, I am fairly certain that many editors, especially in the top-tier journals, read the peer reviews first and won't bother with reading the manuscript if their trusted expert dismisses it. You may have received editorial decision letter yourself testifying to such event. Riposte.
Second, even when the editor has the time and diligence to compare the actual peer review texts to each other and to the content of the manuscript and judge their merits, what is the consequence of that judgement for the reviewer? And more importantly, if there are consequences, how does that impact the scientific endeavour, and us, the scientific community?
If the editor judges a peer review to be hastily written, poorly justified, or even blatantly biased, what is the fallout facing the peer in question? The only direct repercussion I can see is that this peer would not receive (many) reviewing requests from that editor in the future. If the journal maintains a database on reviewer performance, the request-blacklisting might be available to other editors, or to other journals in that family, but I have never heard that such performance databases would be shared with competing journals, let alone made public. Possibly, the peer ends up feeling ignored and silently weeps for the hollowness of his inbox, but chances are he is just relieved.
Conversely, when the editor receives a review on which the peer devoted considerable amount of time and thought, a well-written, carefully justified, succinct and insightful evaluation text, what is the reward for the peer reviewer? Again, the journal's performance database probably gets an entry, resulting in more frequent reviewing requests from that journal and possibly from others affiliated with it. The feeling of being trusted is intrinsically rewarding for most of us, so that alone could motivate keeping up the good work, but for how long? Maybe the journal publishes once a year a list of best reviewers. Perhaps the journal has policy to publish peer reviews along with papers, giving credit only when the target manuscript merits publishing. However, none of this helps much in grant competitions, in job interviews, or with tenure committees.
In short, the old system, even when it works, does not provide an effective and transparent reward and punishment system on this task that has such a central role in science. Double riposte.
Unless... If a reviewer fails, the editor in question may now have a negative personal perception of this person's scientific aptitude, and might feel let down, frustrated and angry, and those negative perceptions may carry many consequences. Conversely, if a reviewer delivers, the editor now has a positive personal perception of that person's scientific aptitude, perhaps even feels gratitude, and those positive perceptions may carry many consequences. Some say this positive or negative personal impression made by the peer reviewer, on the editor or on others in the journal's editorial office, fuels a mechanism that provides rewards and deterrents.
The rationale they present is that the editor is usually a senior figure, an authority in the reviewer's field of science, and whether that person views you favourably or negatively matters. They argue that not only is the editor hoped to be more welcoming with reviewer's next own submission to that journal; he or she probably also sits in various grant evaluation boards, does hiring for his or her own group or the department, has a vote in tenure committees, and more...
Is this not the very definition of corruption!? This, my dear colleague, is the filthy cradle of our old-boys-club in science. And yes, in addition to the general harm and injustice and exclusion, where this particular abomination exists it can also literally exclude women, and other groups of people not currently having fair presentation in the upper echelons of science. Furthermore, injustice and inequality of opportunity can easily result even if the person in power is not purposefully biased and actually tries to actively guard against misplaced gratitude: people have vastly differing initial opportunities to demonstrate genuine ability this way to the right people, depending on who you get to meet at your home institution, and where you can afford to travel to meet people.
All human pursuits are inevitably subject to human nature. Indeed, just scroll to the top and you'll see me starting this very post with an attempt to get adoration by association. Browse back to the early posts and you will find me admitting that seeking name-recognition was an important motive to sign all my reviews when I was young. Touché. But do not claim that our human weaknesses are a feature instead of a bug. Science really needs neutral, double blind peer-review-of-peer-review.
And returning to acronyms, all scientific publishing should be PROPR.
About Janne-Tuomas Seppänen
One of the founders of the service, and a postdoctoral researcher at University of Jyväskylä, Finland.
I should say sorry to Dan Brown; your plot twists are sometimes silly but I've read four of your books so I must have enjoyed some of your writing too. Besides, if ever encountered in fiction, I would also mercilessly ridicule a series of plot twists whereupon a man first discloses to the world a global conspiracy of such proportions that people now receive the number of an IT-specialist instead of medical prescription if they complain to a psychiatrist that the government is eavesdropping their home, and that man is then himself marooned in a city that has television programming compared to which Orwell's nineteen-eighty-four is positively "fair and balanced", and then the little ruler of that city, who sometimes flies with cranes, insists that the little green men he sent to invade the very place where Potemkin's villages once stood, are not really there. Like Haldane almost said: "Now my own suspicion is that the Universe is not only more ridiculous than we suppose, but more ridiculous than we can suppose."