A free service for scientific peer review and publishing

much ado about noting

It's gotta sting [UPDATED]

Peerage of Science
10 Oct 2013

Last week, Science published a news story by John Bohannon, describing a "sting-operation" designed to expose poor editorial practices in some journals. Predictably, people on all sides of several fences were accordingly amused/enraged/shocked, and internet is ablaze with indignation.

Two journals which have been accepted to participate in Peerage of Science – PhytoKeys and PLOS ONE – were targeted by Bohannon and both journals successfully rejected the spoof paper. Beyond that, there is no need to use further bandwith here on the details of the sting, nor on the biases in its design and reporting.

Instead, let's look at what we need to do.

Scrutiny of peer review standards should particularly target those entities we place our trust on.
The mere label of "peer-reviewed journal" is self-designated, and there are few scientists naïve enough to take that self-designated badge at face value. Authors and readers seek external validation of that claim. There are many things to judge, but in all honesty, almost all of us will seek out what the journal's impact factor is, before even considering to judge whether its editorial board is trustworthy, whether its published articles appear to be legitimate science, and whether people we can respect academically have published in the journal. With all its problems, indexing by Thomson Reuters is still used by most scientists for judging journals they have no other information about. It works because the inclusion process is (supposed to be) a guarantee that someone has taken a critical look at editorial practices of the journal, and has analysed how much work published there is cited elsewhere, and if necessary rejects the journal's application to be included.

But inclusion in even the most prestigious indexing databases falls short of ensuring the organisation does not shrink from its responsibility to diligently use peer review in publishing decisions. It is clear we need peer review of peer review. Not just for individual manuscripts (which Peerage of Science already does), but for any entity claiming that material it contains undergoes peer review.

We need a way to take an independent look at a claimed peer review process, evaluate it, and show the result of that evaluation.
A sting operation is one way to do that. Another way is to create an independent certifying entity that only awards certification to journals that regularly demonstrate sufficient peer reviewing standards. And a third way is the one employed by Peerage of Science: continuously employing internal peer-review-of-peer-review, and exposing that to editors, authors and reviewers. Needless to say, we think the last option is a really good policy that all journals should adopt, either by implementing it themselves or by participating in Peerage of Science.

The sting approach has one merit distinguishing it from the less confrontational alternatives: the mere threat of a possible sting operation, and threat of public exposure of failure, forces peer reviewing organisations (whether they are journals, or third-party specialist organisations like Peerage of Science) to be continuously vigilant. Unlike the more benign quality control mechanisms, it forces even those organisations that opt to stay out of certifying bodies, and those that opt to keep their peer review process opaque, to also maintain high standards of peer review or risk being publicly humiliated.

[UPDATE: the word "sting" is ill-advised and needlessly negative. What we endorse is an "audit", done responsibly and in good faith. Wording changed on 12 Oct.]

Peerage of Science welcomes more audit operations in academic publishing.
Importantly, although Peerage of Science does not do any publishing decisions itself, this also means we welcome audit operations targeting Peerage of Science.

Any Peer who wants to conduct an audit operation in Peerage of Science is welcome to do so. You may write a fake paper that claims exciting novel results but has fatal errors planted in methods and inference, get the paper with planted errors peer reviewed in Peerage of Science, and report the outcome of your audit operation in your chosen platform (but note that publishing any Peerage Essay requires consent from the reviewer). You can do this without any fear of repercussions from Peerage of Science, assuming you do not engage in fraud by accepting a publishing offer if one is made to you by a participating journal. To say that again: if you plant an audit paper in Peerage of Science, you must reject any publishing offers you may get.

Naturally, if you then report your audit operation under your own name somewhere (like a blog), possible backlash from reviewers or others is beyond our control. We therefore also offer an option for you to contact Peerage of Science administration once your audit operation is done, and ask Peerage of Science to report the operation on its website. Journalistic protection of source is implemented, and your identity shall not be published, and of course the reviewers also stay anonymous. In case you received a publishing offer, that fact will be stated, but not the identity of the journal. If you want to reveal names in public, you need to have the courage to do that yourself.

Based on the experiences of the first two years of Peerage of Science, we have strong reasons to be confident that planted errors in manuscripts will be identified and peer review shall recommend revision or discarding of those parts of the audit manuscripts. We have already seen that reviewing Peers are extremely conscientious, producing on average much better quality reviews than traditional processes do. If and when audit operations targeting Peerage of Science occur, the process will be analysed and we will do our best to learn and use the opportunity to further improve our peer review process if any weaknesses are found. An audit operation targeting us will be met with a positive attitude. That being said, Peerage of Science is also actively and vigilantly enforcing the qualification requirements of its reviewer community. We have suspended the reviewing rights of one Peer last year, and shall not hesitate to do that again if we find seriously incompetent or purposefully biased reviewing. If you are aware of an audit operation, you shall not engage to review that manuscript.

Peerage of Science shall not initiate any audit operations itself; if any occur they are always independent operations by some colleague somewhere. However, it is understandable that as a reviewer you may feel frustrated if you find out you have just spent time reviewing a paper that was not real. But the work you unknowingly did to build and guard better peer review should be something to be especially proud of, and shall be rewarded as such. Reviewing a paper that is later discovered to be an audit operation shall be positively acknowledged by doubled review balance increase. Furthermore, correct identification of the planted errors shall be rewarded by special recognition in reviewer profile, which you may display if you choose to.

So, let's all enjoy the better peer review we create together, complete with its occasional tests by responsible colleagues. Engage, review, audit like a bee, float like a butterfly.

« | »

peerage of science Facebook peerage of science twitter peerage of science twitter