Many months ago, when Peerage of Science was only a whisper among scientists, a group of prescient colleagues already saw the importance of independent public discussion about this. They decided to write a letter to the foremost trend-keeper of evolutionary ecology, aptly named Trends in Ecology and Evolution. Their letter is now published in TREE, and because Attila, Matteo, Marlene, Shirley, Franziska, Kerstin, Michaela, Wouter, Joël, Sarah and Dustin courteously notified us about their letter in advance, our response to it is published in the same issue.
Still firmly rooted in the age where text needs a wood extract to exist, TREE has tight 800-word limit for the letters on its leaves. I am sure Hettyey et al. had to leave out many things from their letter, so perhaps we'll see a blog post from Vienna here sometime soon. I know our response had space to discuss only the major conceptual issues, and many of the important details were left unanswered. But this blog offers plenty of space for seeding and weeding of ideas, so I will address the missed details here, in the order they occur in Hettyey et al.
Most reviewers prefer anonymity, yes. In Peerage of Science, reviewers are more anonymous than in the traditional system: even the editors are unaware of your identity (though a voluntary option to reveal your identity as reviewer to an editor who makes a referral to you might be a good idea. The idea is that everyone is anonymous during the process, but has option and incentive to reveal identity after the process. If all reviewers choose to stay anonymous, and The Proceedings of Peerage of Science hence never comes to existence, that is perfectly fine.
Remaining anonymous does not mean that writing a high-quality essay would be wasted effort. The reason to put a 1000-word limit and suggest that the text should read as a coherent essay is to facilitate the evaluation by other Peers. It is difficult to evaluate a review that is, for example, just embedded comments in an annotated manuscript file. Writing a high-quality essay - that carefully justifies the arguments and is coherent to read - returns higher score from others, and this builds your PEQ irrespective of whether you publish the text or not. Also, it is much easier for editors to make use of the review if it includes an essay that is of limited length and well-written. Details and informal comments can still be freely included in the comment-field of the review form.
Almost all reviews do contain original content - suggestions for analysis and discussion about the scientific implications of the results for example are mentioned in the evaluation instructions. In other words, correctly pointing out "this statistical analysis is wrong..." is valid criticism but without original content, while continuing that sentence with "...and the correct way to do this is..." adds original content to the review. It would be wonderful of course to also read from reviews novel ideas that lead to new scientific discoveries, but that is not necessary to have top-rated original content in your review.
Peerage Essay usually - and hopefully - becomes outdated after manuscript revision, yes. The Proceedings of Peerage of Science will offer reviewers the opportunity to add a post-script to the text, discussing the changes made in revision. Also, nothing prevents the reviewer taking the text to other publications (like a personal blog) and elaborate ad infinitum, just adding the stamp of peer review to their original arguments by displaying the PEQ of the original essay.
Purposefully biased reviews - either positive or negative - are a possible problem, and the freedom to choose what to review could be misused by a gang of dishonest Peers. However, it might be worth to take a step back here and look at the reality of the traditional system. Journals usually ask authors to suggest reviewers and also people authors wish to exclude from the process. Editors often choose from these names (and I have heard some prefer to choose from the exclusion list). The absolute worst consequence a biased reviewer faces - if the editor compares the review to the manuscript and notices the bias - is that the journal never again requests a review from them. So, any possibility that in the traditional system, sometimes, maybe, possibly, a purposefully biased review does harm and escapes without penalty?
Peerage of Science has three layers of defence against purposeful bias.
1) affiliated scientists and up to ten Peers authors choose to exclude from the process do not have access to the manuscript; compare this to the traditional system where it is not unheard of that a journal sends the manuscript for review to a junior author's supervisor next door - or even to one of the middle co-authors by accident.
2) triple-blind peer-review-of-peer-review incentivizes reviewers to justify their claims and blatant bias exposes them to personal loss in terms of accumulated PEQ; in the traditional system personal or institutional prestige may take the place of justification for arguments, and there is no adverse consequences for biased reviewer.
3) the final layer (which hopefully is never encountered) is investigation of and consequences for scientific misconduct. Suspicion of purposeful bias in either direction would be investigated by an independent panel, and if bias is judged to amount to misconduct, consequences range from temporal reviewing ban to permanent termination of user account. These far outweigh the modest gains one could gain from unethical promoting or attacking of a manuscript.
Can there be too many reviews for a manuscript? Possibly, but limiting the number of possible reviewers would arbitrarily violate the freedom of Peers to contribute to evaluation of science on manuscripts they choose. The issue and some possible changes to the system are discussed more here.
The indicators of interest towards a manuscript (numbers of search clicks, downloads, engagements, reviews) are likely to be useful pieces of information for editors. They may reflect current trends, but then again publishing decisions tend to do so anyway. The indicator does not reflect author or institution prestige, if authors choose to be anonymous as they so far have. Reviewers do not see these indicators, so their decisions on what to search, download, engage and write reviews on must be based on the science, possibly also triggered by referrals they receive from colleagues.
So new ideas stand or fall on their own merit, but have a much better chance of encountering reviewers who recognise their potential. Compare the free choice in Peerage of Sciece to the traditional system where editor consults two established silverbacks of the field, and possibly rejects manuscript based on that advice without even reading it.
One of the founders of the service, and a postdoctoral researcher at University of Jyväskylä, Finland.
Some years ago, I spent a few months in a city with a river flowing through it. Opus number 314 by Johann Strauss II has, unfortunately, become chromatically challenged in the past 146 years. It is grey now, like all the other rivers flowing through what 4000 people per square kilometre equals to. But, on the river, looking over the water at the culprits for its tint, the views are still beautiful. As far as cities can ever be called beautiful.