A free service for scientific peer review and publishing

much ado about noting

On control in peer review

Janne-Tuomas Seppänen
04 Jan 2012

Discussions about Peerage of Science with chief editors of journals have been mostly enthusiastic, though of course doubts and concerns have been expressed as well. One of the concerns in particular is recurrent in different conversations.

It is about control. Or more precisely, as one chief editor put it: "our journal would lose too much control".

Now, in some domains the desire for control in peer review is evil. For example, imagine a publisher instructing editors to favour manuscripts coming from prestigious institutions or individuals, desiring to control whose research is published so that the rub-on prestige is maximised. Or imagine an editor appointing individual reviewers trusted to give a positive opinion on the importance and novelty of a manuscript espousing a pet hypothesis, desiring to control whose expertise is consulted. Evil. But because such evils are almost never discussed in the otherwise abundant contemporary debate on peer review, I am sure they never happen. Or at least, I will not contemplate the darkness further.

Fortunately, the response of the chief editor mentioned above, as well as others, also included a friendly and detailed description of the nature of control that is the subject of concern here. It is not evil desire for domination, but a legitimate worry about the role of established editorial boards, and about the value of reviewer databases and relationships that many journals have built over years if not decades. From discussion with several editors, below is a merged and edited summary of this concern, followed by a somewhat lengthy answer from Peerage of Science:

Concern:

"Our journal has an established editorial board, and substituting our current editorial process could not be undertaken lightly. A lot of effort and resources have been invested into having our own board of reviewing editors and streamlining the review process, and we are managing the process fairly well.

Our editors select reviewers from an extensive database built and maintained over the years by the journal, with information about the reliability of a large number of reviewers. Delegating this critical process to another body prevents developing relationships with editors and reviewers, which is valuable to us. The most reliable reviewers often exhibit loyalty to the journal both as reviewers and authors."

Answer:

The editorial board and the reviewer database and contacts are very important assets of any journal, and Peerage of Science cannot and does not seek to diminish the value of these. On the contrary, journals can gain leverage for both the editorial board and the use of reviewer databases and contacts by taking advantage of Peerage of Science.

The traditional process from journal's point-of-view broadly includes three steps before the final publishing decision: receipt of manuscript, decision whether to peer-review, and management of peer review. Journal policies vary on who is responsible for each step; it could be journal staff, member of the editorial board, or the editor-in-chief (EiC). Journals can have different policies in Peerage of Science too, and the service can facilitate continuity with many of the journals' existing practices. Here are some examples how existing practices in using editorial board and reviewer contacts could be brought to bring enhanced advantages in Peerage of Science:

1) receipt of manuscript In Peerage of Science, "receipt of manuscript" becomes "discovery of manuscript". Discovery does not mean browsing the database daily to find manuscripts, but using the automated and social tools Peerage of Science offers, to get notified about potentially interesting manuscripts. In many journals initial handling of manuscripts is done by paid staff. This can continue in Peerage of Science too, but allocation discovery to the established editorial board would probably offer the journal better advantage. Editors have extensive expertise, judgement and networks in their own fields, and can rely on these to discover manuscripts. They may choose to be alerted on all manuscript submissions, or when particular keywords occur in the submission, and rely on extensive network of trusted colleagues clicking referrals to them, and accept anonymous referrals from authors and reviewers. Having larger or better editorial board, or more skilled staff, translates to quicker and more accurate discovery of promising manuscripts, much like in the traditional process. The difference is that the person tasked with manuscript discovery can be proactive in helping to acquire articles that otherwise would have been submitted elsewhere.

2) decision whether to peer-review In Peerage of Science, "decision whether to peer-review" becomes "decision to track". In essence, a person responsible for this step decides whether the manuscript appears worth the time and effort it takes to follow it as it automatically progresses through the peer review process. In many journals it is the EiC who makes this decision, and EiC can continue to do so in Peerage of Science. For example, if discovery is allocated to members of the editorial board as outlined above, they click referrals to EiC who then decides whether to continue tracking the proceeding or not. However, it is probably advantageous for journals to divide this step between the editorial board and the EiC. In this model, other editors would initially choose whether to track a discovered manuscript or not, and as the peer review process advances they then either drop it or make a referral to EiC (or other deciding editor) depending on their own judgement or pre-defined criteria. The deciding editor thus has a pre-filtered feed of peer-review proceedings from which to choose what to start tracking and what to drop. Decision to drop is not permanent: if the proceeding later reaches a level triggering journal's search criteria or is otherwise re-discovered, editors can start to track it again.

3) management of peer review In Peerage of Science "management of peer review" does not exist, but this does not mean that journals would be left without means to participate in the process. As the editors pointed out in their concerns, a journal may have extensive reviewer database about the expertise and reliability of reviewers, and loyal relationships with reviewers. This advantage can be leveraged in Peerage of Science. If scientists included in their database are not yet Peers, the editors can themselves invite, or the journal can ask service administration to invite on their behalf, any or all of them into Peerage of Science. Most would probably wish to join, given the personal rewards available in the system, and due to the recognition of expertise and reciprocated relationship that an invitation from a journal exhibits. Now, after having discovered a promising manuscript, editors can use their in-house database and existing contacts to identify good reviewers and then click referrals to them in Peerage of Science. Referral coming from an editor is personal, i.e. it identifies the recommending editor and journal by name. Peers may choose to accept referrals from editors only, or only from editors of journals they feel loyal to. In essence, with editors using the referral feature, Peers have the option to do peer review very much the same way they did in the traditional system.

This can be enhanced further with a new feature. Referrals coming from editors could include "opportunity welcomed" and "opportunity passed" links, so that the Peer receiving the referral has the option to inform the editor if they intend to review the manuscript or not. Adding this feature would make joining easier and more attractive for many journals, as they could take fuller advantage of their reviewer databases built over years, and the loyalty relationship they have with individual reviewers. One design principle of the service is that there is little point in disabling interactions that can be achieved outside of the system anyway (Peers can reply to referrals "out-of-band" already), and the benefits of adding such reply links to referral messages are clear. Nonetheless, this issue would benefit greatly from community discussion before the feature is implemented.

About Janne-Tuomas Seppänen

One of the founders of the service, and a postdoctoral researcher at University of Jyväskylä, Finland.

I enjoy thinking and building things at night. The day is about three hours long where I live, so it is occasionally productive also. When the world is away there is more of a chance of finding that quiet immersion to flow. Then I got involved with a web service that has global userbase and the world does not go away quite as much anymore. Fortunately the nights still remain enjoyable enough and sufficiently unproductive that it rarely feels like "work".


« | »

peerage of science Facebook peerage of science twitter peerage of science twitter