I finally get answers to that one. For my most recent review it is 3.8, slightly better than the other three that reviewed the same manuscript. The question is one that has lingered long in my mind, "What are my reviews worth, anyway?" No one has ever really told me, and at this stage in my career, I am mildly embarrassed to admit I even ask it. Over the years, I have had to infer the answer from editorial comments to authors and by comparing my review to those of the other reviewers; I can't help but take notice when the contents of my review do not at all match theirs. This has often led me into the trap of convincing myself that my review must have been lower quality.
Answering the question posed above is a somewhat unexpected perk of Peerage of Science. Even as a member of Peerage's Board of Governors, where I feel I have a pretty good handle on everything, I was a little unsure just how things would go when I clicked the "Engage" button on a recent submission. Let me summarize the process. After reviewing the paper, it is time to send in the "essay". Some wonder exactly what this is. It can be in a typical review format (as mine was), or can wax more philosophical and discuss ideas (potentially being worth publishing itself). In either case, it should give your view of the science presented in the manuscript, in 1000 or words or less, written carefully enough that it can be evaluated by other reviewers and is useful for editors in decision-making. At the same time you submit your essay, be prepared (as I was not) to also enter some words in a "Comments to Authors" box. This can be anything you want and as long as you want; for example short comments summarizing recommended action for authors, or detailed suggestions how to improve the manuscript line by line.
The magic for the reviewers begins when the first review phase is over and you are then allowed to see everyone else's review. The reviewers are identified only by a 'peer number', which is unknown to everyone. You then score each review based on a few multiple choice questions. Rightfully so, the authors do not take part in evaluating the reviews. When all reviews have been scored, your score is still not revealed to you or other reviewers until the proceeding is over (i.e., the revisions are done and the reviewers get a final multiple-choice evaluation of that revision). Because my review did not match the others, it was with great trepidation that I finally clicked to see the scores.
When we first enter science, most of us have had little instruction in the art of peer-review, and the common strategy of learning-by-doing is a fuzzy way forward. We begin as graduate students that critique already-published papers, usually in lab meetings or in reading literature germane to our theses. This is a good start, but probably because these are published papers that have undergone lots of polishing, we quickly learn the bad habit of making mountains out of molehills. Eventually, we get asked to review our first submitted paper and we go from there.
Over time, a phenomenon occurs in some people. Some simply skip wondering about the quality of their reviews; they either are convinced their reviewing skills are excellent, or they truly do not care. Either is an unenviable state. Consider the case of a recent discussion I had with a well-known scientist in my region about peer-review. After I raised the concept of Peerage of Science to him, the discussion quickly turned sour when he said he wouldn't join such a thing because "I don't have time for extra work or trying to write reviews to help authors. When I see a paper I don't like, I simply state that this paper is so poor as to not warrant any further attention. It saves the journal a lot of effort and allows the authors to move on". There are a few interesting aspects to this statement (and the obvious arrogance is not one of them): (1) the last part of the statement makes it seem as if the authors are being done a favour, and (2) the aspect of time/effort management comes up in three spots in the statement. Ironically, a major benefit of Peerage is that you can pick and choose what you review and when you review it, thus saving you time.
However, to me, the most fascinating part of the above statement is the idea that "it saves the journal a lot of effort". How can you save an inanimate object from effort? A corporation can be treated like a person under US law, and we seem to have similarly elevated the journal from a collection of printed pages to a living thing. I realize that the comment was not made consciously; I know he was referring to a journal board of editors. But, the point is the same – the "journal" is created by the authors, not the editors, and if the goal of reviewing is not to help the authors but the editors then isn't this the tail wagging the dog?
you wouldn't be reading this if you subscribed to the idea that an acceptable review could simply be "this paper is so poor as to not warrant any further attention". Thankfully, most people don't. We take pride in our work and care (at least a little) about the person(s) on the receiving end of a review, and therefore, it can be surprisingly gratifying to see an objective opinion of your essay. Click 'engage' next time you see a manuscript that grabs your attention and see for yourself.
A member of the Board of Governors and a Founding Peer. He is a government Scientist in Ontario, Canada and an Adjunct professor at 3 universities.
I could be writing a paper instead of a blog entry. But blogs aren't peer reviewed, so this is easier.