Peer reviews in Peerage of Science are done by scientists who freely engage peer review processes of manuscripts they consider worth their time and expertise. Journals who have full access to the system do have features facilitating soliciting reviewers the traditional way too, but usually the reviewers found the manuscript themselves via automated alert, or via referral from authors or other colleagues.
This reviewer freedom, combined with the default triple-blind policy (though editors have the option of asking if reviewer wants to disclose identity), is something quite new for journal editors and publishers. And the natural questions then are: Who are the reviewers in Peerage of Science? Are they qualified? Can we trust these people?
Here is a look at reviewer career stage distribution of the 252 peer reviews in Peerage of Science to date. The reviews were done by 145 individual scientists, and I googled the present academic position for each.
Reviewers were categorized to these four career stages:
- Professors - people with tenured positions, directors of research facilities, and equivalent.
- Mid-career - people with permanent positions, usually principal investigators starting their own research groups, readers, lecturers, etc, or more than 10 years since obtaining PhD.
- Postdocs - people with postdoctoral projects, or if job status not known, less than 10 years since obtaining PhD.
- Students - mostly people pursuing PhD. To be able to do peer reviews in Peerage of Science, these scientists too must have published at least one peer reviewed article in an established international journal, as the first or corresponding author.
Reviewer career stage distribution of 252 peer reviews in Peerage of Science
As you can see above, almost half of the peer reviews done in Peerage of Science are by firmly established scientists: people with permanent research and teaching positions, and more than 10 years of experience after their PhDs. Reviews by postdocs comprise about the same proportion as those by older scientists combined. Finally, the probability that a peer review is done by a published scientist who does not (yet) hold a PhD, is about one in ten.
But we can tease out more interesting numbers from the database too.
If the career stage of a person has no influence on how likely they are to engage as a peer reviewer in Peerage of Science, then the career stage distribution of peer reviewers should reflect the general career stage distribution of all Peers who have the opportunity to engage. Guess if that null hypothesis holds?
In a random sample (N=145) of our total Peer "population", students comprise 9.0% while 11.5% of reviews are done by students. No significant difference there (χ2=0.629, p=0.428). So, the few students who have the rights to peer review, do so at the expected frequency.
Postdocs comprise 28.3% of our Peers, but 42.1% of peer reviews are done by this class of scientists. Tradition and ritual dictates one must state statistical significance for obvious things too (silly, I know), so yes, this is a clearly significant difference (χ2=7.504, p=0.006). Postdocs do much larger share of peer reviewing than their proportion in the academic population would predict.
Mid-career scientists comprise 39.3% of Peers, and have done 32.1% of the peer reviews to date. Probably no real deviations there; associate professors, lecturers and their ilk perform peer reviews as often as one might expect (χ2=2.085, p=0.149).
The remaining class of Peers (primus inter pares), Professors, comprise 23.4% of the total population, while 14.3% of peer reviews were done by this group. Here is our other deviation, clearly something you would not expect by chance alone (χ2=5.320, p=0.021). Peer reviews are done by Profs less often than you would expect based on their numbers.
There are probably many good explanations for this pattern (including the one that my statistical methods are wrong and I should have used multinomial logistic regression or resampling, but hey, publishing a blog is a button). But below are a few arguments that come to my mind.
Postdocs have fewer teaching and administrative duties. Postdocs want and need to learn what others in their field are doing right now. Postdocs really, really, need and want all things that may help them getting the next grant, or maybe someday one of the few tenure-track positions, and Peerage of Science provides academic reputation metrics, i.e. something quantitative to put into a CV. And perhaps postdocs tend to be more idealistic and more eager to try new ways of doing things.
Professors, on the other hand, have little need to pursue better peer reviewer reputation. They are already well respected, and hence under a deluge of review requests from journals anyway. Many are also journal editors themselves, and (rightfully) contend that the effort they put into making publishing decisions for dozens or hundreds of manuscripts a year is more than enough of a contribution to the community, and thus do very few peer reviews on top of that. And, sadly, Professors have most of their days scheduled full of stuff other than thinking about science (so they hire postdocs to think about science).
Does the Prof vs. Postdoc phenomenon happen just in Peerage of Science, or is the situation similar in traditional peer review systems? I don't know, and chances are journals won't tell if you ask.
But I have heard several journal editors (unofficially) say they prefer to solicit reviews from postdocs rather than from the most senior scientists. The empirical pattern evident here is probably one reason why: it is easier to get postdocs to agree to a reviewing request. Some also say younger scientists tend to be more thorough, and put more effort into justifying their arguments. Whether the peer-reviewed peer reviewer quality, blind to career status (remember, Peerage of Science is triple-blind), is really any better among early-career scientists will be an interesting topic for analysis once there is more data in Peerage of Science.
Meanwhile, only ten days before the winner of the 2nd Annual Reviewer Award will be announced, on August 25th. Is the best peer reviewer this year some newly minted postdoc, or an up-and-coming lecturer, or an eminent professor? You'll find out soon.