How Many Papers Should I Review?

Peer-review is essential to keep science on track. But it can be too demanding on scientists, who are asked to do it for free and anonymously. How many papers are fair to review or to decline?

Sometime ago, I asked a top-notch scientist how often he accepted papers to review. His answer was curious. He told me: “I publish thirty to forty papers a year. Each one takes two or three reviewers, making about 100 reports. I feel that I have to give this same quantity of reports back to keep the system working. Therefore, I review about 100 papers a year.”

I was a bit surprised. First by the reasoning, which was new to me; second by the number. I asked him how could he find time to review 100 reviews per year, roughly two papers per week. And I was again surprised by his answer: “I don’t really do it myself. I ask my students to review them.”

I think he read the ethical concerns on my face, as he continued somewhat defensively: “I think that this is a good exercise for my students. They learn how to critically read a paper, survey references, write a technical report. Of course, I check each review before submitting it.”

I don’t know numbers for that, but I don’t think that outsourcing a review to students is rare. Many of my colleagues told me that, as students, they were asked by their advisors to review papers. The practice is even captured in this PhD Comics cartoon.

phd111214sWhen I asked that professor about his “review policy,” I was trying to get some insight to myself. I usually get two or three review requests per week. Naturally, I have to decline most of them, otherwise I wouldn’t do anything but peer-reviewing, which would ironically grant me to be excluded from the “peer” club in no time.

I basically only accept to review papers that are of immediate interest for me. It may sound opportunistic (and it really is), but seeing what other people in the field are doing ahead publication gives me some competitive advantage.

On the other hand, I don’t ever ask students and co-authors to do the job for me. What I often do is to write back to the editor declining the invitation and suggesting that the review request is sent instead to one of my co-authors, who I think is qualified to do the job. Maybe, in few occasions, I also suggested that some especially boring and troublesome papers were sent to people who I didn’t have in good esteem. (Don’t look at me like that. Nobody can be nice 100% of time…)

To transfer the review job to students is against the rules of most journals. Normally, they explicitly state that the person invited to review must not show or discuss the paper with anyone else, and if an additional opinion is required, the editor should be informed.

Even more serious, to transfer the review to students most probably lowers the review quality. I’ve received many reports in the past, filled of primary statements, which clearly denounced that someone with little experience wrote them.

Especially concerning postdocs, who are usually qualified to review a paper, it’s much better to suggest them as alternative reviewers than to ask them to informally do the job. This will help them to improve their CVs and also help editors to be aware of new people coming into the field.

In any case, I liked that professor’s rational that to keep the system working we should give back the same amount as requested for our own papers, although I don’t agree that the number of reports is the measure of this work.

Then, how many reports should I give to break things even?

Suppose I submit a paper and this paper gets, say, two reports before final decision (no matter whether is was accepted or rejected). Suppose yet that I co-author this paper with two other colleagues holding a doctorate (and are eligible as reviewers). Therefore, we three owe the system two reports. And I alone owe 2/3 of a report for that paper alone, where 2/3 is obviously the ratio between the number of reports and of PhD authors.

Then, to know the number of papers I should review in a certain year, I should compute the sum over the reports-number/PhD-authors ratio for each of my papers submitted in the previous year. For me in 2014, for instance, it makes ten papers.

But this is too complicated. I don’t want to keep tabs on these ratios. I want a simple rule of thumb.

I think it is fair to assume that the mean reports-number/PhD-authors ratio is approximately one. Therefore, to break things even and keep system running, I should write one report for each paper I submit.

Putting everything together, I’ve been sticking to the following rules:

1. I will not ask any other person to do the review for me.

2. If I should decline an editor’s invitation, I may suggest one of my young co-workers holding at a doctor degree as an alternative reviewer.

3. For each paper I submit, no matter whether it is accepted or not, I should review one paper.

In principle, these rules sound fair, but it still a lot of work to cope with them. It takes me about five hours to review a paper. Then, I should dedicate a total of more than one work-week per year just reviewing papers. (And there is still the question whether it is fair to transfer these costs to my university, which is paying me for these hours.)

As for the last rule, still with my previous example, I submitted (and published) ten peer-reviewed papers in 2014, therefore, I should have reviewed ten papers in 2015. I just checked my records, and ,in fact, I reviewed fourteen papers. This means I overworked… and I hate overworking. Probably I should compensate for that now in 2016.

MB

 



Categories: Productivity

Tags: , , , ,

8 replies

  1. There needs to be a mechanism by which graduate students/PDFs are given the opportunity to learn how to review papers and do it well. Rather than ‘downloading’ reviews to students/PDFs, one approach is to review a paper and have a student/PDF in the lab also review the paper. Then sit down with the student/PDF to compare both sets of comments and discuss reasons for similarities/discrepancies between them. The supervisor can then submit a review to the journal that combines both sets of comment, acknowledging the contributions of the student/PDF in providing the final results. Yes, this takes more time, but may well lead to a better review and, at least as important, help the student/PDF become a better reviewer/researcher and future mentor.

  2. Actually, if you published 10 papers, assuming that sensible editors ask for 2 or 3 reviews, you should review 10 * (2 to 3) / (1- average rejection rate from the journals you publish in).
    That’s about 20-30 reviews for a 0% average rejection rate, or 40-60 reviews for 50%, or infinite reviews for a 100% rejection rate.
    If my math is right.

  3. So why doesn’t the prof ask that the journal deal directly with the students as the reviewers? That’s what happened to me as a PhD student (I work in mathematics). I guess that means he would be shown up as not actually doing any reviewing?

  4. I think the student review issue depends a lot on the approach. My PhD mentor was an AE for several journals and he would occasionally assign me to review (officially). He would tell me to send my comments to him before submitting my review and then we would discuss them together and compare them with comments from the other reviewer(s). It was a great way to learn how to be a peer reviewer (etiquette, in addition to critical analysis). I was embarrassingly successful as a student (>20 pubs in 5 years-several in top notch journals; h-index in the teens by the time of defense) and eventually got invitations from other AEs to review papers with work that was close to mine. The first time that happened it was a paper submitted to a high IF journal and I emailed the AE directly to let him know that I was just a student. He responded that he knew that and asked me to review it anyway. Apparently, he thought I did alright cause the review invitations started rolIing in after that. I’m glad my PhD mentor had prepared me for that moment. I’m several years past my PhD now and continue to be an in-demand reviewer for roughly 2 doz. journals so I’ve recently had to adopt a system such as the one you’ve described. I publish around 10 papers/yr so I try to review min. around 20 per year (due to the need for usually 2 per paper and the fact that not all co-authors are PhDs or otherwise qualified to review so the co-author ratio falls apart there), but I’ll take on more when I have a lull in my other activities. I really believe that it’s important to give back to the community that has embraced me as a scientist and allowed me to do something I love.

  5. Best advice I got was
    1) Review a number of manuscripts proportionate to the number you submit
    2) Do not review for a journal that you would not feel comfortable publishing in yourself.

Trackbacks

  1. How Many Papers Should You Review? – Much Bigger Outside
  2. Weekend reads: ORI staff revolt?; Excel creates big typos in papers; how to reward reviewers - Retraction Watch at Retraction Watch
  3. Credit for reviewing & editing — it’s about bloody time | ConservationBytes.com

A penny for your thoughts...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: