How Many Papers Should You Review?

Peer-review is essential to keep science on track. But it can be too demanding on scientists, who are asked to do it for free and anonymously. How many papers are fair to review or to decline?

Note: A full and much better revision of this post at this link. I strongly recommend you just go there. 

Not long ago, I asked a top-notch scientist how often he accepted papers to review. His answer was curious. He told me: “I publish 30 to 40 papers a year. Each one takes two or three reviewers, making about 100 reports. I feel that I have to give this same quantity of reports back to keep the system working. Therefore, I review about 100 papers a year.”

I was a bit surprised. First by the reasoning, which was new to me; second by the number. I asked him how could he find time to review 100 reviews per year, roughly two papers per week. And I was again surprised by his answer: “I don’t really do it myself. I ask my students to review them.”

I think he read the ethical concerns on my face, as he continued somewhat defensively: “I think that this is a good exercise for my students. They learn how to critically read a paper, survey references, write a technical report. Of course, I check each review before submitting it.”

I don’t know numbers for that, but I don’t think that outsourcing a review to students is rare. Many of my colleagues told me that, as students, they were asked by their advisors to review papers. The practice is even captured in this PhD Comics cartoon.

phd111214sWhen I asked that professor about his “review policy,” I was trying to get some insight to myself. I usually get one, maybe two review requests per week. Naturally, I have to decline most of them, otherwise I wouldn’t do anything but peer-reviewing, which would ironically grant me to be excluded from the “peer” club in no time.

I basically only accept to review papers that are of immediate interest for me. It may sound opportunistic (and it really is), but seeing what other people in the field are doing ahead publication gives me some competitive advantage.

On the other hand, I don’t ever ask students and co-authors to do the job for me. What I often do is to write back to the editor declining the invitation and suggesting that the review request is sent instead to one of my co-authors, who I think is qualified to do the job. Maybe, in few occasions, I also suggested that some especially boring and troublesome papers were sent to people who I didn’t have in good esteem. (Don’t look at me like that. Nobody can be nice 100% of time…)

To transfer the review job to students is against the rules of most journals. Normally, they explicitly state that the person invited to review must not show or discuss the paper with anyone else, and if an additional opinion is required, the editor should be informed.

Even more serious, to transfer the review to students most probably lowers the review quality. I’ve received many reports in the past, filled of primary statements, which clearly denounced that someone with little experience wrote them.

Especially concerning postdocs, who are usually qualified to review a paper, it’s much better to suggest them as alternative reviewers than to ask them to informally do the job. This will help them to improve their CVs and also help editors to be aware of new people coming to the field.

In any case, I liked that professor’s rational that to keep the system working we should give back the same amount of reports as we get to our own papers. Indeed, I’ll adopt it to guide my peer-review policy from now on.

Putting everything together, I’m thinking about sticking to the following rules:

1. I will not ask any other person to do the review for me.

2. If I should decline an editor’s invitation, I may suggest one of my young co-workers holding at a doctor degree as an alternative reviewer.

3. In a given year, I will review a number of  papers that equals the number of reports I had in the previous year divided by the number of my co-authors holding a doctor degree. <This rule is bullshit, see the Erratum below.>

As for the last rule, let me give an example. In 2014, I published 10 peer-reviewed papers. It was a good year, there were no rejections or resubmissions. These papers—I just counted—demanded 25 reports and were co-authored by 24 colleagues holding a doctorate. Therefore, each of us should give back one report now in 2015 to break the system even.

These rules sound fair, but I can’t avoid thinking I’m missing something about the last one <And in fact I was…>. If I’m supposed to review only a single paper per year to keep the system balance, why do I still receive over 50 invitations to review every year? Are the editors too restrictive with their reviewers’ lists? Do too many people suggest me as a potential reviewer when submitting their papers?

Anyway, I should have thought of that third rule before. Only one or two reports per year, I do really like how it sounds.

It also means I already did too much review work this year. And, you know, I hate overworking.

MB

 


 

Erratum or “Ops, my bad…”

In the comments below, Richard Mostert pointed out that my Rule 3 was fishy. He wrote:

“Your reasoning is sound if you wrote the 10 papers as a group of 25 researchers. Then one each on the basis of these ten papers evens things out. When one of these papers was a review article with you as sole author, you ‘owe’ the community 2.5 reviews just for publishing that single paper. Your 24 co-authors on the other nine papers will not do that for you, in all fairness.”

And he’s right.

Using the example I gave, my publications in 2014, I should count how many reviews I should do to even things out, in the following way:

Paper Reports PhD Authors Reports/Author
1 2 3 0.7
2 3 9 0.3
3 3 1 3.0
4 2 1 2.0
5 3 4 0.8
6 2 4 0.5
7 3 5 0.6
8 3 3 1.0
9 2 3 0.7
10 2 4 0.5
Sum 25 37 10

For Paper 2, with 3 reports and 9 PhD authors, each author owes only 1/3 of report. But for Paper 3, where I was a single author, I owe 3 full reports. Summing over all papers, I should review 10 papers in 2015 to break things even.

Then, I should re-state rule 3 as:

3. In a given year, I will review a number of  papers that equals the number of reports I individually received in the previous year. This number is given by the reports/PhD-authors ratio for each paper published in the previous year, summed over all those papers.

It’s amazing how we can always use mathematics to get an answer reasonable, logical, and wrong.

MB



Categories: Science Policy, Scientific Culture

Tags: , , , , ,

11 replies

  1. To this: “why do I still receive over 50 invitations to review every year? Are the editors too restrictive with their reviewers’ lists?”. It is ok, as you may serve like a hub distributing them to your “young co-workers”. Editors (and authors) are also humans and they do not necessarily have to know how qualified are all those “young co-workers”, so they do a better job by sending it to more qualified “hub” than taking risk of sending it to less established scientist.

  2. Your reasoning is sound if you wrote the 10 papers as a group of 25 researchers. Then one each on the basis of these ten papers evens things out. When one of these papers was a review article with you as sole author, you ‘owe’ the community 2.5 reviews just for publishing that single paper. Your 24 co-authors on the other nine papers will not do that for you, in all fairness.

  3. As Richard indicated, if you want to know how many reports you “caused”, you have to calculate it for each paper and then take the sum:

    N=Sum_n R_n/A_n=R_1/A_1+R_2/A_2+R_3/A_3+…

    N: number of reports to write
    R_n: reports received for n^th paper
    A_n: authors with a doctorate on n^th paper

    You will probably end up with around ten reports to write, not one (sorry to take you out of your dreams).

    In your formula, you take the sum as N=(R_1+R_2+R_3+…)/(A_1+A_2+A_3+…). Some high school students might feel relieved to know that even professors make such mistakes…

Trackbacks

  1. ביקורת עמיתים – עבר, הווה ועתיד | חשיבה חדה
  2. Two Things Papers Don’t Show, but Should | Much Bigger Outside
  3. How Many Papers Should I Review? – Much Bigger Outside

A penny for your thoughts...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: