11 Ideas to improve peer review – Part I

Peer review is at the core of contemporary science. Although it has been doing an excellent job keeping junk research out, it has many flaws. I have a few ideas that could be used to improve the system.

Academic science turns around papers. As scientist, you do your research and then you make it public in a specialized journal. But before your paper is published, the journal editor will ask one or more specialists in the field of your research to review your paper and tell whether your findings are worth being communicated or not. This is the peer-review system and it is the core of science today.

Along the years, I’ve been playing the three roles: reviewer, editor, and specially author. Based on this experience, I collected a few ideas that could be tested to improve the system.

1. Clarification step

If someone tells you something and you do not understand well, what do you do? You say something like, “Pardon me, could you please explain your question?”

Reviewers can be awfully obscure sometimes. Authors often have to guess what they want, without the chance of asking for clarification.

The first step in the review process should be to grant the authors a period – say 48 hours – to request clarifications about the reviewers’ reports.

2. Review check-list

Reviewers’ reports are completely arbitrary. Apart of few general journal guidelines, each reviewer will focus on what he or she wants.

Editors should take reviewers by hand and guide them through the revision. A check-list is the way of doing that:

  • Is the abstract consistent with the paper findings?
  • Does the introduction adequately survey the literature? Anything missing?

3. Tiebreaker

Many top journals are adopting a “two positives or die” practice.

Editors send the paper to two reviewers and it is considered for publication only if it gets positive recommendation from both. This is bad. The reports are very much subjective. It’s comically common to get:

  • Reviewer 1: “Awesome work! Publish as is.”
  • Reviewer 2: “Are the authors out of their minds? Throw it into the garbage can.”

In such tie situation, to reject is a non-sense. But to save time and work, editors do it!

Please, editors, just ask for a tiebreak report.

4. Pay for review

Review work is unpaid. It is a tradition in the scientific community to do that for free. But we know, nothing is for free.

Reviewing a paper costs several working hours. Who is paying for that time? Of course, either the institution where the reviewer works or the reviewer, if the work is done in the free time. Thus, scientific journals have been cleverly transferring part of their costs to a third-party, many times, the public sector.

The practice of asking for a report on a paper comes from simpler times, where the editor was asking to a colleague a favor. Today, however, science and scientific journals are professional enterprises. They should adopt better practices.

The moral is:

Don’t ask me to do for free, one of the few things I know how to do.

5. Public reviewers list

To be a reviewer for a journal has symbolic value. It shows that the invited person is considered a peer, a specialist in the field. Specially for young researchers, this is an accomplishment that should appear in their CV’s.

However, how can you prove that you reviewed for a specific journal?

Editors could make public a simple general list of all people who reviewed papers for that journal. Simple like that, the value of being a peer reviewer would be acknowledged.

6. Authors shouldn’t do editor’s job: suggesting reviewers

Every journal asks for suggestions of potential reviewers (and also of people who should not review).

In principle, this is not bad, but it is often annoying. You have to waste time trying to remember people who could do the job well. Then, you tend to suggest big-shot names to show that you are not afraid of the revision, but also knowing that big shots are not going to take papers to review.

Editors could help with that by providing lists of potential reviewers from where authors could select.

. . .

That’s enough for today. I will tell the remaining 5 ideas to improve peer review in the next post.

MB

  • What do you think? Would these ideas help to improve peer review? Did I miss anything? Leave your comments orย likeย below.
  • If you enjoyed this post, follow the blog by signing up in the sidebar. You can also follow me onย Twitter.
  • Do you know how much $papers a scientist is worth? Check this post.
  • The illustration is based on the painting “The Alchemist” by David Teniers the Younger.

 



Categories: Science Policy, Scientific Culture

Tags: , , , , , , ,

19 replies

  1. I like your ideas. I’m a freelance librarian and lately I’ve been fact-checking for publishers of children’s books (arduous but worthy!). I know that not just anyone could peer review an academic text, and peer review is a lot more than fact-checking, but surely that makes the work even more valuable!

  2. I agree with most of your suggestions.
    regarding #3, from my limited experience in most cases there are 3 reviewer, but maybe i’m wrong.
    Anyway, in case of only two, with opposite opinions, the tiebreaker is the editor. After all, the editor has scientific training ( I hope…). He can judge which reviewer has more relevant points/criticism. It also depends how much the editor likes the paper. Sending it to a third reviewer means more work (obviously) but also another delay of several weeks for the authors, with no guarantee of the outcome.
    Regarding payment – I think there’s a problem with that – it can lead to biases and conflicts of interest. Who pays? If the university pays a global sum and same for everyone, then it ok.
    but if the payment comes from the journal, then whoever pays most will get more attention by the reviewer. Will experts be paid more than new researchers? will editors pick the “cheapest” reviewers? etc…
    item #6 – The editors may not know who is expert and who isn’t: fields get narrower and narrower…
    Also, remember that the editor has to deal with many submissions and the editor’s time is limited. you submit one paper every few months (if you have a prolific lab). spending an hour thinking about reviewers isn’t that bad.

    What is missing from your list (maybe its in the other 5) is a training program of how to peer-review.

    • The number of reviiewers denpends on the journal. Some of them really take three. Others use a third as tiebraker. But I have seen cases of “two positive reports or reject.” This I think is a bad practice.
      Concerning payments, I don’t really know how it should be done. I also see several practical problems there. But right now, publishers are transfering costs to research institutions and researchers. It doesn’t sound right to me.
      Finally, suggesting reviewers, I agree, is not a big deal. It is just annoying. I would prefere to be able to select names from a list provided by the journal.

      • “publishers are transfering costs to research institutions and researchers”
        Well, remember that in most cases, the researcher pays the journal for the publication, so the cost of publication is already transferred to the researcher… I imagine that if reviewers are to be paid by the journals, publication costs would go up. And if reviewers are to be paid by the university, then university funds would need to be allocated at the expense of other costs.
        Also remember that being a reviewer, you are privileged to unpublished data – which may inspire you and gets you advantage over another researcher in your field.

        Now, a penny, please, for my thoughts. Paypal is acceptable ๐Ÿ™‚

        • It really back fired… I never thought anyone would ask me for this penny! ๐Ÿ™‚
          Anyway, it’s true science has many symbolic ways of paying for stuff (I wrote about that before in http://wp.me/p3W89e-dC). Being reviewer has its value on itself. But symbolic value doesn’t replace real currency completely. Funding agencies commonly pay for review of projects, for instance.

  3. Dear friend, one more time you brought to us an important and essential question! Thank you for the criterious list! Release the last five!
    Well, I have one consideration about this subject. Nowadays, only reviewers can practice the comfortable anonymous judgement on the works of your peers. In my point of view, both of them, authors or reviewers, have to be anonymous. Otherwise, neither of them will have to be. This improvement can make the publishing game a little more fair!

  4. Dear Professor Baratti, I am a young scientist with limited publication experience and many interrogatives about how the editorial and peer review process actually work. I red your post with great interest. I find your proposals very interesting and very useful, but they seem to me not to address specifically some of the big issues that I seem to find with the actual publication system. One year ago I had my first paper published on a journal with a relatively low IF (~6). I must say that I found the observations of my reviewers very acute, very pertinent and very useful, and they definitely contributed to improve my paper. After a few months, another paper came out on the same subject on a Journal with a higher IF. This latter paper briefly cited the previous works on the subject by quickly listing them and generally stigmatising them as inconclusive; it claimed the novel discovery of things that were actually already reported in those publications that it cited; it drove speculations that were in open contrast with previously reported experimental evidence without discussing them; it drove some incoherent conclusions. Where did the peer review process failed? I was happy to read a new paper on the subject and to have access to new experimental data, nevertheless reading this paper was a very disappointing experience! I am still puzzling about how the peer review and the editorial system may have allowed the publishing of that paper such as it came out.. A discussion with my young colleagues pointed out that we all have many examples like the one that I have just reported and that we would like a more “fair” and deep scientific discussion in published literature and we would expect peer reviewing to guarantee this fair discussion more throughly. Are we the only once to see these problems? Do you think your proposals would solve these problems?

  5. I’m using your post as a checklist to see how we’re doing at Rubriq (rubriq.com) where we’ve created an independent peer review system outside of the journal lens. So far, we seem to be headed in the right direction based on your points, and I thought I’d give you some examples of how we’re trying to make all these work. Looking forward to the next five!

    1. Clarification step – to make sure that comments to authors are clear and complete, we have a member of our own team (all published researchers who have been trained in review evaluation) check over all the reviews. If needed, we will go back and ask reviewers to revise or explain their comments. The author can also contact our team with any questions about the reviews.

    2. Checklist – this is the piece we put the most work into before launch: a standardized scorecard that guides reviewers through the process. Check out a sample scorecard here to see how we divided up sections and assigned points to key criteria – http://goo.gl/rXv8ez

    3. Tiebreaker – we found out early on that three is definitely the magic number for reviewers because of this exact situation. Though the reviewers in our case aren’t making an accept/reject decision for any particular journal, there can still be areas of disagreement that benefit from a third perspective.

    4. Pay for review – we give our reviewers a $100 honorarium/stipend for each review. Most of them report that the standardized scorecard is more efficient, so it takes them even less time to complete a high-quality review. Our customers are either authors who are paying to get a pre-publication review (to make improvements and help find the best journal), or journals that are paying to outsource some of their reviews.

    5. Public reviewers list – we haven’t implemented this yet, but are working on a way for our reviewers to get public credit for their reviews. We hope to do this in partnership with ORCID, so that the review record becomes a part of a researcher’s official profile.

    6. Not asking authors to suggest reviewers – finding appropriate reviewers is one of the hardest parts of the peer review process, which is why journals ask you for help. We’re happy to get reviewer suggestions from authors if they have them, but almost always expect to find them on our own. That’s why building a database of reviewers is so important to us. If any of you are interested in becoming a reviewer for Rubriq, see the info here http://www.rubriq.com/reviewers/

    Lisa Pautler
    Research Square

Trackbacks

  1. 11 ideas to improve peer review – Part II | Much Bigger Outside
  2. Scientific Data Visualization: A Revolution in Standby | Much Bigger Outside
  3. Let Science Manage Scientists | Much Bigger Outside
  4. How Many Papers Should You Review? | Much Bigger Outside
  5. ื‘ื™ืงื•ืจืช ืขืžื™ืชื™ื โ€“ ืขื‘ืจ, ื”ื•ื•ื” ื•ืขืชื™ื“ | ื—ืฉื™ื‘ื” ื—ื“ื”
  6. Two Things Papers Don’t Show, but Should | Much Bigger Outside

A penny for your thoughts...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: