Conventional guidelines for referee reports

How to write a good review (general conventional guidelines)

Some general key points to consider

• Cite evidence and reference specific parts of the research when giving feedback.

• Try to justify your critiques and claims in a reasoning-transparent way, rather than merely ‘"passing judgment."

• Provide specific, actionable feedback to the author where possible.

• When considering the authors’ arguments, consider the most reasonable interpretation of what they have written (and state what that is, to help the author make their point more clearly). See steelmanning.

• Be collegial and encouraging, but also rigorous. Criticize and question specific parts of the research without suggesting criticism of the researchers themselves.

We are happy for you to use whichever process and structure you feel comfortable with when writing a peer review.

One possible structure

Core

• Briefly summarize the work in context

• Highlight positive aspects of the paper, strengths and contributions, considered in the context of existing research.

• Most importantly: Identify and assess the paper's key claim(s). Are these supported by the evidence provided? Are the assumptions reasonable? Are the authors using appropriate methods?

• Note major limitations and potential ways the work could be improved; where possible, reference methodological literature and discussion and work that models what you are suggesting.

Optional

• Discuss minor flaws and their potential revisions.

  • Please don't spend a substantial amount of time copyediting the work. If you like, you can give a few specific suggestions and then suggest that the author look to make other changes along these lines.

• Offer suggestions for research agendas, increasing the impact of the work, incorporating the work into global priorities research and impact evaluations, and enhancing future work.

• Desirable: formal 'claim identification and assessment'

Remember: The Unjournal doesn’t “publish” and doesn’t “accept or reject.” So don’t give an Accept, Revise and Resubmit, or Reject-type recommendation. We ask for quantitative metrics, written feedback, and expert discussion of the validity of the paper's main claims, methods, and assumptions.

Writing referee reports: resources and benchmarks

Economics How to Write an Effective Referee Report and Improve the Scientific Review Process (Berk et al, 2017)

Semi-relevant: Econometric Society: Guidelines for referees

Report: Improving Peer Review in Economics: Stocktaking and Proposal (Charness et al 2022)

Open Science

PLOS (Conventional but open access; simple and brief)

Peer Community In... Questionnaire (Open-science-aligned; perhaps less detail-oriented than we are aiming for)

Open Reviewers Reviewer Guide (Journal-independent “pre-review”; detailed; targets ECRs)

General

The Wiley Online Library (Conventional; general)