Our latest blog post is written by me with the wonderful Sue Fletcher-Watson, a colleague whose intellectual excellence is only exceeded by her whit and charm.
Peer review is a lynch-pin of the scientific process and bookends every scientific project. But despite the crucial importance of the peer review process in determining what research gets funded and published, in our experience PhD students and early career researchers are rarely if ever offered training on how to conduct a good review. Academics frequently find themselves complaining about the unreasonable content and tone of the anonymous reviews they receive, which we attribute partly to a lack of explicit guidance on the review process. In this post we offer some pointers on writing a good review of a journal article. We hope to help fledgling academics hone their craft, and also provide some insight into the mechanics of peer review for non-academic readers.
What’s my review for?
Before we launch into our list of things to avoid in the review process, let’s just agree what a review of an academic journal article is meant to do. You have one simple decision to make: does this paper add something to the sum of scientific knowledge? Remember of course that reporting a significant effect ≠ adding new knowledge, and similarly, a non-significant result can be highly informative. Don’t get sucked into too much detail – you are not a copy editor, proof-reader, or English-language critic. Beyond that, you will also want to consider whether the manuscript, in its current form, does the best job of presenting that new piece of knowledge. There’s a few specific ways (not) to go about this, so it looks like it might be time for a list…
1. Remember, this is not YOUR paper
First rule of writing a good peer review: remember that this is not your paper. Neither is this a hypothetical study that you wished the authors had conducted. Realising this will have a massive impact on your view of another’s manuscript. The job is not to sculpt it into a paper you could have written. Your job as a reviewer is two-fold: i) make a decision as to the value of this piece of work for your field; and ii) help the authors to present the clearest possible account of their science.
Misunderstanding the role of the reviewer is perhaps at the heart of many peer review horror stories. Duncan does a lot of studies on cognitive training. Primarily he’s interested in the neural mechanisms of change, and tries to be very clear about that. But reviewers almost always ask “where are your far transfer measures?” because they want to assess the potential therapeutic benefit of the training. This is incredibly infuriating. The studies are not designed or powered for looking at this, but instead at something else of equal but different value.
Remember – you can’t ask them to report an imaginary study you wished they had conducted.
2. Changing the framing, but not the predictions
In this current climate of concern over p-hacking and other nefarious, non-scientific procedures, a question we have to ask ourselves as reviewers is: are there some things I can’t ask them to change? We think the answer is yes – but it may be less than you think. For starters, you can ask authors to re-consider the framing of the study to make it more accurate. Let’s imagine they set out to investigate classroom practice, but used interviews not observations, and so ended up recording teacher attitudes instead. Their framing can end up a bit out of kilter with the methods and findings. As a reviewer, with a bit of distance from the work, you can be very helpful in highlighting this.
If you think there are findings which could be elucidated – for example by including a new covariate, or by running a test again with a specific sub-group excluded – you should feel free to ask. At the same time, you need to respect that the authors might respond by saying that they think these analyses are not justified. We all should avoid data-mining for significant results and reviewers should be aware of this risk.
What almost certainly shouldn’t be changed are any predictions being made at the outset. If these represent the authors’ honest, well-founded expectations then they need to be left alone.
However, there may be an exception to this rule… Imagine a paper (and we have both seen these) where the literature reviewed is relatively balanced, or sparse, such that it is impossible to make a concrete prediction about the expected pattern in the new data. And yet these authors have magically extracted hypotheses about the size and direction of effects which match up with their results. In this case, it may be legitimate to ask authors to re-inspect their lit review so that it provides a robust case to support their predictions. Another option is to say that, given the equivocal nature of the field, the study would be better set-up with exploratory research questions. This is a delicate business, and if in doubt, it might be a good place to write something specific to the editor explaining your quandary (more on this in number 5).
3. Ensuring all the information is there for future readers
In the end the quality of a paper is not determined by the editor or the reviewers… but by those who read and cite it. As a reviewer imagine that you are a naïve reader and ask whether you have all the information you need to make an informed judgement. If you don’t, then request changes. This information could take many forms. In the Method Section, ask yourself whether someone could reasonably replicate the study on the basis of the information provided. In the Results ask whether there are potential confounds or complicating factors that readers are not told about. These kinds of changes are vital.
We also think it is totally legitimate to request that authors include possible alternative interpretations. The whole framing of a paper can sometimes reflect just one of multiple possible interpretations, which could somewhat mislead readers. As a reviewer be wise to this and cut through the spin. The bottom-line: readers should be presented with all information necessary for making up their own minds.
4. Digging and showing off
There is nothing wrong with a short review. Sometimes papers are good. As an editor, Duncan sometimes feels like reviewers are really clutching at straws, desperate to identify things to comment on. Remember that as a reviewer you are not trying to impress either the authors or the editor. Don’t dig for dirt in order to pad the review or show how brainy you are.
Another pet hate is when reviewers introduce new criticisms in subsequent rounds of review. Certainly if the authors have introduced new analyses or data since the original submission, then yes, this deserves a fresh critique. But please please please don’t wait until they have addressed your initial concerns… and then introduce a fresh set on the same material. When reviewers start doing this it smacks of a desperate attempt to block a paper, thinly veiled by apparently legitimate concerns. Editors shouldn’t stand for that kind of nonsense, so don’t make them pull you up on it.
5. Honesty about your expertise
You don’t know it all, and there is no point pretending that you do. You have been asked to review a paper because you have relevant expertise, but it isn’t necessarily the case that you are an expert in all aspects of the paper. Make that clear to the authors or the editor (the confidential editor comments box is quite useful for this).
It is increasingly the case that our science is interdisciplinary – we have found this is especially the case where we are developing new neuroimaging methods and applying them to novel populations (e.g. typically and atypically developing children). The papers are usually reviewed by either methods specialists or developmental psychologists, and the reviews can be radically different. This likely reflects the different expertise of the reviewers, and it helps both authors and editor where this is made explicit.
Is it ok to ask authors to cite your work? Controversial. Duncan never has, but Sue (shameless self-publicist) has done. We both agree that it is important to point out areas of the literature that are relevant but have not been covered by the paper – and this might include your own work. After all, there’s a reason why you’ve been selected as a relevant reviewer for this paper.
Now we know what not to do, what should you put in a review?
Start your review with one or two sentences summarising the main purpose of the paper: “This manuscript reports on a study with [population X] using [method Y] to address whether [this thing] affects [this other thing].” It is also good to highlight one or two key strengths of the paper – interesting topic, clear writing style, novel method, robust analysis etc. The text of your review will be sent, in full and unedited, to the authors. Always remember that someone has slaved over the work being reported, and the article writing itself, and recognise these efforts.
Then follow with your verdict, in a nutshell. You don’t need to say anything specific about whether the paper should / should not be published (and some journals actively don’t want you to be explicit about this) but you should try to draw out the main themes of your comments to help orient the authors to the specific items which follow.
The next section of your review should be split into two lists – major and minor comments. Major comments are often cross-cutting, e.g. if you don’t think the conclusions are legitimate based on the results presented. Also in the major comments include anything requiring substantial work on the part of the authors, like a return to the original data. You might also want to highlight pervasive issues with the writing here – such as poor grammar – but don’t get sucked into noting each individual example.
Minor comments should require less effort on the part of the authors, such as some re-phrasing of key sentences, or addition of extra detail (e.g. “please report confidence intervals as well as p-values”). In each case it is helpful to attach your comments to a specific page and paragraph, and sometimes a numbered line reference too.
At the bottom of the review, you might like to add your signature. Increasing numbers of reviewers are doing this as part of a movement towards more open science practices. But don’t feel obliged – especially if you are relatively junior in your field, it may be difficult to write an honest review without the safety of anonymity.
Ready to review?
So, hopefully any early career researchers reading this might feel a bit more confident about reviewing now. Our key advice is to ensure that your comments are constructive, and framed sensitively. Remember that you and the original authors are both on the same side – trying to get important science out into a public domain where it can have a positive influence on research and practice. Think about what the field needs, and what readers can learn from this paper.
Be kind. Be reasonable. Be a good scientist.