Tips for Reviewing Scientific Manuscripts: Part 1 – When To Be Suspicious

When you get invited to review your first scientific publication for a journal, you're probably excited because that means that somebody recognizes your expertise in a field (by asking you to do unpaid work). However, you might also be a little nervous since peer review is an important process and figuring out whether a manuscript provides good science is not always easy.

In this first article of a series of tips for first time reviewers, we will go through findings that should make you suspicious that there might be something wrong with an article and that you should take an even closer look.

As a young researcher, you will most likely not be asked to review for the most prestigious journals in your field, which means that you are more likely to encounter low-quality science because the editors are often a little more permissive with the articles they send for review. This is actually a good thing because articles that are submitted to lower-tier journals tend to make more obvious mistakes that are easier to spot for a person with limited experience.

While I have a medical background, I will try to keep this article helpful for researchers from different fields and encourage you to comment your own experiences with suspicious behaviors of authors that indicate bad science so that it can be helpful to other young researchers.  

 

Questionable use of p-values

Some people believe that getting a p-value below 0.05 is all that matters in research. As you hopefully know, this is not the case and if you don't, there are plenty of resources to improve your knowledge. The way authors use p-values can give you hints regarding the quality of a manuscript:

• Unclear use of p-values: Sometimes you see authors who put p-values in parenthesis behind a sentence or in a table even though the context does not really tell you, what was tested against what. If the authors are intentionally ambiguous or simply do not care about being explicit, this is not a good sign.
• Not giving exact p-values: Some authors think that writing 'p < 0.05' is sufficient. There is, however, a difference between p = 0.049 (borderline significant) and p = 0.00001 (highly significant). After all, the threshold 0.05 is a rather arbitrary value. If the authors think the exact p-value is not relevant, there is a high chance that there are other problems with their manuscript.
• Meaningless p-values: Some authors like to put p-values behind every single sentence. If your retrospective chart review included 52 men and 54 women but sex plays absolutely no role in your analysis, there is no need to conduct a t-test to see if there was a significant difference between the number of men and women.  

 

Big findings

If you conduct research that results in astonishing, practice-changing findings, you will most likely submit them to one of the top journals in your field and enjoy the resulting glory. If you are a young researcher starting your career, you're most likely not reviewing for one of the top journals. That means if you encounter a manuscript that claims to cure cancer, there is likely something wrong with it. Either it was submitted to a more prestigious journal before and the reviewers found significant flaws, or the authors know that their claims won't have a chance with a more prestigious journal where the peer review is thought to be more thorough (though that’s not always the case). The same applies to manuscripts that describe a methodology which requires a lot of time and effort. If you put a lot of work in your research, you want to publish it at the best journal. If you do a review for a less prestigious journal and you encounter a manuscript which makes you think 'Wow that was a lot of work!', you should ask yourself why the authors did not aim for a top journal.  

 

Hiding the effect size

When you want to show whether a drug helps against blood pressure, it is not enough to just provide the p-value. If you measure the blood pressure after giving a placebo to one group and a drug to the other and the median blood pressure is significantly lower in one group, this means that the difference is statistically significant. However, it does not mean that the difference is clinically relevant!

For example, if you conduct the trial with 5000 participants in each arm, a minimal difference between the median blood pressures becomes statistically significant (because it is unlikely that the difference is due to chance when the samples are so large). However, it is completely unclear if the minimal difference in blood pressure actually improves outcomes, i.e. makes patients life longer and/or better.

If you really want to assess a finding, you will need both — the p-value and the effect size.

Hopefully this will help you to weed out the bad apples when you start reviewing! In the next parts of the series, we will have a look at different article types and what to look for when reviewing each of them. If you found this article helpful, let me know! If you disagree with some of the things I said, let me know as well!