In my current position as Editor-in-Chief of the American Naturalist, I read all kinds of reviews of scientific papers from all kinds of people. I routinely get asked, particularly by graduate students, what makes a good review. Here are a few thoughts on the subject.
The first task of the reviewer is to prove to the author and the editor that you have actually read the manuscript and that your critique is based on that reading. The most common reaction by authors to negative reviews of their papers are that “this reviewer didn’t even read the paper.” The reviewer can accomplish this by following a few simple style characteristics in structuring the review. First, start out the review with a very brief summary of what the paper is about and what the paper is trying to accomplish, based on your reading. This only needs to be 2-3 sentences usually, but summarizes for the author what the reviewer took away as the main thesis of the manuscript. Second, for each main critique point, note where in the manuscript the relevant text is. For example, start a section of the review with something like “On page 6, you write that … “. Finally, quoting small relevant passages from the manuscript also helps focus the author on the specific areas you are discussing.
The most important features that should characterize your writing for the review should thoroughness and constructive tone. Your number one task is to do a thorough job in evaluating the manuscript. Summarize what you see as the major strengths and weaknesses of the paper. Explain your opinions about the main thesis of the paper, and how the authors have framed the work. Is the paper appropriate for the journal to which it has been submitted? If it is an empirical paper describing a set of observational and experimental studies, (1) are signficant hypotheses being tested, (2) do the studies adequately test the stated hypotheses, (3) are the results significant and compelling, (4) does the discussion fairly characterize the results of the study and tie these results to the broader literature on the subject? If it is a theoretical/modeling paper, (1) is the subject of the model compelling, (2) is the modeling approach appropriate to the question, (3) have the authors done the model correctly [i.e., is the math or computer code right], (4) do the results of the model make a significant advance in our insights into the problem, and again (5) does the discussion fairly characterize the results and conclusions of the model and tie these resutls to the broader literature? You should organize your discussion of these points into a structure that expresses what you see as larger problems with the manuscript first in your review, and then work your way to minor points.
In writing your review you should always maintain a constuctive tone to your writing. This does not mean that you should not forcefully express your critique, or that you should hold back from clearly stating problems you find with the manuscript. It means that you should forcefully describe the issues and problems you identify in the manuscript, but along with describing the problems you should also lay out remedies to these problems. For example, if the experimental design of the study is not appropriate for the hypotheses being tested, describe what design should be used. If you find the stucture of the paper confusing, describe how you would change the structure to make the paper flow better. If you think more experiments or more analyses are needed, explain what they are and why they will improve the manuscript.
You also need to stay focused on the manuscript and not stray into other areas. Critiques of the entire field in which the paper is embedded are not useful. And above all, ad hominum attacks of the author are completely inappropriate and the best way to get your review completely ignored by the editorial staff of a journal. Your job is to critique the manuscript before you.
Also, do your review in a timely fashion! Always follow the Golden Rule of Reviewing.
What if you loved the paper and you have no substantive criticisms of the paper? The worst thing you can write is a very terse review that says basically “I loved this paper, and it should be published.” In this case, you must become the advocate for the paper. You need to explain why the paper is a significant work and why it makes a strong contribution to the literature. Imagine if the other review that comes in for the paper is negative. Your review will become the counterpoint, and so you need to provide the rationale and arguments to the editor why this is a significant work that they should publish.