As noted earlier, referees are not always consistent in what they recommend. Different referees have different opinions, and there has been much research on the reliability and validity of peer review systems (e.g. see Godlee and
Please rate the paper as follows:
GOOD AVERAGE POOR
Strength of supporting data/evidence ___ _
Originality of ideas and approach ___ _
Significance of topic ___ _
Completeness of discussion ___ _
Intelligibility to non-specialists ___ _
Accept_ Accept with minor revisions_
Re-submit after major revision_ Reject_
Please add any comments for the editor here:
COMMENTS FOR THE AUTHOR(S)
Please type comments for the authors on a separate page
Figure 4.4.1 A typical evaluation sheet for editors and referees.
Jefferson, 2003; Hojat et al, 2003; Weller, 2001). Weller, for example, reviewed over thirty studies on the topic and concluded that they indicated that there was not a lot of agreement between referees. She also suggested, however, that agreement between referees about whether or not a paper should be rejected was usually higher than it was about whether or not a paper should be accepted.
There are few personal accounts in the literature of how different referees go about the process of refereeing but the ones that have been published are instructive (see, e.g. Benos et al, 2003; Hoppin, 2002; Lee, 1995). Here I describe my own procedures in this respect.
First of all, I download the paper for review and then I read it twice -trying hard not to scribble any comments on it on the first reading. This preliminary reading allows me to think about the overall recommendations I shall be making on the evaluation scale, the opening paragraph of my comments to the author(s), and any comments that I might make privately to the editor.
Next, I read the paper again, making specific notes/comments/queries to myself on the actual text, paragraph by paragraph and even line by line.
Then, I word-process a general set of opening remarks for the author(s), trying to be positive, and indicating what my general recommendations will be. Here, I typically summarise the purpose of the paper (to help the editor remember what is was about) and I might indicate my expertise — or lack of it — with respect to certain parts of the paper. I head these comments 'General remarks'.
Next, I list, in sequence and by appropriate page number, paragraph or line, any specific comments I might have (under the heading 'Specific comments'). Making these comments sometimes helps me to clarify or add to the general remarks written above. Finally, I have a third section 'Minor points' — where I might note an occasional ungrammatical sentence, a reference that is cited in the text but not in the reference list, and a correction to a date of publication etc. I do not usually bother with these minor points if I feel the paper should be rewritten or rejected.
As noted above, I list my specific comments and minor points in page sequence (and not in order of importance). I do this simply to help me and the authors locate the focus of my remarks. The kinds of comments I typically make are:
• suggestions for improving the clarity of the title/abstract;
• queries about the procedures, the data and inferences;
• wondering about the need for additional or more appropriate statistics;
• implying the necessity for additional/updated references; and
• requesting more detail and/or clearer explanations, etc.
Table 4.4.1 lists the more general concerns of referees. Information such as this may be instructive for new referees. Godoy (2006), in an interesting paper, describes how a group of young faculty members in engineering benefited from comparing their reviews with those written by more experienced colleagues.
Was this article helpful?