Abstract
OBJECTIVE. In this article, we provide a step-by-step guide to reviewing a manuscript that we hope will improve the quality of reviews for the AJR. CONCLUSION. We have provided a detailed series of guidelines for providing excellent reviews of manuscripts. The template we have provided can be used to serve as a checklist for important questions to ask about manuscripts during the review process. Finally, the principles presented here also can be used as a guide for authors by providing a list of important features to include during manuscript preparation and thereby prospectively address questions that good reviewers are likely to ask.
The process of properly reviewing a manuscript is not intuitive but instead requires training and experience, which are not easily acquired. Journal editors depend on high-quality reviews and are often faced with reviews that do not quite achieve that desired level. A question the authors of this article asked is, “How is this experience gained?” The answer seemed to be, “By trial and error,” which is not the easiest or most systematic method.
In light of these facts, we decided to create a primer on reviewing manuscripts and to disseminate it to our growing team of reviewers by publishing it in the AJR and sending it by e-mail to AJR reviewers. We have also included a template (Appendix 1) that reviewers can use while reviewing a manuscript. The template provides a basic format from which reviewers can systematically proceed through a manuscript and answer important questions. In fact, we encourage reviewers to type their review on the template itself and send their review as an attachment when they submit their review electronically. In addition, the template also serves as a good model for composing a manuscript. In other words, by following this template, authors should be able to compose a well-written manuscript that prospectively addresses the questions good reviewers are likely to ask.
Although we provide this primer at risk of insulting our very-well-qualified reviewers, we designed it to be informative for reviewers at any point in their reviewing career. We hope the primer will serve as a good introduction to the review process for new reviewers and also will reinforce subtleties of the review process for experienced reviewers. In doing so, we hope to bring all reviews up to a high standard that is helpful to editors and instructive for authors.
THE ROLE OF THE REVIEWER
The role of the reviewer is a very important one for any journal. The journal places its confidence in reviewers as the arbiters of quality in submitted manuscripts. Essentially, the reviewer serves two major functions. The first function is to judge whether the manuscript merits publication (usually after revisions) by providing a global rating—that is, “Accept,” “Accept Pending Revisions,” “Reconsider After Major Revisions,” or “Reject.” The second role is to provide constructive criticisms for the authors, regardless of whether the manuscript is deemed acceptable for eventual publication. Many reviewers capably fulfill the first task but could perform more ably in the second capacity—that is, to also serve as an advisor. As one author stated it, the task of the reviewer is to see what the authors have not seen: “The reviewer can be fully as helpful as an involved laboratory colleague or a visiting professor” [1]. The purpose of this primer is to provide suggestions for ways in which reviewers can excel in both roles.
COMMON SENSE RULES FOR REVIEWERS
A few rules exist for reviewers that, although based on common sense, deserve to be stated explicitly. The overriding theme is that reviewers should treat the manuscripts they review as they would like their own to be treated [2]. For instance, because most reviewers would like their manuscripts to be treated with respect and criticisms to be levied in a polite manner, so should they handle others' manuscripts. The reviewer should avoid statements that are demeaning or insulting and should avoid sarcasm. It is also appropriate for the reviewer to direct all statements about the manuscript (e.g., “This manuscript suffers from alack of attention to detail”) rather than about the authors (e.g., “The authors should have paid more attention to detail”).
REASONS REVIEWERS DECIDE TO ACCEPT OR REJECT MANUSCRIPTS
As one might expect, many different reasons exist why reviewers accept or reject manuscripts. It is worthwhile to briefly review these reasons because it is instructive with regard to how reviewers approach manuscripts and the common issues with which they must deal. Furthermore, this information is helpful to authors, especially those who are relatively new to the field of manuscript preparation.
One recent review of the reasons why reviewers accepted manuscripts for publication examined reviewers' comments on 151 research manuscripts submitted to the 1997 and 1998 Research in Medical Education Conference Proceedings. The three reasons cited most often by reviewers for acceptance of a manuscript (which accounted for approximately 50% of positive comments) were, first, the manuscript was considered timely and relevant to a current problem; second, the manuscript was considered well written, logical, and easy to comprehend; and third, the study was well designed and had appropriate methodology [3]. These points are important ones that prospective authors (and not solely reviewers) should keep in mind. The same study indicated that the six most commonly cited reasons for rejection of a manuscript (which accounted for 40% of negative comments) were, first, incomplete or insufficiently described statistics; second, overinterpretation of the results; third, a suboptimal or insufficiently described means of measuring data; fourth, a sample population that was too small or was biased; fifth, text difficult to follow; and sixth, an insufficient problem statement [3]. As the author of the study noted, many of these flaws can usually be adequately addressed by the authors (thereby potentially allowing the manuscript to be salvaged).
Some manuscripts exhibit only one or two of these flaws, while others exhibit many. One of the issues the reviewer must address is whether the sum total of these deficits, if present in a manuscript, allows the manuscript to still be considered a viable candidate for publication or whether the cumulative effect is to render the manuscript unsuitable for publication or require substantial revision before publication can be considered. One factor that is hardest to address is difficulty in following the logical flow of the manuscript. Poor writing cannot be fixed with suggestions by the reviewer; instead, the manuscript often needs to be rewritten. When faced with a manuscript in which logical flow is difficult to follow, an exasperated reviewer may throw up his or her hands and simply recommend rejection. On the other hand, the same reviewer might well have offered an opportunity for revision if the writing had simply been clearer.
A SYSTEMATIC APPROACH TO MANUSCRIPT REVIEW
In this section, we provide a systematic method for manuscript review that reflects the review template, which is included at the end of this article in Appendix 1, that is an amalgam of our opinions about the most important questions to ask during a manuscript review. Interestingly, we subsequently searched the medical literature and found previously published suggested methods for reviewing that were quite similar to ours [2, 4].
Before Reviewing the Manuscript
An initial question that the reviewer should answer is, To what manuscript category used by the journal does this manuscript conform? It is incumbent on the journal to clearly provide this information to the reviewer but, when the authors do not clearly specify which category is appropriate, the issue can be difficult to settle.
The issue of potential reviewer bias is also one with which the reviewer must deal [5]. Bias can be either positive (i.e., unfairly favoring the manuscript for publication) or negative (i.e., unfairly favoring rejection) [6]. The problem becomes more complex in the absence of double-blinded reviews. Reviewers who recognize, at the time of rendering an “Accept for Review” or “Decline to Review” decision that they are strongly biased in either direction should decline to review out of fairness to the authors.
Another issue with which reviewers must deal is whether they have sufficient scientific background to perform a substantive review of the manuscript. A good review requires an awareness of the medical literature and a mastery of the underlying science [1]. If the reviewer believes that the topic of the manuscript is outside his or her area of expertise, then the prospective reviewer should decline to review the manuscript.
Approaches to Reading the Manuscript
Clearly, the first step to reviewing a manuscript is reading it. Hidden in that simple statement is the fact that various approaches exist for performing the initial reading, and there is no one clear-cut best method. Instead, individual reviewers will find a style that suits them best. Some individuals prefer a quick and superficial initial reading of the entire manuscript from which the reviewer can determine the type of manuscript (e.g., Original Research, Case Report, and so on) and the type of study (e.g., prospective cross-sectional study, retrospective case study, and so on) [7]. Some initial questions one might ask during the short overview are, What were the authors intending to study? Does this manuscript address a topic that will be of interest to readers? and Does this study attempt to provide answers to important, previously unanswered questions? Alternatively, the reviewer may take the approach of reading through the manuscript in a detailed manner and asking important questions as one goes along. Whichever method is chosen, many reviewers opt to allow time to pass between detailed reading of the manuscript and writing the review, to allow maturation of initial impressions.
The Abstract
The abstract is the portion of the manuscript where the authors provide a summary that presents the manuscript's most important features. Full abstracts accompany Original Research papers; abbreviated abstracts with only an Objective and Conclusion are used with all other type manuscripts, except Case Reports, Radiologic–Pathologic Conferences, and On the AJR Viewbox, all three of which have no abstract.
This portion of the manuscript is the one that readers most often read if the manuscript is published because subsequent investigators often initially (or only) read it when preparing their manuscripts. Therefore, the abstract should be able to stand alone from the manuscript and be understood without reading the manuscript. In the Abstract, the authors should explain the major objective of the study in an Objective section, explain how the study was done in a Methods section, describe the findings in a Results section, and report whether the major goal was met in a Conclusion. In general, the reviewer should ask, If I could not read the entire manuscript, would the abstract adequately summarize it? Some common ways in which authors fail to do this are as follows.
Providing an abstract that does not adequately represent the manuscript—The reviewer should assess whether there are major discrepancies between the abstract and the remainder of the manuscript (e.g., differences between the methods as outlined in each) and differences in factual statements between the two sections (e.g., differences in numbers of patients).
Providing an objective that is unnecessarily vague—For instance, if the authors had a hypothesis, such as “We hypothesized that MDCT would be more sensitive for the detection of renal calculi than single-detector CT,” it is appropriate to state it here rather than replace it with vague phrases, such as “The objective of our study was to assess the reliability of MDCT for evaluation of renal calculi.”
The Introduction
It is easy for authors to develop a form of tunnel vision and write the manuscript as if the readers were involved in the study and understand all the reasons for performing the study, the assumptions underlying the methodology, and the nuances of the performance of the study. The Introduction of a well-written manuscript is free of this bias and clearly explains why the authors went to all the trouble of performing the study and writing a manuscript. The purposes of the Introduction are, first, to provide the rationale for the study and, second, to explain the study's goals. The Introduction should include a problem statement that conveys the important issues and provides the context for the study [8]. The authors need to provide a rationale to address the two most important questions on the reviewers' mind: Does this manuscript cover an important topic? and Has the research question previously been answered (or the topic of the manuscript previously been well covered)? The answers to these questions may allow the reviewer to decide whether the manuscript is likely to provide a true contribution to the medical literature.
The authors can provide a rationale in the Introduction by showing both that an important problem exists and that previous investigators have failed to adequately address the problem. Both tasks usually require a succinct review of the pertinent literature. Sometimes authors exceed this mandate by attempting to provide a lengthy and detailed review of the medical literature, which is inappropriate for the Introduction. The reviewer should then suggest which portions of the Introduction should be moved to the Discussion section and which portions can be safely deleted without detracting from the manuscript.
The Methods Section
The Methods section is the portion of the manuscript in which the authors outline how they performed their study. In many cases, the Methods section is the most important portion of the manuscript because poor methodology can only lead to results that are suspect, thereby seriously impairing the credibility of the manuscript. On the other hand, if the methods are scientifically sound, even uninteresting results can have merit.
In a sense, the Methods section represents a blueprint by which another investigator could reproduce the study, quite similar to the manner in which a recipe outlines the steps by which a cook can prepare a culinary dish. From a practical standpoint, if another investigator tries to reproduce the study results and fails, the failure could potentially be due to lack of clarity in the Methods section. This factor should be carefully considered by reviewers and commented on in the review. In other words, if the reader could not use the Methods section as a guide to replicate the study, then the Methods section is lacking.
In the Methods section of most manuscripts, the authors should provide a rationale for specific methodologic choices. For instance, if there are alternative techniques that could have been used but were not performed, the authors should justify the choice of the technique they did use. The Methods section is also the appropriate site to explain various other study design choices, such as entry criteria for their study population, specific imaging techniques, and methods of data analysis.
One potential flaw in a scientific manuscript in which the authors have framed a hypothesis is failure to design methods that can adequately test the hypothesis. For instance, if the authors hypothesize that MDCT is more sensitive for the detection of renal calculi than single-detector CT, then they should design a study that uses comparable parameters on both types of scanners, assess the same size of calculi in all patients, and hopefully study the same patients in close temporal proximity on both systems. Failing to follow these guidelines would result in biases that skew the results and either fail to show a difference that really exists or falsely show a difference when none exist.
The Results Section
In the Results section, the reviewer should examine whether the authors systematically and clearly announce the study findings. If the results are unclear, the reviewer must decide whether the analysis of the data was poorly executed or whether the Results section is poorly organized. The latter need not be a fatal flaw, whereas the former usually indicates that the manuscript is unacceptable for publication [9]. Therefore, the organization of the Results section is an important consideration for authors and reviewer alike. If the authors outline a sequence of steps in the Methods section, presenting the results of each step separately will help the reader and reviewer place the findings in perspective.
The Discussion Section
The Discussion section is the part of the manuscript in which the authors should state whether their hypotheses were verified or proven untrue or, if no hypotheses were given, whether their research questions were answered. The authors should also comment on their results in light of previous studies and explain what differences (if any) exist between their findings and those reported by others and attempt to provide an explanation for the discrepancies.
The Discussion section should be long enough to discuss the findings against the background of previous work and explain discrepancies with previously published reports. However, it should not be lengthy to the point of appearing rambling or unfocused, which can substantially detract from the merits of an otherwise good manuscript. Many authors tend to reiterate the results in the Discussion section, which is an unnecessary step that distracts the reader from the more important points of the discussion. Another problem to which some authors succumb is to use the Discussion section to review the entire medical literature surrounding a problem rather than simply reviewing the portion that is relevant to their study. Finally, on occasion, authors become lost in the myriad details of discussing their findings without actually stating basic information, such as whether their findings support their hypothesis or whether their research question was answered. A good reviewer will note the authors' performance on all these points.
In a good manuscript, the authors will attempt to explain unexpected findings rather than ignore them. This process is especially important for findings that are not supportive of the authors' claims or that do not serve as evidence in favor of their hypothesis. To fail to do this is to risk unjustifiably emphasizing only some of the results and reaching inappropriate conclusions. The reviewer can provide a valuable service to the journal by commenting on these possible problems.
One important feature on which reviewers should also concentrate is whether the authors have noted limitations to their study. It is a rare study that does not make fundamental assumptions that may be erroneous or impose limitations that alter the manner in which data are collected and analyzed. This factor can be something as simple as the choice of patient entry criteria or, alternatively, as complicated as the use of an analysis program. Therefore, the lack of a limitations statement suggests that the authors did not prospectively take these factors into account when they designed the study or did not retrospectively assess these features when they reviewed their data.
Reviewers are also requested to assess whether the authors' conclusions are justified by their results. In other words, the reviewer should ask the question, Based on the findings presented in this manuscript, are the authors' claims reasonable? For instance, if the authors have conducted a study that showed that MDCT is more sensitive than routine CT for the detection of small renal calculi, then a claim that MDCT is the preferred technique for evaluation of the entire urinary system is overly broad and cannot reasonably be stated on the basis of this study.
The Figures and Graphs
The figures and graphs should illustrate the important features of the methods and results. The reviewer can help in the review process by deciding whether the figures and graphs are of high quality, appropriately serve their intended purpose, and have figure legends that adequately explain their meaning. The figure legends should allow the reader to understand the figure or graph without having to refer back to the text of the manuscript. Common mistakes made by inexperienced authors are failing to include figures that best depict their findings, writing unclear figure legends, and making poor use of arrows. For instance, it is easy for the authors, who are familiar with the images used in the manuscript, to erroneously believe that all readers will readily see the findings in figures without the need for arrows. The reviewer can be helpful by pointing out the need, if it exists, to improve the figures and graphs and suggesting the means to do so.
The Tables
The purpose of tables is to summarize the data, make the data more easily understandable, and point out important comparisons. The reviewer can assist by commenting whether the number of tables is appropriate and whether the tables adequately summarize the data. Because tables take up valuable journal space, it is important that journals publish tables in a judicious manner, and the reviewers can assist the editor in deciding whether duplication of data is found in the text and in the tables. Authors should use one or the other, not both. Description of the data in the text, if possible, is preferable to the use of a space-consuming table.
The References
The quality of the references often reflects the quality of the manuscript as a whole. Poorly written manuscripts frequently have a References section filled with mistakes indicating lack of citation accuracy, incorrectness of abbreviations and punctuation, and failure to adopt the journal's citation format.
Reviewers do not generally have the time or inclination to review every citation for correctness. However, as a first step reviewers can perform a spot check to determine whether references are cited correctly [7]. The reviewer can rapidly scan the reference list to determine whether important articles were not included and whether appropriate format was followed.
Another important characteristic on which reviewers may comment is whether the authors have misinterpreted articles to buttress their own arguments or to support their results. This problem can be difficult to detect and, in general, the reviewer must depend on his or her knowledge of the medical literature to detect it. In an age when published articles are often relatively accessible via electronic sources, a quick reading of the article in question can answer any questions the reviewer may have.
Summary Opinion
After assessing the various components of the manuscript, the reviewer can perform a useful service by providing a summary statement. In this statement, the reviewer should determine if the manuscript is a substantial addition to the medical literature or if it simply substantiates previously reported studies. The reviewer should also decide whether the manuscript has overall value given its flaws, if any. However, the reviewer should not provide the global rating for the manuscript in this summary. AJR reviewers should instead simply choose the rating in the separate portion of the review form provided to them.
BEYOND THE REVIEW: TIPS FOR PROVIDING THE EDITOR WITH THE MOST INFORMATIVE REVIEW
Deciding on a Global Rating
After writing a review of the strengths and weaknesses of the manuscript, the reviewer is asked to provide a global rating (i.e., a recommendation for the manuscript). It is important that the reviewer be familiar with the possible global ratings because they can differ from one journal to another. AJR reviewers are provided with four possible global ratings: Accept, Accept Pending Revisions, Reconsider After Major Revisions, and Reject.
The global rating of Accept is clear-cut and unambiguous; this rating implies that the reviewer does not see any need for revision of the manuscript and that it is suitable for publication “as is.” In fact, because most reviewers (with good reason) suggest changes to any manuscript, the Accept rating is granted to few manuscripts on initial review. Given that it is a rare manuscript that cannot be improved in some way, sometimes the Accept rating is an indication that the reviewer has not looked at the manuscript with an eye toward improvement. When revisions are suggested, the decision category always should be Accept Pending Revisions rather than Accept.
The Accept Pending Revisions rating indicates that the reviewer finds some ways in which the manuscript should be changed before final acceptance. The suggested changes may include items such as a request for clarification of the methods (e.g., details regarding study design, entry criteria, whether film readers were blinded to information that might produce a biased reading, and so on). However, it is implied in this rating that the authors can reasonably make these changes and that doing so will more or less result in publication of the revised version of the original manuscript. For instance, it is not appropriate for the reviewer to provide the rating of Accept Pending Revisions if the reviewer is suggesting one or more major changes in study design. As an example, occasionally a reviewer will recommend that the manuscript be accepted pending revisions but request major changes in the methodology. Even if this suggestion is warranted, adopting it would necessitate performing the study over in a manner different from the first version of the study. In essence, this rating is a Reject operating under the guise of an Accept Pending Revisions. On a related note, the judgment as to whether the appropriate rating should be that of Accept Pending Revisions, rather than that of Reconsider After Major Revisions, does not rest on how many changes are suggested, but in the degree to which the sum of the changes alters the manuscript.
A rating of Reconsider After Major Revisions indicates that the reviewer believes that considerable changes are needed but that a reasonable possibility exists for the manuscript to proceed to publication. Examples of indications for providing this rating include a belief that, first, the reported data need to be analyzed in a different manner; second, additional data are needed; third, the authors have failed to appropriately take certain study factors into account; or fourth, the authors have not appropriately discussed their results against the background of previous studies. This rating is probably underused by many reviewers who instead recommend Accept Pending Revisions for a manuscript that needs substantial rewriting or reorganization before acceptance. In such instances, reviewers often request substantive changes but for one reason or another are reluctant to place a manuscript that has potential for publication in a category other than Accept Pending Revisions. It may be that reviewers believe that providing a rating of Reconsider After Major Revisions means that the manuscript is unlikely to be accepted for publication, but that is not, in fact, the case. Most manuscripts that receive a Reconsider After Major Revisions recommendation are ultimately published, with many of them published in the AJR [10].
The Reject rating is provided when the reviewer is of the opinion that no amount of revision will make the manuscript suitable for the journal to which it was submitted. It is worth emphasizing that, in some cases, the rating is based not on the opinion that the manuscript is poorly written or an inadequate study. Instead, sometimes a reviewer recommends rejection on the belief that the manuscript was submitted to the inappropriate journal.
After receiving manuscript reviews, the journal editor must decide the outcome of a manuscript. It is important that the reviewer provide a clear explanation as to whether he or she deems the manuscript to be worth publishing. Although that statement may seem obvious, in a substantial number of manuscripts the reviewer's overall assessment is less than definitive. For example, sometimes the written review leads the reviewer toward one decision but the numeric rating provided by the reviewer indicates a different decision. It is not rare for a reviewer to offer comments that are strongly negative but to then recommend that the manuscript be accepted pending revisions. Such a review requires that the editor make a difficult choice: Either accept a manuscript that the reviewer appears to say is not worthy of publication or fail to accept a manuscript that the reviewer has technically asked to be accepted.
The Informative Review
The type of review that is most helpful to the editor is one that shows that the reviewer performed a close reading of the manuscript, thought carefully about the most important sections of the manuscript, provided constructive criticisms for the authors, and assigned a rating that is commensurate with the remainder of the review. An abbreviated version of such a review for the purposes of illustration (and for contrast with other versions of the same review) follows.This manuscript describes a new method for use of 16-MDCT for increasing the sensitivity of diagnosis of renal calculi. The authors appropriately noted some of the limitations of conventional CT for this purpose in the Introduction. However, they should also note the study by Stanley and Provenzale (J Irreproducible Results, 2003) in which the authors performed a similar study but with different results. Also, the study does not have a hypothesis but has a “look and see” quality. The study design as outlined in the Methods section suffers insufficient detail regarding how patients were chosen, lack of description whether film readers were blinded to clinical symptoms, and no detail whether the film readings were performed by consensus review or by independent readings. The Results section is unclear. The authors state that all calculi with attenuation coefficients greater than 300 H were smaller than 2 mm, but Table 1 indicates that only calculi larger than 2 mm had attenuation coefficients greater than 300 H. In the Discussion, the authors fail to account for why 20% of calculi seen on MDCT are not seen on single-detector CT. Finally, the references do not take into account many recently published articles on this topic. Confidential Note to the Editor: This manuscript should be rejected.
The Noninformative Review
Although the majority of reviews provided by AJR reviewers are of high quality and helpful in deciding the outcome of the manuscript, on occasion reviews are less than helpful. This circumstance is unfortunate because the reviewer may have spent considerable time reviewing the manuscript but with little yield for the purposes of the journal. We provide some examples of the types of reviews that would benefit from closer attention to our proposed format.
The snapshot verdict—This type of review basically solely indicates that the reviewer has read the manuscript and whether he or she liked it or did not. To use our previous example, this type of review simply states something like the following:I read the manuscript on renal calculi that you sent me. I found no problems with it. I think it is the first time that this work has been done. This manuscript should be published. Recommendation: Accept.
As ludicrous as these comments sound because of their brevity and superficial nature, reviews of this type are not rare. This review is not helpful for a number of reasons. First, it is generic, rather than specific, and noninformative. It could have been sent by any reviewer about any manuscript. For instance, one cannot ascertain that the reviewer actually read this manuscript. Second, the reviewer failed to provide a critical analysis. The editor is not left with the belief that the reviewer fully assessed the manuscript and may wish to send the manuscript to an additional reviewer for an in-depth evaluation. Such a process wastes a valuable resource: the journal's reviewers' time. Finally, the reviewer is not providing a fundamental component of the review—that is, a means for the authors to improve their manuscript.
The mixed-signals review—In this type of review, to which we have alluded earlier, the reviewer provides mixed signals by virtue of a discordance between the written review, the numeric ratings, and the “Recommendation” selected from the drop-down list. In other words, the narrative summary leads to one conclusion, but the reviewer provides a Recommendation counter to the expected one. Most commonly, the reviewer is highly critical in the narrative summary but then provides a Recommendation of Accept Pending Revisions, indicating that the manuscript should proceed along the path to publication essentially unimpeded. Again, an example follows:This manuscript has a number of major shortcomings: The authors have not adequately reviewed the medical literature on renal calculi, they provide insufficient detail regarding how patients were chosen for the study, and the Results section has many inconsistencies. Recommendation: Accept Pending Revisions.
In this type of review, it is as if the reviewer is reluctant to actually reach the natural conclusion of his or her argument—that is, to reject the manuscript. As one author noted [11]:[T]his reluctance is understandable from the perspective of the reviewer, who is likely to have had his or her own share of negative publication decisions and is quite familiar with the angst such decision letters cause.
The hidden-agenda review—On occasion, the reviewer will provide opinions in the section of the review that is labeled “Confidential Note to the Editor.” When the opinions in the confidential note substantially differ from those expressed in the portion of the review available to the authors, problems arise and the editor is sometimes left in an awkward position with regard to determining the outcome of the manuscript.
For instance, the reviewer may offer relatively benign comments in the portion of the review available to authors but then provide negative comments in the Confidential Note to the Editor section of the review form and recommend rejection. In essence, this type of review is a variant of the mixed-signals review but with the disparity being between two types of written comments (rather than solely between the written comments and the final Recommendation). The editor is faced with the difficult task of having to either include confidential information, with the permission of the reviewer, in a rejection letter to the authors (to justify the rejection) or disregard the confidential information. Using our previous example, we provide a sample of this type of review:This study is a good comparison between conventional CT and 16-MDCT for evaluation of renal calculi. The manuscript could be improved by better documentation of the previous medical literature, a clearer explanation of how readers scored the images and of the selection criteria, clarification of some inconsistencies between the stated results and Table 1, and a better explanation of how the CT scan parameters affect the sensitivity of MDCT. Recommendation: Accept Pending Revisions. Confidential Note to the Editor: There are some serious flaws here. The authors show a lack of understanding of the factors that come into play in renal calculus detection by CT. The Methods section needs a lot of work.
As this example shows, this type of review puts the editor in an awkward position. The review would be more helpful if the statements in the confidential note accurately reflected those that have been provided for the authors' viewing.
SUMMARY
We hope that new reviewers and experienced reviewers alike benefit from this brief primer and make use of the accompanying review template. Although the primary beneficiaries will be new reviewers, we hope that even experienced reviewers will gain insights into what journal editors need from a review. We also hope that this article will be used by senior reviewers to advise young academic faculty on the review process. To expedite this process, we not only are publishing this primer in the AJR but also will send this document by e-mail to all of our current reviewers to ensure better dissemination of its content.
APPENDIX 1: A Systematic Guide to Reviewing a Manuscript
Before Writing the Review
-
To which manuscript category does this manuscript best conform?
-
Are there any potential biases in reviewing this manuscript?
-
Does the manuscript address an important problem?
-
Has the manuscript been previously published?
The Abstract
-
Does the Abstract appropriately summarize the manuscript?
-
Are there discrepancies between the Abstract and the remainder of the manuscript?
-
Can the Abstract be understood without reading the manuscript?
The Introduction
-
Is the Introduction concise?
-
Is the purpose of the study clearly defined?
-
Do the authors provide a rationale for performing the study based on a review of the medical literature? If so, is it of the appropriate length?
-
Do the authors define terms used in the remainder of the manuscript?
-
If this manuscript is Original Research, is there a well-defined hypothesis?
The Methods Section
-
Could another investigator reproduce the study using the methods as outlined or are the methods unclear?
-
Do the authors justify any choices available to them in their study design (e.g., choices of imaging techniques, analytic tools, or statistical methods)?
-
If the authors have stated a hypothesis, have they designed methods that could reasonably allow their hypothesis to be tested?
The Results Section
-
Are the results clearly explained?
-
Does the order of presentation of the results parallel the order of presentation of the methods?
-
Are the results reasonable and expected, or are they unexpected?
-
Are there results that are introduced that are not preceded by an appropriate discussion in the Methods section?
The Discussion Section
-
Is the discussion concise? If not, how should it be shortened?
-
If a hypothesis was proposed, do the authors state whether it was verified or falsified? Alternatively, if no hypothesis was proposed, do the authors state whether their research question was answered?
-
Are the authors' conclusions justified by the results found in the study?
-
If there are unexpected results, do the authors adequately account for them?
-
Do the authors note limitations of the study? Are there additional limitations that should be noted?
Figures and Graphs
-
Are the figures and graphs appropriate and are they appropriately labeled? Would a different figure better illustrate the findings?
-
Do the figures and graphs adequately show the important results?
-
Do arrows need to be added to depict important or subtle findings?
-
Do the figure legends provide a clear explanation that allows the figures and graphs to be understood without referring to the remainder of the manuscript?
Tables
-
If there are tables, do they appropriately describe the results? Should one or more tables be added?
The References Section
-
Does the reference list follow the format for the journal?
-
Does the reference list contain errors?
-
Have the authors appropriately represented the salient points in the articles in the reference list? Alternatively, have the authors misquoted the references?
-
Are there important references that are not mentioned that should be noted?
-
Are there more references than are necessary?
Summary Opinion
The reviewer should provide a short paragraph that summarizes the strengths and weaknesses of the manuscript. The actual Recommendation (e.g., recommend to Accept, Accept Pending Revisions, Reconsider After Major Revisions, or Reject) should not be stated in this paragraph, which is sent to the authors, but should be indicated separately in the drop-down list. It may also be stated in the separate box called “Confidential Note to the Editor.” However, the overall tenor of this paragraph should support the reviewer's recommendation.
Footnotes
-
↵* © American Roentgen Ray Society. Reprinted with permission from the American Journal of Roentgenology (2005;185:1–7).