Introduction

Peer review was first used by the “Philosophical Transactions” of the Royal Society of London, in 1665 [1], and by the mid-20th century had become a key mechanism underlying science publishing. Almost all research journals used it to select papers to print in their always limited space, and make them better papers in the process. There still is debate on whether peer review improves papers in all or only in specific aspects, and which format of peer review (single-blind, double-blind, open) using what specific instructions and reporting forms to be completed by the referee, is the best. There also are claims, not entirely unfounded, that peer reviews (a) sometimes are biased and uninformed; (b) slow down the process of reporting the results of scientific work; (c) often fail to identify duplicate articles, plagiarism, unnecessary duplication of studies, fake data, misuse of data, salami science, and other malfeasance on the part of authors. I will not go into that debate.

Spinal Cord (SC) uses peer review of the single blind format (the reviewers know who the author is, but the author is not told who the reviewers are) and is likely to continue doing so in the foreseeable future. The purpose of this paper is to set forth the role of the peer reviewer in the process that SC follows, and how reviewers ought to fulfill that role so that their review is maximally useful to author, Editor-in-Chief (EIC) and Associate Editor (AE). Perceptive readers will note that the basic principles and suggested approaches apply equally to other journals from which they may receive invitations to review. The current article joins a handful of similar papers in the literature, which have been consulted [2,3,4,5,6,7,8,9,10,11]. Box 1 provides some reasons for you to accept an invitation from SC to review a manuscript.

Collaborative triangle

Once the EIC has decided that the article is within scope, not plagiarized, and of sufficient quality that it seems likely to be published (possibly after one or more rounds of revision), a collaborative triangle is established: author, AE, peer reviewer (See Box 2). (For quite a few papers, the EIC manages the peer review process, and handles all tasks here described as performed by an AE. For the sake of simplicity, the EIC here is considered an AE in such cases).

Peer reviewers have a dual role: they are a colleague of the author who wants him/her to succeed, and therefore, as an author advocate, referees offer constructive critiques on the manuscript that will enhance the science being reported, improve the report itself, and make it more likely that it will be published. But aside from being the author’s collaborators, peer reviewers also have a duty to the AE and to the journal: offering advice, as objective as possible, on whether the paper is publishable, and if it is not printable in its current state, what major and minor changes need to be made to make it so.

Until the author turns copyright over to the journal, the manuscript is her/his property, and AE, EIC and referees should keep confidential all materials received from the author, as well as all relevant communications with one another and with the author.

The process of writing a peer review

This section focuses on the steps you as a referee need to take in the reviewing process; later sections concentrate on the content and wording of a review.

Step 1. Upon receipt of the review invitation by the AE, read the abstract and decide whether to accept; communicate your decision quickly

Before you accept the invitation, consider 4 issues:

  1. 1.

    Do you have the relevant expertise (for at least some of the content or methodology of the paper) needed to judge the value of the paper and provide useful feedback to the AE and author?

  2. 2.

    Do you have a conflict of interest (COI)? A COI can be financial (you or people close to you may gain or lose money if the paper is published), but it can also be social (your close colleagues or academic competitors are the author), moral, or even intellectual (for instance, the manuscript endorses a procedure that you have spent 10 years arguing against). COI in journal peer reviewing is discussed in a few papers only (e.g. [12]); it is not too much different from COI in grant proposal peer reviewing, which is addressed in the policies and procedures of many grant making agencies (e.g. [13]).

  3. 3.

    Do you have the time? SC routinely asks for a review to be submitted within 14 days, and writing a thoughtful review may take from several hours to a day or more, depending on your experience reviewing, the length of the manuscript (including figures, tables and supplementary digital content [SDC]), and the complexity and newness of the methods used in the research.

  4. 4.

    Do you have the interest required to write a good review of this particular paper?

Communicate with the AE if (a) your expertise does not extend to all aspects of the manuscript (e.g. advanced statistics), (b) you are willing or able to review but will need more time than the 14 days allotted, or (c) you have a potential COI which you want the AE to judge.

Let the AE know your decision or question(s) as soon as possible—if you are not available, the AE needs to find a replacement; if your knowledge is limited, an additional reviewer with the expertise you lack needs to be found, and that takes time. The author is waiting for a decision, and SC’s policy is to communicate decisions quickly.

Step 2. Make sure you have all the materials needed for writing your review

The Editorial System SC uses offers various options to obtain the manuscript and the related materials. The easiest presumably is to download the zipped file called “merged”, which will contain a single PDF with the manuscript, including the title page, the text itself, references, tables, appendices (if any), figure captions, and the figures themselves. Other documents can be downloaded too: the author’s letter to the editor (if any); and one or more files with SDC, which may consist of additional graphs and tables, references for documents extracted in a systematic review, methodology details, etc. For re-reviews, you also should receive a copy of the author’s “response to reviewers” (RTR) (rebuttal). Perform a quick scan to make sure that all pieces called for in the manuscript are present. If not, notify the AE or the editorial office. Under no circumstances contact the author directly; all communications are to go through the AE.

Also, have at hand SC’s “Guide to authors”, which describes the scope of the journal and gives specifications for various article types. A copy of an applicable reporting guideline (Box 3 part a) may also be useful. Your own personalized manuscript checklist or one published in the literature (Box 3 parts b and c) similarly might come in handy in making sure that you are not overlooking any important questions.

Step 3. Schedule time to do the review

Most people do their review in two steps: (a) a quick and complete read-through to get the “lay of the land”, form an idea of any major problems, and start thinking about how serious these are in the overall scope of things; (b) a later session consisting of a second and in-depth complete reading, and the writing of a well-organized list of major and minor concerns. This step will cost experienced reviewers minimally 3 h for a “simple” paper; novices might want to double that time. Some referees add a third session, in which they read the comments they have written in their review, for completeness, clarity and the right tone, before they submit the review to SC. If you involve others in performing the review (a student, resident, post-doc, junior colleague, local methodological or statistical expert, etc.), you also need to schedule time to talk with them, and in the case of a student or mentee, much time to go through what they have written, and discuss positive and negative aspects of the manuscript, as well as your own observations. (SC has no objection to you involving a second person in performing a review, as long as s/he adheres to the confidentiality rules you yourself agree to in accepting to perform the review. You are asked to tell the AE that you consulted someone else, and give details, when you submit your review).

Step 4. Perform the review

There probably is much variation between referees in how they go about the actual process of reviewing. Very few people are able to read the manuscript and supplementary materials, and then open a word processor document and write a well-organized two-page review, without ever referring back to those materials. Most will, as suggested above, read the materials at least twice, in hardcopy or on their computer or tablet, and make notes on the hardcopy, in a notebook, or in a draft review document. The actual text of your review can be written directly into the SC Editorial System, or initially in a word processor document. The latter is presumably a better idea; the system may “time out” when you are not actively using it, and lose what you have written.

Some additional reading might serve to enhance the utility of what you deliver. You may want to search PubMed or Embase whether the authors have previously published on the topic of the manuscript. Entering the authors’ names combined with the word “spinal” may be enough. Have a quick look at the abstracts that come up, and investigate if this is a second (or third etc.) version of essentially the same paper the authors have published, possibly with a rotating cast of first authors. Inform the AE immediately if you see duplicate publication, and for the time being suspend your review; if you have only minor concerns, complete the review but tell the AE about them in the “Comments to editor/publisher” feedback box (see below).

In case of a clinical study, you need to inspect the trial registration. The registration number should be provided in the paper, but if not, you need to search the registries, e.g. clinical.trials.gov [14] or the WHO’s International Clinical Trials Registry Platform (ICTRP), which consolidates entries from all registries worldwide [[15]. For a systematic review, the PROSPERO registry may be consulted [16]. Determine what the authors promised to do in the protocol and if they are actually delivering it in the manuscript. If there are major discrepancies (unreported subgroups, unreported outcomes, etc.), and they are not addressed and adequately justified in the paper, it is appropriate to bring the issue up in your review.

Step 5. Upload your review and complete the list of questions the Editorial System poses

In the SC Editorial System, there are two boxes to enter comments on the paper. In the one marked “Comments to the editor/publisher” write all confidential remarks that you do not want the author to see—suspected plagiarism, undeclared COIs, etc. Here you also may want to note limitations in your expertise that prevented you from adequately judging specific aspects, as well as the involvement of other persons in completing the review.

In the box marked “Comments to the author” enter your comments on the manuscript, directly or using cut-and-paste from a word processor document. You also can attach a word processor file, uploading it from your computer. (Please do not attach a file AND paste its content to the “Comments to the author” box—you create double work. Also, do not paste the content of this latter box to the “Comments to the editor/publisher” box—you create double work for the AE.)

In the “Comments to the author” box, do not write anything that expresses your opinion on the publishability of the article—for instance: “this should be accepted without further revision”. If the other referees or the EIC/AE disagree, you may put the EIC/AE in a difficult position. Such judgments should be expressed using the multiple-choice options provided in the SC Editorial System (see Box 4) which are not shared with the author. It is important that whatever you write in the boxes, whether to the AE or to the author, matches the quality rating you express in the Editorial System’s questions (a)–(n) (Box 4). Negative ratings should be supported by disapproving judgments expressed in your review, and vice versa for positive evaluations. Do not recommend “accept” if you have pointed out 3 fatal flaws. Do not recommend “reject” if you note nothing more than 2 typos and a misplaced decimal point.

Step 6. Destroy all review-related information

To reduce the risk of loss of confidentiality, all printed copies of the manuscript and its supporting materials need to be destroyed. It probably is a good idea to also destroy any electronic copies. You should provide the same instruction to everyone who has helped you write your review. If the EIC invites the author to submit a revised version, the author must also provide a version of the original manuscript, marked up to show any and all changes made—which means there is no need for you to hold on to those PDFs.

The review’s content

The key responsibility of a peer reviewer is to write a professional, fair, honest, critical assessment of the manuscript’s contents, analyzing strengths and weaknesses, merits and deficits, as well as ways the paper can be made stronger in ways large and small. The important thing for you to remember is that the reviewer advises (to the AE), but that the AE and EIC decide. As was stated above, avoid (in the comments you write to the author) any language that expresses your opinion on what SC should be doing with the paper.

Many experts recommend starting your review with a 2–3 sentence summary of the author’s objective, and what s/he did, found and concluded. This then can be followed by a critical assessment addressing all strengths and weaknesses, the latter accompanied by suggestions for improvement, if possible.

Many papers on “how to peer review” make a distinction between Major and Minor problems, with the latter including things as mundane as typos [2,3,4,5,6]. It might be better to distinguish in your written review four categories, as follows:

  1. 1.

    Major, and fatal, flaws. These are issues, such as not using blinding of assessors for subjective outcomes, that cannot be remedied without essentially redoing the entire research. (Sometimes the AE/EIC may decide that the study still has value, subject to a rewrite that makes these weaknesses very clear.)

  2. 2.

    Major, non-fatal flaws whose correction would require quite some additional work and possibly major rewriting by the author. For instance, in a systematic review some key primary studies were missed, and fixing this problem requires the author to retrace many steps.

  3. 3.

    Medium-size defects that are easily corrected, for instance rewriting a paragraph of the Discussion that the reviewer considers to be “spin”—the misleading reporting, interpretation, or extrapolation of study results [17].

  4. 4.

    Trivial defects such as typos, erroneous references or referencing, and all other minor flaws that can and should be corrected.

The issue is not so much the number of categories and their definition, but the reviewer having a clear classification for his comments, and a straightforward way of communicating to the author and the AE which problems belong in what category, and which ones need to be absolutely fixed before the paper can be judged acceptable.

It is best to match the length and details of your review to the recommendation you will be making to the AE (see step 5 above). It makes no sense to spend hours writing detail comments if you observe that, for instance:

  • The language is so poor that you have to guess what the author is claiming, reporting, or concluding.

  • The paper’s organization diverges widely from IMRaD—the standard order of Introduction, Methods, Results and Discussion.

  • The methods used are ones that have been discredited by scientific authorities.

  • There are major errors (especially, fatal flaws) in analysis, reporting or interpretation.

  • The Discussion is unrealistic or mostly consists of spin.

In cases like these, where your recommendation to the AE/EIC is “Acceptable with revision but requiring reconsideration by referee” or “Not suitable for publication”, you can just limit your review to noting the major problems you see. But if you recommend “Acceptable with revision not requiring consideration by referee”, you might as well point out all the minor problems and suggest improvements.

In your review, try to make clear which major changes are absolutely necessary, and which ones are strongly recommended but are not crucial. Presumably, a similar concern does not exist for the “minor” issues that you note –the author should be able to follow through on all of those easily. Being clear about what needs to be changed (with a strong suggestion of how) may avoid multiple rounds of re-write and re-review, which would mean more work all around (including for you, when the AE invites you to look at a new version), and delayed publication.

If you are asked to review a revised manuscript, make sure you check whether the author has corrected all major flaws and medium-size deficits that you and your fellow referees pointed out in the first review, or has satisfactorily explained, in the RTR, why the changes you suggested cannot be made. If the paper and/or RTR are not acceptable with respect to these elements, do not drop the issue, but point it out again.

In a re-review, do not bring up all-new points needlessly. It suggests that your original review was not thorough. Sometime a problem becomes evident only in a revised version—in which case it of course should be pointed out. Similarly, all issues that are first brought up by your reading of the new information provided by the author in the revised manuscript should be attended to. If you really overlooked an issue in your first review, and it is important to see it addressed, apologize to the author for missing it.

In principle, hundreds of questions can be asked relevant to a paper, and every time a reviewer states that there is a problem with a manuscript, she could make many detail comments and suggestions for improvement. In Box 5 are the key questions that, as an AE, I want answered, in the “Comments to Editor/Publisher” box, via the “multiple-choice” questions on the SC Editorial System, and especially in the “Comments to author” box. These major questions can be separated into two groups: those concerning the science per se, and those concerning the presentation of the science.

The details that specify where and how a manuscript is deficient, and your suggestions for improvement, will vary tremendously by type of research (traditional quantitative primary research, qualitative research, scoping review, etc.) and the topic of the study. Clearly, even a “simple” study has multiple components, and a reviewer should inspect each part and its contribution to the whole. When writing your review, it is hard to remember to check and approve or disapprove each cog in the machine. There is help available in the form of two types of checklists: peer reviewer study quality checklists (more or less corresponding to the “science questions” in Box 3b and c), and reporting guidelines (corresponding to the “science reporting” issues in Box 3a). Reporting checklists are lists of the study elements that are to be reported for a particular study type, where in a manuscript. They typically have been put together by a panel of experts using the Delphi process to come to a consensus on what is needed to achieve high reporting quality. The EQUATOR website [18] collects reporting checklists, and can be searched for lists applicable to study designs (e.g. RCTs) and subject matter (e.g. acupuncture).

However, reporting checklists do not necessarily get at the quality of the research being reported. While there presumably is a correlation between research design and implementation issues on the one hand, and reporting issues on the other, it is far from perfect. A poor study may have been written up with perfect clarity, and an excellent study can be reported in such disorder and poor language that it is hard to determine what was being studied, let alone to appreciate how outstanding the protocol and its implementation were.

For some referees, a reporting checklist is enough—an item in the list reminds them of the fact that they have to examine whether something is reported (blinding, for instance), and whether that which is reported was performed to a high scientific standard. (SC requires authors to submit a completed version of the checklist most appropriate to their research, which generally can be found in SDC, but you should verify that the language on the manuscript page(s) indicated is indeed adequate). Other referees also need a checklist that helps them to ask the proper questions about the contents of all the IMRaD pieces. Box 3b and c provides references for a number of these checklists.

SC does not require the use of science quality or research reporting checklists by reviewers. But I advise that at least those new to reviewing use checklists in preparing their review. Printing out one or more relevant ones and mentally checking each item to make sure you have not overlooked anything in your review is recommended.

Tie all your comments, questions, and suggestions to the line numbers in the manuscript. This will make it easier for the author to see where the problematic statement or language etc. is, and grounds you in the details of the paper, rather than making wild generalizations—which may easily bring you to using language that needs to be avoided, as discussed in the next section.

Some more suggestions:

  • Check the abstract—it is the most widely read part of the paper, and squeezes much information into 250 words. Can it be understood without reference to the text? Also, check if the abstract concurs with the text.

  • Only ask for a change if you have evidence (which may include expert consensus) that the approach the author took is wrong. If you prefer a particular method, but the one the author used (for presenting data, for structuring the Discussion, etc.) is just as good and serves the paper well, let her be. This is her paper, not yours. Do not create unnecessary work.

  • It is proper to direct the author to published papers that should be mentioned in rewriting the Discussion or Introduction, or consulted in re-analysis of the data. If those papers are your own, think three times before recommending (let alone requiring) them. Of course, in some instances there is no other option: a crucial lapse has to be noted. However, mention the omission in a way that does not reveal your identity as a referee: “Two relevant papers by Smith et al. were for some reason not consulted: …” rather than “You omitted two key papers that I wrote: …”.

  • If the paper is disorganized or written in extremely poor English, remark on it and suggest a solution. There are professional editors who can help organizing the report, and professional translators who can help with the English.

The language of your review

Almost no papers, not even invited ones, are accepted as submitted, and the rejection of a manuscript, or an invitation to revise, is “normal science”. That does not mean that a rejection, even if based on a long list of well-argued comments contributed by three referees, does not have an emotional impact on the author. Referees should do their best to deliver their message in language that does not contribute to the distress possibly caused by the content of the EIC decision letter.

That means first and foremost writing your review in language that is as professional, balanced, impartial, courteous, collegial, and constructive as is possible. Keep any biased, sarcastic, insulting, patronizing etc. words out of your review. Stick to the facts: what the author wrote, what you consider to be correct and incorrect in his report, and specific suggestions for improving the science itself and the report on the science. If you worry that some untoward language has slipped into your review, set it aside for two days and reread (and edit) it before submitting it. Ask yourself: is there any word here that I would change if my name were to appear at the end? If the answer is yes, make the necessary changes: drop certain remarks, soften others, and/or move some comments from the “to author” box to the “to editor” box.

However, it cannot be denied that in some instances nothing much about a paper strikes you as good (in spite of the EIC assigning it to peer review), and even a long list of negative comments is or at least feels hurtful. It always is a good idea to try and find something positive to write in a review. For all of us, it is hard to separate “your paper is substandard” from “you are substandard”, and one way you as a referee can help the author to distinguish self and product is to avoid using the word “you” in your review. This word comes naturally—after all, you are talking to a colleague—but it may put him on the slippery slope of identifying self-worth with the worth of his paper, something you do not want to be responsible for. Even such a term as “The author” may need to be avoided—if you can rephrase that sentence to be about “The manuscript”, it probably helps to maintain that distance between person and product.

If you are reviewing a paper from non-English speaking authors, try to avoid jargon, abbreviations that are non-standard in science communications, and anything that might get in the way of the author correctly understanding your comments, questions, and suggestions.

Lastly—spellcheck your comments. Make sure that all typos and grammatical errors are eradicated to the degree possible. If you are not a native English speaker, it might be worthwhile to install English spellcheck on your computer. The last thing you want to do is point out problematic language in the manuscript, in a review document that is not any better on that score.

Conclusion

To select for publication in Spinal Cord articles that are innovative, important, factual, reliable, and well-reported, the EIC and AEs depend on a cadre of peer reviewers. They rely on these referees to offer them objective advice on which papers are worth publishing, and to present to the authors constructive suggestions for improving the science and the presentation of the science in their manuscripts. Success in this dual role requires you as a peer reviewer to pay careful attention to many components of a manuscript, and to write a peer review report that is expert, evenhanded, unbiased, and constructive. This paper aimed to help you to fulfill this role, and referred to various resources that might be of help (Box 3). I hope that it convinced you that accepting a request to submit a review leads to a challenging but satisfying task that contributes to a product that is of value to SCI science and practice, and to you personally.