No “gold standard” critical appraisal tool for allied health research

Abstract

Data sources Electronic database and internet site searching was performed with no language or time limits. The search terms were broad, and consisted of words such as critical appraisal tools, critical appraisal, critical review form, systematic review form, appraisal of research methodology and research design review.

Study selection Critical appraisal tools were included if they were applicable to at least one research design (quantitative and qualitative research), had clear and unambiguous criteria, and could produce a numeric quality score. Critical appraisal instruments that were not published in full, or were not in English, were excluded. Tools that were used for appraisal of diagnostic instruments and clinical guidelines were excluded.

Data extraction and synthesis The critical appraisal tools were classified according to the study design for which they were intended. Their items were then classified into one of 12 criteria based on their intent. Commonly occurring items were identified. The empirical basis for construction of the tool, the method by which overall quality of the study was established, psychometric properties of the critical appraisal tools and whether guidelines were provided for their use were also recorded.

Results The search identified 193 articles, of which 108 met inclusion criteria. A total of 121 different critical appraisal tools were retrieved. Eighty-seven percent of critical appraisal tools were specific to a research design, with most tools having been developed for experimental studies. There was considerable variability in items contained in the critical appraisal tools. Twelve percent of available tools were developed using specified empirical research. Forty-nine percent of the critical appraisal tools summarised the quality appraisal into a numeric summary score. Few critical appraisal tools had documented evidence of validity of their items, or reliability of use. Guidelines regarding administration of the tools were provided in 43% of cases.

Conclusions There was considerable variability in intent, components, construction and psychometric properties of published critical appraisal tools for research reports. There is no “gold standard” critical appraisal tool for any study design, nor is there any widely accepted generic tool that can be applied equally well across study types. No tool was specific to allied health research requirements. Thus, interpretation of critical appraisal of research reports currently needs to be considered in light of the properties and intent of the critical appraisal tool chosen for the task.

Commentary

Critical appraisal has been defined as the process of assessing and interpreting evidence by systematically considering its validity, results and relevance to an individual's own clinical work.¹ By developing critical appraisal skills, it is envisaged that healthcare workers will become more adept at identifying research articles that are both of a high quality and relevant to their clinical practice. Numerous tools have been published to assist with the process of critical appraisal. The systematic review by Katrak et al² attempts to identify the most appropriate tool for the appraisal of allied health research. The authors undertake a detailed search and identify 121 different critical appraisal tools. Clear details of much of the review process are presented, however, there are some fundamentals issues left uncovered. Firstly, there is no definition as to what constitutes a critical appraisal tool. Consequently, publications are included in the assessment, which, it could be argued, are not critical appraisal tools. For example, publications such as the QUOROM,³ MOOSE⁴ and CONSORT⁵ statements are aimed at improving the reporting of research studies rather than the appraisal of them. Similarly, assessment tools developed for use within a systematic review often focus purely on key issues of internal validity relevant to that particular systematic review; they are not meant to be used to provide a full critical appraisal of individual studies.

The authors categorize the identified tools according to the study design(s) for which they are meant to be used. Eleven criteria are then used to evaluate the component items presented in each tool. Several of the criteria are irrelevant to certain study designs making comparisons across categories of appraisal tools unfair. For example, tools developed for the appraisal of systematic reviews, observational studies and qualitative studies are all assessed for their inclusion of ‘method of randomization’; the study designs themselves preclude randomisation.

Emphasis is placed on those appraisal tools that provide a numeric summary. There has been much debate over the use of such composite scales and it is suggested that the assessment of individual components may be preferable to producing an overall summary score.⁶

Despite problems in the methodology used, the conclusions of the systematic review² are probably true; there is no “gold standard” critical appraisal tool. Practitioners wishing to undertake critical appraisal of the research literature may find study design specific tools more useable than generic tools as they are less likely to contain irrelevant items. Several organizations have developed easy-to-use, study design-specific checklists to guide the reader through a research article (eg Critical Appraisal Skills Programme⁷). Whichever tool is chosen to aid critical appraisal, consideration must be given not just to the validity of the research but also to the results of the study and the relevance to the reader's own clinical setting.

Practice point

• Practitioners may find study design specific tools more useable than generic tools, but whichever tool is used consideration must be given not just to the validity of the research but also to the results of the study and the relevance to the reader's own clinical setting.