Letters to EBD

Re: Commentary: Poor quality evidence suggests that failure rates for atraumatic restorative treatment and conventional amalgam are similar. Evid Based Dent 2012; 13: 46–47.

Dear Sir

We would like to thank Dr Dominic Hurst for reading our systematic review update.¹However, most of his criticism of our article seems unnecessary.² His objections to the appraisal of evidence regarding the question as to whether the placement of an ART (atraumatic restorative treatment) or a conventional amalgam filling would yield lesser, same or higher chance of failure appears to be underlined by the opinion that ART may not be relevant for modern dental settings. This is an unfortunate misconception.³ Of course, other questions to the clinical merits of ART are important, too, and we look forward to learn of the findings of Dr Hurst's own two, still ongoing systematic reviews to the topic.^4,5

Dr Hurst is correct in his assumption that we did not know about the five Chinese medical databases.⁶ We would like to thank him for pointing this out and will focus on it in our next systematic review update. However, the concept of language bias needs to be regarded within the context that the exclusion of non-English trials may have little effect on summary treatment effect estimates.^7,8 The results of all non- English trials (including the Chinese trials) in our review update seem to confirm this.⁹

Some further points of criticism raised by Dr Hurst are: the question as to which literature source our identified Chinese trials originate from (but we made all literature sources explicit in Tables 6 and 7, as well as in the text); the question to why 'ART' [MeSH] as a search term was used (but it can be discerned from our Table 4 that this was the first and broadest search term (#1), which then got narrowed down to the final search term #6 resulting in 260 hits); his confusion regarding the randomisation status of accepted trials (but this is presented in our Table 14 on hand of all verbatim quotes extracted from the text of accepted trials that had relevance to selection-, performance- and detection bias risk); lack of summary of the included trials (but important characteristic of trials are presented in Tables 8–10); the impression that more evidence is given for primary than for permanent teeth (but our Table 13 rather indicates a higher number of evidence relevant to permanent teeth in terms of measured outcomes and number of subjects); the questioning of the 'usefulness' of grouping 'restorations evaluated according to USPHS criteria after 1 year' if type of dentition nor restoration class was specified (yes, 'usefulness' is indeed limited when important information are not reported in trials, but a pooled effect estimate – aspects of heterogeneity allowing - represent available evidence better than a number of individual ones).

In addition, when reading through the PRISMA checklist¹⁰ it is difficult to understand which section we may have missed. Indeed, our review report is rather complex but this is due to the detail and quantity of the evidence and not due to non-adherence to the current reporting guidelines for systematic reviews.

Careful reading of the latest version of the Cochrane handbook for systematic reviews of interventions¹¹will reveal that we have adopted most of its recommendations in our review update – with one exception: we did not follow Cochrane recommendations concerning particularly the assessment of selection bias risk. The reasons are important:

the Cochrane handbook provides the rationale (Section 8.9.1) and the criteria (Section 8.5d) for assessing adequacy / inadequacy of methods regarding randomisation (the generation of a random sequence) and allocation concealment.¹¹ Both are based on the premise that adequacy in randomisation and allocation concealment are sufficient in most cases to protect against selection bias. It is important here to note that this exclusively means the attempt to randomise and conceal adequately and not whether such attempt was indeed successful at the end of a trial.

The Cochrane handbook acknowledges that trials may carry the effect of selection bias, such as biased inflation of effect estimates, due to correct prediction of the random sequence, when the attempt of randomisation and allocation concealment was adequate (according to Cochrane criteria), particularly in cases when block randomisation is used and all allocations are known after enrolment. Here the Cochrane handbook gives reference to the work of Berger et al. (Section 8.15.1.3) but, unlike Berger et al., the Cochrane handbook (i) assumes such an event as a minor exception, (ii) does not propose any solution for systematic review authors in such case and (iii) does not provide evidence in support of its assumption that such an event may indeed be a minor exception, only.

Instead, the Cochrane handbook cites a number of meta-epidemiological studies as evidence in support of its stance that clinical trials with inadequate randomisation and/or allocation concealment attempt are at higher risk to report inflated effect estimates than clinical trials where such attempt is judged as adequate.

These meta-epidemiological studies were appraised among others as part of a systematic review concerning the effect of randomisation and allocation concealment on the results of healthcare studies.¹² The summary outcome of this review shows conflicting evidence. It is our opinion that this may be partly due to the fact that a mere adequacy of attempt to prevent selection bias cannot guarantee lack of inflation of effect estimates due to such bias. We remain therefore somewhat sceptical of the Cochrane recommendations for assessing selection bias risk, and adopted the position by Berger et al.¹³ Of course, such a position differs from the mainstream line of thought and seems, due to unfamiliarity with it, not easy been understood. However it can be summed up by the simple allegory: if one wants to lose weight, an adequate attempt to do so is to join a gym (an inadequate one is the plan to watch more sport on TV). However, such adequate attempt won't do anything for one's weight problem, if one has not actually gone to the gym, or went to the gym regularly to only drink coffee there. In other words the result of an attempt (= focus on outcome) counts, and not the attempt itself (= focus on the process).

We have chosen such a position as premise of our systematic review, explained it in the text and provided the necessary references.^14,15 Since 2005, Berger et al. advocate the inclusion of tests into the methodology of RCTs that provide empirical evidence as to whether randomisation and allocation concealment were indeed effective in protecting against selection bias.^13,14,15 We believe that only the reporting of such evidence merits the judgement of a trial as being of 'low selection bias risk'. According to such a rather stringent standard, we agree with Dr Hurst that all of our appraised evidence is to be regarded as 'poor'. However, such judgement has to be seen against the background that there is no clinical trial in dentistry, yet, (at least that we know of) in which Berger's recommendations have been applied. Thus the internal validity of our identified evidence seems not distinct from most, if not all, clinical dental trials. There simply seems to be little or no difference between the effect of bias in a clinical controlled trial without randomisation and one that has reported the application of adequate attempts (methods) to randomise and conceal but could have been biased by eg foreknowledge of the random sequence due to correct prediction or by its intentional/accidental direct observation during the process of the trial. Current systematic review conclusions appear to confirm this point.¹²

Of course, we only can know for sure when we can compare selection bias test results with the adequacy of reported randomisation/allocation concealment attempts. In order to make such a type of investigation possible, selection bias testing needs to be adopted in a sufficient number of RCTs, first. For this purpose, the Berger-Exner test has been recommended.¹⁴ A further, maybe simpler, type of testing that is also based on the reverse propensity score (RPS) shows promise but still requires further investigation.^16,17

Against the background of such high-level scepticism, care needs to be taken not to throw the baby out with the bathwater: in order to be of any use to clinicians and policy makers it is important that systematic reviews establish with due precision what is currently known for a particular clinical question; they need to assess further the validity of such current knowledge in view of potential bias risk and on that basis recommend where and how further research is required. Until future research provides falsification/verification of what is currently known and puts to rest justified doubts due to assumed poor internal trial validity, it is important (i) to remain sceptical of the existing evidence; (ii) to know the extent of the current knowledge as reflected by the dental literature but also (iii) to appreciate that (with all its known limitations) such knowledge is the best that is available to us – for now.

Re: Commentary: 10% chlorhexidine varnish did not reduce caries in an adult population. Evid Based Dent 2012; 13: 45.

Dear Sir

With regard to Dr. O'Keefe's review of the 10% chlorhexidine varnish in the current issue of Evidence-Based Dentistry, we state these facts:

• 1 The Papas et al. article in the Journal of Dental Research did state in its Appendix that this treatment significantly reduced cavitated (D2) caries.

• 2 D2 lesions were the endpoint of the FDA protocol. Using that protocol, this treatment reduced dental decay vs placebo by 70% (p=0.003) in high-risk patients (those who started the treatment plan with three or more lesions – about 40% of total study participants). Importantly, the high-risk participants in this study accounted for the vast majority of caries.

• 3 Dental examiners in this study could not reliably diagnose the D1, non-cavitated lesion at baseline. Only two of seven examiners had satisfactory kappa scores. Hence, the data on changes from D1 throughout the study have no integrity. This supports the position of the FDA that general practitioners cannot reliably diagnose an early lesion and hence most practitioners use the D2 lesion to formulate their treatment plan.

• 4 Three European pharmaceutical regulatory agencies have thoroughly reviewed the two controlled adult studies of this treatment, and have issued approval of the first therapeutic indication for the prevention of dental decay in high-risk adults. This is the only in-office treatment which has undergone this rigorous and detailed review – a evaluation which far surpasses that of any peer-reviewed journal.

Our pioneering research of the prevention of adult dental decay using randomised controlled clinical trials and protocols acceptable to the regulatory authorities has taken years and is entirely unique. We expect you will want to balance Dr. O'Keefe's incomplete and misleading review by printing the above facts.

Thank you.

Dr. O'Keefe responds

I would like to thank Mr. Perry for his observations. I am fully supportive of the robust methodology used in this multi-centre RCT but I believe it is correct to question the benefit of chlorhexidine varnish in the adult population. There is a body of evidence, mainly based on children and adolescents, that has considered the role of chlorhexidine varnish in the management and prevention of dental caries and at this point in time there is inconclusive evidence to support its use in the prevention of dental caries.¹⁸

My commentary was based on Papas et al.'s paper,¹⁹ and used additional information on the rationale and study design from Vollmer et al.'s paper.²⁰ The paper critiqued supports the conclusions from previous research²¹ and the commentary is a true reflection of the study's findings available in the paper. The study concluded that chlorhexidine varnish 10% was not effective in the prevention of coronal caries. The paper supports the need for additional research into the use of chlorhexidine varnish in the prevention of root caries in high-risk patients.²² A more recent systematic review²³ concluded that chlorhexidine varnish may provide a beneficial effect in patients in need of special care.

Whilst this is a disappointing finding for the company and for those suffering from one of the most prevalent chronic diseases in the world; the researchers and journal editors have to be commended for reducing publication bias by publishing negative findings²⁴ and encouraging further research in this area.

In the exploratory analyses in the appendix the authors state that there was an unplanned analysis of the 580 individuals with two or more D2 lesions at screening which found that there was a significant reduction in D2 lesions in the active group when compared to the placebo group. The authors further explore this finding and highlight to the reader that the 'suggestions of a beneficial effect in those with two or more D2 lesions at screening in the net D2FS net increment should be viewed cautiously in light of the null results of the planned primary and secondary outcome analyses and the opposite (though not significant) effect in those with only one D2 lesion at screening'. The authors in the main paper concluded that there was no significant difference. The analysis within the paper was based on D1-2 according to ClinicalTrials.gov. I have been given to understand that the FDA analysis was subsequently conducted on the data by CHX Technologies and Schiff and was therefore not part of the PACS study.

Regarding the observational accuracy of D1 and D2 I do agree that diagnosis at the D1 level is challenging without the support of radiographic information.

Given the robust methodological approach of this multi centre RCT my focus was not about the use of chlorhexidine varnish in high-risk adults but to question whether there is sufficient evidence to recommend chlorhexidine varnish as a treatment option for the adult population. Consequently, it is encouraging that two protocols for systematic reviews into the use of chlorhexidine interventions are registered with the Cochrane Collaboration. The systematic review by Bailey et al.²⁵ will focus on the adult population and has a number of objectives including:

• to evaluate the effectiveness of chlorhexidine-containing oral products in adults on reducing dental caries rates as compared to topical fluoride treatment, placebo or no treatment

• to assess if there are optimum parameters of administration of chlorhexidine-containing oral products (eg form, concentration, frequency) and

• to examine whether the level of caries in the population influences the effect of the chlorhexidine product.

While the Eberhard et al. protocol²⁶ will compare chlorhexidine and fluoride treatment for the prevention and management of dental caries in children and adolescents.

In due course the recommendations from these reviews will hopefully be able to provide evidence-based guidance on how best to manage dental caries in all age groups.

Send your letters to the Editor

c/o Rowena Milan

Evidence-Based Dentistry

Nature Publishing Group

The Macmillan Building

4 Crinan Street London

N1 9XW

E-mail: ebd@nature.com