Introduction

This paper presents a qualitative and quantitative evaluation of the effectiveness of a cross-infection control CD-ROM. The CD-ROM was developed for NHS dental teams as a continuing professional development training tool. The program was commissioned by the Department of Health and developed by a project team through the UK Committee of Postgraduate Dental Deans. One copy of the CD-ROM was sent to each of the 8,926 dental practices in England by the Dental Practice Board during April and May 2004. The intention in creating this innovative interactive e-learning program was to raise awareness among the dental team of the importance of cross-infection control, to provide a relevant information and training resource and to enhance understanding of good practice in this area. A combination of text, film and audio information with careful attention to design had resulted in an attractive learning resource (Figs 1 and 2).

Figure 1: Opening screen shot of CD-ROM that lists the different content sections in the left hand bar.
figure 1

The program contains several video clips, an example of which can be seen on this page

Full size image
Figure 2
figure 2

Example of a question from the pre-test. Participants subsequently submitted the answer online

Full size image

Continuing professional development for dental practitioners is now part of a 'never ending cycle of training'.1 Therefore it is important to evaluate initiatives such as this CD-ROM to establish whether overall they make an effective contribution to lifelong learning for the dental team. As stated above, this evaluation used two complementary approaches. The qualitative evaluation was chosen as a means of determining 'real world' opinions and attitudes to the CD-ROM, via text based responses to a survey. The quantitative evaluation was chosen to give quantifiable and measurable information, via a statistical analysis of a set of pre- and post-test results, generated by users of the CD-ROM. The sample sizes used were relatively small, but conform to best practice in social science research in a number of respects,2,3 including resembling the overall demographic profile of the dental profession.

Method

The evaluation of online learning is notoriously difficult.4 The key aim for the evaluation team in this case was to attempt to establish the effects of the CD-ROM on individual performance and to measure users' attitudes towards the CD-ROM as well as its perceived effectiveness as a training tool for professional training and continuing professional development (CPD). Key research questions included:

  • Do practitioners feel that the CD-ROM supports their continuing professional development usefully and appropriately?

  • Has the CD-ROM raised their awareness of cross-infection control?

  • Is the CD-ROM considered to be a useful guide to best practice in this area?

  • Was the design and construction of the CD-ROM considered to be fit for purpose by practitioners?

As a means of addressing the inherent difficulties in evaluating the CD-ROM, the evaluation was based on two kinds of data. The first category involved data derived from a group of users of the CD-ROM who used the online assessment developed with the program. All users were invited to complete an online test, assessing their knowledge of cross-infection control. Ideally, users completed the test twice, before and after using the CD-ROM. The tests were integrated into the CD-ROM design and individual users submitted their test results electronically over the Internet. Out of the 8,926 practices that were sent the CD-ROM, there were 326 online users who completed the assessments. A range of statistical tests was carried out on this data.

The second category of data was derived from a paper and online survey of 118 practitioners, whose practices had been drawn from the Dental Practice Board database as part of a random sample of 1,000 dentists across England. (This had been preceded by carrying out a pilot study of twelve practices, inviting them by telephone to receive and return a copy of the paper questionnaire. Eleven of the practices subsequently returned the questionnaire). There was also an invitation to respond to the online survey published in the British Dental Journal. Due to necessary uncertainty about how many practitioners actually saw the invitation to respond to the online survey, we have no way of knowing actual response rates, but we would estimate them to be fairly low, in the region of 11.8%. Despite the fact that there were some access problems relating to the sample, this response rate is normal for surveys of this type in the present research climate as individuals are increasingly reluctant to respond to surveys,2 although useful and relevant data can still be collected using this research tool. In this case, the subsequent data were analysed both quantitatively and qualitatively. In this evaluation, we were not able to make any precise assumptions as to how representative this sample might be, or whether the group of respondents overlaps with those already assessed as part of the quantitative study of pre- and post-test results. However we believe the evaluation to be valid, as we ensured that best practice in survey design3 had been followed by applying the following design criteria:

  1. 1

    We used a large sample group, in this case 1,000 dentists drawn from the main database, to ensure a sufficiently large data set of respondents for subsequent analysis

  2. 2

    We checked that the demographic profile of the respondent group resembled that of the dental profession overall5,6,7

  3. 3

    We ensured the questionnaire was concise, clearly laid out and had a simple design, and ran a pilot study of 12 practices to check this

  4. 4

    We sent a cover letter from the project's principal investigator with each copy of the questionnaire

  5. 5

    We supplied contact details for the lead researcher on each copy of the questionnaire

  6. 6

    We offered an electronic option for responses, although only 12 respondents took advantage of this.

We therefore offer this evaluation as a useful case study to support the future development of computer-based continuing professional development activities in dentistry.

Results

Quantitative analysis of early online respondents

Use of the cross-infection control CD-ROM course was significantly related to an increase in knowledge of the subject. Survey participants who had a lower initial level of knowledge of the subject, as shown in the pre-test, tended either to 'drop out' before completing the CD-ROM course content, or not to complete the post-tests, which was demonstrated from their online responses. There could be several reasons for this. It may have been because they did not consider the CD-ROM sufficiently engaging to motivate their learning, or alternatively because they were not particularly interested in the subject matter in the first place.

About the survey cohort

  • Three hundred and twenty-six dentists took the CD-ROM assessment and submitted answers to the pre-tests between 17 May 2004 and 8 May 2005

  • Only 42 participants (out of 326) completed both pre- and post-tests. Within the survey cohort, no significant differences were present in respect of demographic characteristics (age, number of practising years, practice type, or genderFootnote 1). On average, the group who completed post-tests scored more highly in the self-assessment test (M -75.2, SD 15.6) than the group who did not complete post-tests (M -47.2, SD 37.8)

  • Around 45.2% of survey participants worked in an NHS practice, 10.3% in private practice and 44.6% in practices that offer both private and NHS dental health care. On average, each practice employed three practitioners (M -2.91, SD 2.5) and one hygienist (M -0.92, SD 1.21)

  • Around 39.4% of participants were female. Female dentists among the survey cohort tend to be younger than the male dentists, and this difference is statistically significant at 0.01 level (χ2 = 31.54, df 2, n = 287). This can be attributed to demographic shifts as a consequence of changing recruitment patterns in the profession over the last two decades5,6,7

  • The average number of practising years was 19.69 (SD 9.55). Within the survey cohort, dentists working in private practice tend to have accumulated significantly (p <0.05)Footnote 2 more professional experience, with this group reporting a higher average number of practising years than dentists working in NHS-only practices (M -21.05, SD 11.2 and M 18.7, SD 9.2, respectively).

Pre- and post-test results

Mean scores and standard deviations for each of the tests can be seen in Tables 1 and 2.

Table 1 Average test scores for all 326 participants who completed the pre-test
Full size table
Table 2 Test scores for participants who completed both pre- and post-tests (n = 42)
Full size table

As can be seen in Table 1, the average score in the post-tests (24.9%, SD 35.2) for all participants is considerably lower than for the pre-test score in the self-assessment section (50.8%, SD 36.9). This difference is most likely attributable to the fact that the majority of participants are not likely to have completed the post-tests, as they were shown by the system as having scored 0% for these. Therefore to give an accurate picture, these participants were excluded from the subsequent analysis (see Table 2).

Table 2 shows a comparison of pre-test scores (self-assessment section) to post-test scores only for those participants who completed both pre- and post-tests. This table indicates that all post-test scores for those participants who completed both pre- and post-tests were higher than scores in the pre-test, ie self-assessment section (M -75.2%, SD 15.6).

This difference between average results in pre- and post-test results for participants who completed both tests was significant at 0.01 level.Footnote 3

Quantitative and qualitative analysis of survey responses

The questionnaire was an A3 piece of paper, printed on both sides and folded, with a range of open and closed questions in six sections. The open questions offered space for a free text response, giving dental practitioners the opportunity to express views and opinions using an unrestricted format. The closed questions were used for data gathering purposes and tended to require yes/no answers, or for a choice to be made from a pre-determined list of possible answers, eg Deaneries. The names of the different sections and their titles are described below. The online version had identical questions but was designed using an online service with a bespoke URL (web address) for the survey. It involved the use of radio buttons, in which dental practitioners were required to choose one or more answers from a range of options, the use of drop down menus, in which practitioners were required to choose one option from a list, and free text boxes. The data from the two versions were analysed statistically in the case of the closed questions. In the case of the open questions, the repetition of key themes derived from the responses was tabulated and reported accordingly.

Sample

Dentists who responded to the paper or online survey were most likely to be aged between 35 and 44 years of age (30.5%), but were half as likely to use the cross-infection control CD-ROM if they were over 55 years of age (15.3%). Just over two-thirds of the dentists who responded were male (68.6%). This may reflect the gender balance of dentists nationally or relate to female workforce issues as outlined earlier in this paper. There was a geographical spread of dentists, but the largest single group was from the Northern Deanery with 20.3% of respondents, whereas the Wessex Deanery provided the smallest group at just 6.8%. Nearly half of the respondents reported that they had not seen a copy of the CD-ROM, but still responded to questions relating to their IT skills and access to IT equipment.

IT skills and equipment section

60.6% of dentists who responded to this question reported habitually using a workplace computer for training and professional development activities. The location of activity varied. Just over a quarter of respondents were most likely to use the CD-ROM exclusively at home (28%), although many used the CD-ROM exclusively in the workplace (23.2%) or a mix of the two locations (17.1%). There may have been competition for technological resources in the case of some dental practices. Often the computer was shared with three or more other people, as was the case for approximately a third of responding dentists.

Just under half of the dentists reported experiencing few barriers in relation to using IT for training and professional development. However 41% reported that lack of time proved to be a barrier and 14.1% reported that their physical environment was too cramped. Of those that used computers to engage in continuing professional development, 41% used them for online non-interactive activities such as reading journal articles and two thirds (66%) used CD-ROMS, which reflects a growing trend within academe and the professions nationally. Computer use was generally widespread, as only 3% of dentists reported not owning or having access to a computer. Many dentists felt that the use of IT for training purposes helped them to communicate and collaborate with their colleagues (43%) and 46% reported that the use of IT helped them to take control of their professional environment.

Use of the CD-ROM section

Unfortunately just over half (50.4%) the dentists reported not receiving the CD-ROM, which had been sent out originally by the Dental Practice Board (DPB). At the time of the distribution, the DPB database held addresses for dentists at which they received payment; therefore, it is unlikely that there were this number of inaccurate addresses. It is more likely that whoever received the package in the practice retained it for personal use, put it on a shelf or, possibly, binned it. In these instances, others in the practice would not be aware that it had arrived. In turn, this may well have had the effect of reducing practitioner engagement with the CD-ROM nationally.

Of those that did receive the CD-ROM, 54.2% reported using it personally and 55.6% reported using it as part of their dental team. 12.5% used it as part of wider group learning or blended learning. The main users of the CD-ROM were dentists (40%) and dental nurses (35.7%), although 7.8% of the respondents reported hygienists using it.

Overall assessment section

Overall, the CD-ROM appears to have been regarded as a success. Of those respondents that had had access to the CD-ROM, the overwhelming majority rated it highly, with 94.5% feeling that it was relevant to clinical dental practice and 85.5% feeling that the CD-ROM built effectively on existing knowledge. Eighty-three percent felt that the CD-ROM met the needs of practice staff in helping to prevent cross-infection control. Seventy-nine percent felt that they would be able to recommend the CD-ROM as a training tool to other colleagues and staff. Eighty-six percent felt that the CD-ROM raised awareness of relevant issues, and 78% felt that it had had a direct impact on clinical practice.

Subject matter section

The CD-ROM also appeared to be regarded as fit for purpose in terms of its design. Most dentists (86%) reported that they felt the reading level of the CD-ROM was appropriate and that technical terms were explained adequately (88%). Almost as many felt that abbreviations were used appropriately (74%) and that the CD-ROM avoided culturally biased references or examples (63%). The majority of dentists (86%) felt that the CD-ROM worked well visually on the whole, although 40% commented that the screen layout was cluttered and 40% felt that the type styles were difficult to read.

Overall, however, 84.2% of dentists felt that the CD-ROM was easy to navigate. Of the remainder, 80% commented that it was hard for them to know when they had completed each section. The use of images and videos was considered to be relevant by 80.7%. The questionnaires and self-testing exercises were generally highly regarded. 89.5% of dentists reported that the questions seemed to be relevant and 70.2% reported that the questions delivered adequate feedback. 71.9% reported that it was clear how to make a choice when answering questions and 66.7% felt that the CD-ROM contained adequate help and guidance notes.

Using the CD-ROM section

Just under a third of dentists (29.1%) spent 1-2 hours using the program. However, nearly half of the dentists (47.3%) reported spending 2-4 hours using it. A smaller number, 18.2%, reported spending 4-7 hours using it and a small minority (5.5%) spent 7 hours or more using it. Over half the dentists (52.5%) used it in the evening, with a quarter (25.5%) putting aside a dedicated time for using it. The next popular time for using it was lunchtime, when 18.2% of dentists used the CD-ROM. Smaller numbers used it in the morning (7.3%) and afternoon (10.9%), with 9.1% using it between patients.

Overall views

Dentists made a range of qualitative comments relating to different aspects of CD-ROM use. When asked whether the survey met their personal training objectives for practice staff in cross-contamination, one or two dentists commented that more information was needed about dual contamination and also CJD.

In relation to the main benefits of the CD-ROM, a range of comments was made. Generally, it was felt that the ability of individuals to work at their own pace and at a time of their own choosing was important. One respondent reported that the portability of the CD-ROM was useful, as well as the fact that it was free, on account of his NHS work. Another dentist reported that as a result of the CD-ROM, his practice had booked on a cross-infection course with a well-known company. Several dentists commented that the CD-ROM provided a good general reference source, as well as being up to date.

Dentists were asked to indicate how their clinical practice was likely to be improved or updated as a result of using the CD-ROM. Over 50% of respondents who used the CD-ROM reported that they would be purchasing more sterilising equipment and/or reviewing their procedures generally.

Finally, dentists were asked if they would like to make other comments or questions about the CD-ROM. There were many requests from dentists for copies of the CD-ROM, as they had not received it and felt it would be very useful. One dentist commented that larger practices needed more than one copy. There were a few comments about online learning in general, including comments that CD learning was dry and uninteresting, as well as time consuming. However when the quantitative findings are taken into account, such views about online learning appear to represent a minority of users.

Conclusions

It can be concluded that the use of the CD-ROM program is related to the observed increase in the level of knowledge. We suggest, therefore, that use of the CD-ROM appears to have had a positive impact on users' level of knowledge of cross-infection control. However, this finding is restricted to a subset of the total survey cohort. It should be taken into account that those survey participants who had a lower initial level of knowledge of the subject tended either to stop using the program or not to complete the post-tests. As stated previously, this may have been due to lower levels of overall interest in the subject matter, or the fact that the CD-ROM proved to be less effective at accommodating their learning styles. Further research would be required to establish whether this was the case. In addition to this finding, we discovered that there were no significant differences in pre-/post-test results between NHS, NHS/private and private dentists.Footnote 4 Also pre-/post-test results were not significantly related to respondents' gender, age or number of practising years.Footnote 5

Overall, the content of the CD-ROM was considered to have been a success. This was shown in several ways. The quantitative analysis of pre- and post-test scores demonstrated that use of the cross-infection control CD-ROM course is significantly related to an increase in knowledge of the subject and this is not dependant on age, gender, or whether the practitioner works privately or within the NHS. Additionally, in the survey responses, it could be seen that practitioners rated the CD-ROM extremely highly. The majority felt that it was well-designed and fit for purpose, in that it supported and extended their subject knowledge in the area of cross-infection control whilst at the same time acting as a useful reference tool when necessary. It appeared to be having a positive influence on everyday practice in surgeries, for example by encouraging investment in new cross-infection control equipment and revision of surgery procedures. In this sense, it conforms to Maidment's view8 that effective adult learning should be both interactive and relevant, whilst building on dentists' own experience. Overall it was well received by the majority of practitioners. This appears to be typical for dental education CD-ROMS or computer assisted learning programmes of this type.9,10,11

Negative comments in relation to the CD-ROM were comparatively rare, and mainly related to typestyle and layout, use of online learning for CPD and distribution failures. Consequently, it may be that in developing future CD-ROMs, further thought needs to be given to the suitability of typestyles and layouts for practitioners with learning difficulties such as dyslexia. A range of CPD opportunities should continue to exist, as some practitioners find online learning of this type unsuitable for their purposes.

The distribution failures were probably the most concerning aspect of the development of this CD-ROM. Unfortunately, despite the CD-ROM proving to be a valuable resource for those practitioners able to access it, we estimate that up to half of all dentists either did not receive the CD-ROM or have sight of it. There are several possible reasons for this, some of which were alluded to earlier in this paper. It may also have been as a result of the CD-ROM being sent to the principal of each dental practice, who may have failed to hand it on to colleagues. Alternatively, others in the practice may have opened the package and not made it available to all in the practice. Therefore, it would be advisable for more effective distribution systems to be put in place if further CD-ROMs are to be sent out in the future. Effectively, this probably means that each dentist should have been sent a CD ROM rather than just sending one to each practice.

Alternatively, in the future it may be more appropriate to provide this training online via the Internet. In addition to this, e-learning materials should be advertised more widely in order that those dentists who are overlooked for any reason are made aware of their development and, therefore, are able to make their own arrangements to access them as necessary.