Introduction

Rehabilitation in spinal cord injury (SCI) is crucial to maximize the physical functioning, to prevent secondary complication and to facilitate reintegration in the community and in society. Most of the patients with traumatic SCI are young1 and perceive that they are able to work.2 However, involvement in employment is significantly lower than that observed in the population without disabilities.2 Hence, it becomes essential that work participation in SCI needs to be addressed and work disability be mitigated.

The quality of life in people with SCI depends, among other things, on vocational reintegration.3 The early expectations of a person with SCI are an important indicator of successful reintegration.4 Therefore, the period at the end of the first rehabilitation (early post-acute context) becomes critical in initiating steps to start vocational rehabilitation (VR). VR is a key process in work disability management to engage or re-engage individuals in work participation and employment.5 From the social context, VR is designed to maximize work participation of persons with disabilities and to promote their full integration and participation in the society.6 However, VR is a complex process as return to work is a complex outcome. To address this complexity, numerous health status measures are used to assess the functional limitations associated with SCI. However, most of these measures cover only limited aspects of the SCI-related experience of work and vary considerably with regard to content and psychometric properties.7 This degree of variation presents a number of challenges to professionals who would like to compare or transfer patient data across the continuum of care.

The International Classification of Functioning, Disability and Health (ICF) is a universal conceptual framework and classification system.8 Functioning is a result of the interplay between body structures and body functions at the body level and activities and participation at the community and social level, taking into account the personal and environmental factors. ICF can be used across different health settings; hence, its utility can facilitate comparison of outcome data across successive phases of the continuum of care and interventions in SCI.9 In 2010, the ICF Core Set for VR was developed.10 It has 90 ICF categories relevant in assessing functioning and disability in VR. As the ICF Core Set only provides ‘what’ to assess, it becomes imperative to develop an instrument of ‘how’ to assess. At the time, there had been no ICF-based questionnaire specifically developed to capture functioning in vocational settings. Hence, the Work Rehabilitation Questionnaire (WORQ) was developed based on the Brief ICF Core Set for VR. The initial version of WORQ was interview administered, its psychometric properties (test–retest reliability, internal consistency, construct/content validity and feasibility) have been established in a population of patients with various health conditions in a return to work program.11 To promote better practicability and feasibility in the clinical setting, a self-reported version (WORQ-SELF) that contains 40 functioning questions and 18 socio-demographic and work-related questions representing a total of 46 ICF categories was developed. WORQ-SELF can be accessed at http://www.myworq.org/. As a psychometric property fundamental and essential to any measurement, content validity requires relevance of a construct to the intended setting and purpose of an instrument and may be evaluated by experts.12 To date, the content validity of WORQ-SELF in the context of people with SCI in the early post-acute phase has yet to be validated.

Hence, the objective of this study is to establish the content validity of the WORQ-SELF in the early post-acute context in SCI population in VR.

Materials and methods

Mixed methods were employed to study the content validity of the WORQ-SELF in the SCI population in the early post-acute phase by comparing the contents of WORQ-SELF with three different sources: individual patient interviews, the Comprehensive ICF Core Set for SCI and assessment instruments used in VR and SCI (Figure 1: Method synopsis). The ICF was used as the common taxonomy for content comparison across all three sources. The interviews and instruments were linked to the ICF, using published linking rules13 while the ICF Core Set is already presented in terms of ICF categories. All comparisons were performed at the second level of the ICF categories.

Figure 1
figure 1

Method synopsis. ICF, International Classification of Function, Disability and Health; SCI, spinal cord injury; VR, vocational rehabilitation; WORQ-SELF, Work Rehabilitation Questionnaire-Self-Report Version.

Full size image

Phenomenological study using semi-guided interviews

Semi-guided interviews with SCI patients using five ICF-based questions were conducted by one interviewer in an SCI clinic in Switzerland (will be called ‘study center’ from now on) from August 2012 to February 2013. Phenomenological analysis, a popular approach in psychology and nursing, looks at the subjective states and an insider’s perspective, and is often combined with elements of content analysis.14

Study population

A subgroup of the general population, the study included patients with SCI in the post-acute context. This is the period at the end of first rehabilitation and when VR already began. Sampling in qualitative research is concerned with the richness of information and the number of participants required, therefore the full spectrum of individual factors should be represented and samples should consist of participants who best represent the research topic15 (Table 1).

Table 1 Characteristics of participants (N=9)
Full size table

Inclusion criteria for participants included a diagnosis of SCI, must be participating in VR, is age 18 years or older, proficient in German, has no intellectual disability, has been informed of the purpose of and reason for the study, and has signed the informed consent form. Sample size depended on whether saturation was reached. Saturation is the point at which additional interviews are not expected to yield new or valuable information and when depth and breadth of information is achieved.15 Saturation was expected to be reached with about 7 to 10 patient interviews.16

Data collection procedure of the qualitative study

A case report form was administrated to collect socio-demographic, health and work related data. To guide the personal interview, five open questions, similar to the questions used by Glässel et al (2011)17 in the focus group study to explore the lived experiences of persons in VR with regard to functioning and contextual factors related to the ICF components were used. They addressed body function, activity and participation, environmental factors (barriers and facilitators) and personal factors (Table 2).

Table 2 Open-ended questions of the patient interview
Full size table

The study was approved by the Ethical Board of the Canton of Lucerne and conducted in compliance with the Declaration of Helsinki. The recruitment of patients was initiated by health professionals in the VR department of the study center in collaboration with a physician. After giving written informed consent one trained interviewer performed the interviews. The interviews were audio recorded.

Analysis of the qualitative study

The interviewer transcribed the interviews in verbatim and from which concepts were identified. These concepts were ‘linked’ to the best-fitting ICF categories. To check the quality of the linking process, 10% (randomly selected) of the concepts were linked independently by a second linker. Inter-linker agreement between the two linkers was established using kappa statistics. The obtained ICF categories were compared with the ICF categories of WORQ-SELF.

Comparison of the Comprehensive ICF Core Set for SCI with the WORQ-SELF

The ICF categories of the WORQ-SELF were compared with the Comprehensive ICF Core Set for SCI in the early post-acute context. It contains 162 second-, third- and forth-level categories.18 The extent to which WORQ-SELF captures the ICF categories from the ICF Core Set for SCI in the early post-acute context was assessed.

Comparison of assessment measures used in VR and SCI with the WORQ-SELF

Contents of standardized instruments used in both VR and SCI rehabilitation were first identified compared with the WORQ-SELF (Table 3). These instruments were identified based on the systematic reviews conducted in the context of the development of the ICF Core Sets of SCI19 and VR5 and have already been previously linked to the ICF. Contents of the instruments and the WORQ-SELF were compared.

Table 3 Instruments used in SCI and in vocational rehabilitation
Full size table

Frequency analysis of the three sources

A frequency analysis of the ICF categories from the interview, the ICF Core Set for SCI and the identified four standardized instruments with the WORQ-SELF was conducted to examine content validity. Content validity of each ICF category was established if the respective category from WORQ-SELF was identified in at least two of the three sources. This was an arbitrary cutoff decision we made.

Results

WORQ-SELF and interview contents

Characteristics of the interviewed patients are presented in Table 1. In the saturation process patient’s statements added up to 115 ICF categories. Saturation was reached after nine interviews (Figure 2: Saturation process). Thirty-seven of forty-six ICF categories (80%) of WORQ-SELF were also indicated in at least one of the nine interviews. Nine categories of WORQ-SELF were not confirmed by the interviews: b160 Thought functions, b210 Seeing functions, b230 Hearing functions, d160 Focusing attention, d166 Reading, d210 Undertaking a single task, d315 Communicating with receiving non-verbal messages, d270 Complex interpersonal interactions, and d855 Non-remunerative employment (Table 4). The inter-linker agreement indicated an overall kappa of 0.82 (confidence interval: 0.75, 0.89).

Figure 2
figure 2

Saturation process. BF, body function; A&P, activity and participation; E, environmental factors.

Full size image
Table 4 Frequency analysis comparing WORQ-SELF contents versus three sources
Full size table

WORQ-SELF and the Comprehensive ICF Core Set for SCI

Twenty-five of the forty-six ICF categories (54%) of the WORQ-SELF were confirmed by the Comprehensive ICF Core Set for SCI (Table 4).

WORQ-SELF and standard instruments

Out of 106 instruments, four questionnaires were selected for the content comparison with WORQ-SELF, that were used in SCI rehabilitation as well as in VR setting.5, 19 The four questionnaires include the Short Form (SF36)20 to address the concept of health-related quality of life, McGill Pain Questionnaire21 to address pain and the Hospital Anxiety and Depression Scale22 and Beck Depression Inventory23 to address depression and anxiety (Table 3). Fourteen of the forty-six ICF categories (30%) of the WORQ-SELF were confirmed by the four questionnaires (Table 4).

Frequency analysis

In all, 11 of the 46 ICF categories (24%) in the WORQ-SELF were confirmed by all three sources and 15 (33%) were named twice (for a total of 26 (57%) WORQ-SELF categories named at least twice). Thirteen of the forty-six categories (28%) were identified in only one source. Seven categories (15%) were not contained in any of the studies: b210 Seeing functions, b230 Hearing functions, d160 Focusing attention, d166 Reading, d210 Undertaking a single task, d315 Communicating with receiving non-verbal messages and d720 Complex interpersonal interaction (Table 4).

The following five categories of the activity and the participation component were named in all three sources but were not included as a concept in WORQ-SELF: d410 Changing basic body position, d445 Hand and arm use, d460 Moving around in different locations, d640 Doing housework and d760 Family relationships.

The following four categories from environmental factors were named in the Comprehensive ICF Core Set for SCI and in at least eight interviews, but were not contained in WORQ-SELF: e115 Products and technology for personal use in daily living, e120 Products and technology for personal indoor and outdoor mobility and transportation, e150 Design, construction and building products and technology of buildings for public use, and e155 Design, construction and building products and technology of buildings for private use.

Discussion

Our study examined the content validity of WORQ-SELF, an ICF-based self-reported generic questionnaire, to assess work-related functioning of patients with SCI in VR. Contents of WORQ-SELF were compared with three different sources: patient interviews, the ICF Core Set for SCI and standard questionnaires used in SCI and VR.

The interviews gave us a profound insight about the lived experience of the interviewed patients, therefore they weigh strongly.14 The same interviewer conducted all of the interviews to ensure consistency in the interview and evaluation techniques employed across the interviews. The inter-linker agreement with a kappa of 0.82 suggests that consistency was upheld. The ICF categories generated from the content of the interviews confirmed 80% of categories of WORQ-SELF. The results showed that the remaining 20% not mentioned by the interviewed patients reflected predominately sensory functions and cognitive skills. Contrary to the patient’s statements, neuropsychological problems that have been described in the SCI population primarily related to executive functions like attention and concentration,24 in which the interviewed patients at the beginning of their VR process did not experience. Hence, it was not surprising that sensory functions and cognitive skills were not mentioned in the interviews.

The comparison of the Comprehensive ICF Core Set for SCI and WORQ-SELF showed a congruency of 54%. While work-related concepts such as d825 Vocational training, d840 Apprenticeship, and d845 Acquiring, keeping and terminating a job were missing in the ICF Core Set for SCI, WORQ-SELF, in contrast, captures aspects of the VR process as intended, since it is based on the setting-specific ICF Core Set for VR.

With regard to the comparison between four instruments that are commonly used in both VR and rehabilitation of persons with SCI, the overall findings indicated a poor agreement of 14 of 46 categories (30%). While 22% ICF categories in SF36 were confirmed by WORQ-SELF, McGill Pain Questionnaire, Hospital Anxiety and Depression Scale and Beck Depression Inventory had categories mostly related to the ICF component of body function (Table 3). This finding is understandable considering that these four instruments are generic and not specifically developed for use in rehabilitation of SCI patients nor for VR. If these generic instruments are to be used within the context of VR and SCI, then it would be beneficial if a work-specific instrument like WORQ-SELF is used in parallel. WORQ-SELF ensures adequate coverage of activities and participation domains that are essential for designing appropriate work-related intervention.11

In the frequency analysis the categories of WORK-SELF that were not confirmed by any of the three sources belong mainly to mental and sensory functions, learning knowledge and communication and major life areas (education, and work and employment), corresponding to the missing categories in the patient interviews and the ICF Core Set for SCI.18 Even if the missing categories played a limited to no role in the lives of the interviewed patients, these areas of functioning may be relevant for the content validity of WORQ-SELF for the general population.

The five categories identified in all three sources that are not covered by WORQ-SELF represent aspects of activities and participation that SCI patients deal with on an everyday basis. These categories could be added to round out an SCI-specific version of WORQ-SELF.

Although the environmental factors category of e120 Products and technology for personal indoor and outdoor mobility and transportation and e150 Design, construction and building products and technology of buildings for public use were named in two sources and were considered as highly relevant in the interviews (eight of nine patients), they are not covered by WORQ-SELF. The findings suggest that these particular environmental factors would be important to consider to ensure successful work participation of persons with SCI.25 In addition to these environmental factors, the patient’s physical environment like accessibility (transportation and buildings) and work place adaptation can also been seen as important factors for consideration in the return to work process.25

Limitations

The impact of the interviewer on the content of the statements should be reflected upon. Even though the interview procedure was standardized as much as possible and the interviewer was trained in conducting semi-structured interviews, the actual execution of the patient interviews could have impacted on the results.

Moreover, since personal factors (pf) are yet to be classified in the ICF, the pf that were identified by the three sources limits our ability to make comparison in terms of personal factors. Some concepts were coded with ‘not defined’ (nd), for example: in the seventh interview: ‘I am not able to do certain things anymore’. ‘Things’ are too broad term to be linked precisely. Some concepts were coded with ‘not covered’ (nc). They were referred to as ‘not covered’ concepts in established linking rules, for example, autonomy.

In addition, some categories indicated in all three sources were coded at the third or the fourth level. To enable comparability, the third- and fourth-level categories were transformed to the second level, thus making the corresponding categories less specific. As a result, some information may have been lost during the transformation. For example, in the second interview the patient stated: ‘I’m in pain above and below the operating area (back).’ The corresponding fourth-level category b28013 Pain in back was transformed to the second-level category b280 Sensation of pain. As a result, the detail about the location of the pain disappears.

Our findings support the content validity of WORQ-SELF for the subgroup of SCI patients in the early post-acute context. Most of the important aspects of needs and resources concerning VR in this subgroup were considered by the WORQ-SELF. In the near future, further utility of WORQ-SELF in SCI should also consider concepts that were identified as lacking in WORQ-SELF, for example, e120 Products and technology for personal indoor and outdoor mobility and transportation and e150 Design, construction and building products and technology of buildings for public use. Further studies to evaluate content validity in post-acute or chronic SCI patients engaged in VR are needed to gain evidence of the validity of the WORQ-SELF throughout the whole VR process across the health-care continuum.

Data archiving

There were no data to deposit.