Main

Patient-completed electronic family history (FH) collection tools have been proposed as a way to overcome several of the barriers to collection and use of FH information in health-care settings.1,2 Several patient-completed tools designed to aid in the collection and interpretation of FH have been developed and validated, often in settings such as cancer and primary-care clinics.3,4,5 A 2009 review of FH literature conducted for the US Agency for Health Care Research and Quality and the National Institutes of Health identified a paucity of validated FH tools suitable for use in primary-care or public health contexts.6 In 2014, the National Colorectal Cancer Roundtable found that electronic health record systems are poorly prepared to store and interpret FH relevant to colorectal cancer (CRC) risk assessment.7

The US Centers for Disease Control and Prevention–sponsored Family Healthware Trial investigated behavioral change related to the clinical use of a patient-completed electronic family history tool (FHT) that provided risk assessment for several conditions, including type 2 diabetes, heart disease, stroke, and colorectal, breast, and ovarian cancer.8 The FHT was not designed for stand-alone CRC risk assessment, but extensive expertise was brought to bear in its development, suggesting that the FHT may be a useful point of comparison for the evaluation of other automated CRC risk assessment tools. The public version of the My Family Health Portrait (MFHP) tool was developed and launched in 2004 as part of the US Surgeon General’s Family History Initiative. The tool was designed to help the public collect, organize, and share FH information with relatives and health professionals.9 The current version of the MFHP was designed to collect FH in a format compatible with electronic health record systems using nomenclature and data standards for storing and sharing FH; the MFHP has been studied in several settings and is freely available to the public.10,11,12,13

An important limitation of the current public MFHP tool is that it does not provide users with feedback or education regarding their individual disease risk. Here, we report on the development of a CRC risk assessment module compatible with the public MFHP tool and, potentially, electronic health record systems. Moreover, we present data regarding the analytic validity of the new module and the FHT according to the analytical validity; clinical validity; clinical utility; and ethical, legal, and social issues (ACCE) framework proposed as a method for evaluating family health history and other genomic health tools.14

Materials and Methods

Creation of the CRC risk module

The MFHP tool output is a structured XML output file that is computable for risk assessment. An algorithm based on National Comprehensive Cancer Network and United States Preventive Services Task Force CRC risk assessment guidelines was developed to dichotomize individuals into “elevated” and “not elevated” risk categories based on data entered into the MFHP tool (Supplementary Appendix S1 online). To avoid inducing potentially inappropriate distress in individuals without benefit of immediate access to evaluation by a health professional, individuals at “elevated” risk were not further subdivided into “high” and “moderate” risk categories by the MFHP tool. The model did not incorporate other data elements, such as proband gender, body mass index, or self-identified race, because there were no evidence-based guidelines for assigning risk for these factors in conjunction with FH data. The final algorithm was reviewed by a group of cancer risk assessment experts prior to being programmed into an open-source software tool. This input resulted in the final decision to provide a dichotomous rather than a more quantitative risk assessment and to consider all reported polyps as potential sources of risk warranting further discussion with a health-care provider. The output of the MFHP tool is risk-specific CRC information printable as letters for public users and their health-care providers.

Study population and risk algorithm validation

We used 150 patient-entered pedigrees sequentially derived from the ClinSeq cohort as the substrate for assessing the MFHP tool and the US Centers for Disease Control and Prevention FHT CRC module’s analytic performance.15 The cohort consists of self-selected volunteers with an interest in learning their personal genomic risks through genome or exome sequencing; individuals entered the study without investigator foreknowledge of cancer FHs. The cohort is more likely to be white, well-educated, and affluent than the general US population. Forty additional patient-entered pedigrees derived from the cohort served as internal controls; 20 were deliberately modified to increase CRC risk to different levels (10 “strong” risk and 10 “moderate” risk) by adding additional accepted risk factors to the pedigree data.

Three different methods were used to derive risk estimates from the test pedigrees: the MFHP tool (including the new CRC risk module, referred to as MFHP hereafter); the Centers for Disease Control and Prevention FHT (1.0 Beta Version for Research Studies); and independent review by three genetic counselors (GCs) with expertise in CRC risk assessment. The GCs were blinded to which pedigrees had been modified to alter risk, the risk assessments made by the MFHP tool, the FHT, each other’s risk assessments, and proband self-reported body-mass index and race. The GCs were instructed that they should categorize the proband’s CRC risk as “weak,” “moderate,” or “strong” according to the criteria outlined by Yoon et al.8 This study was approved by the National Human Genome Research Institute Institutional Review Board.

Statistical analysis

Review of pedigrees by a panel of three expert cancer GCs was defined as the gold standard for risk of CRC in the test pedigrees.16 Risk estimates of “weak” by the FHT and GC review were considered “not elevated,” whereas “moderate” and “high” risk assignments were considered to be “elevated.” Pedigrees were scored as showing “elevated” risk for CRC only when at least two of three counselors reported risk levels as “moderate” or “strong” (true positives). The κ statistics were generated to assess GC interrater agreement for CRC assessment using the “weak,” “moderate,” and “strong” risk estimates for all 190 pedigrees. For the calculations of sensitivity, specificity, positive predictive values (PPVs), negative predictive values (NPVs), and κ statistics for the presence of CRC risk for the MFHP tool and the FHT versus the GC review, only the 150 unmodified pedigrees were used.

Results

GC risk estimates for CRC in the ClinSeq pedigrees

Sixteen of the 150 (11%) unmodified pedigrees were identified as having “elevated” risk by at least two of three GCs rating risk as “moderate” or “strong”; 6 of 150 (4%) were identified as having “strong” risk by at least two of three counselors. For the modified internal control pedigrees, the three counselors agreed for 10 of 10 (100%) and 6 of 10 (60%) pedigrees intended to be at “strong” or “moderate” risk, respectively. Pair-wise κ statistics for the three counselors calculated for all 190 pedigrees were 0.760, 0.776, and 0.851, indicating substantial agreement between the GCs according to the Landis–Koch guidelines.17

Comparison of the MFHP tool versus expert GC risk estimates

Twenty-seven of the 150 (18%) unmodified pedigrees were assigned “elevated” risk status by the MFHP tool. The MFHP tool had sensitivity and specificity of 81% (95% confidence interval (CI): 54–96%) and 90% (95% CI: 83–94%) for pedigrees judged to be “elevated” by GC review. The PPV and NPV were 48% (95% CI: 29–68%) and 98% (95% CI: 93–99%), respectively ( Figure 1 ). All 20 of the modified pedigrees intended to be at “strong” or “moderate” risk were identified as being at “elevated” risk by the MFHP tool. The κ statistic comparing the MFHP tool and GC review suggested moderate agreement (0.54).

Figure 1
figure 1

Diagram showing the process used for developing interpretations of 150 unmodified pedigrees derived from the ClinSeq population (see text) and resulting calculated values for sensitivity, specificity, PPV, NPV, and calculated κ for the My Family Health Portrait colorectal risk algorithm for detection of elevated colorectal cancer risk using expert genetic counselor pedigree interpretation as the “gold standard” for risk interpretation. CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value; TN, true negative; TP, true positive.

Full size image

Comparison of the FHT versus GC risk estimates

Fourteen of 150 (9%) of the unmodified pedigrees were considered to be at “elevated” risk by the FHT; of these 14 pedigrees, 3 of 150 (6%) were classified as being at “strong” risk and 11 of 150 (7%) were rated to be at “moderate” risk. The FHT had sensitivity and specificity of 75% (95% CI: 48–93%) and 99% (95% CI: 95–100%), respectively, as compared with pedigrees judged to be “elevated” by GC review. The PPV and NPV for FHT were 86% (95% CI: 57–98%) and 97% (95% CI: 93–99%), respectively. The κ statistic comparing the FHT with GC review indicated substantial agreement (0.78). For the modified control pedigrees, there was agreement for 10 of 10 (100%) between the FHT and pedigrees intended to be at “strong” risk. Eight of 10 (80%) of the modified pedigrees intended to be at “moderate” risk were identified as such by the FHT; for 2 of 10 (20%) the FHT rated the risk as “strong” instead of “moderate.”

Assessment of disagreement between GC risk estimates and the MFHP

There were three unmodified pedigrees that were identified as being at “elevated” risk by GC review that were rated “not elevated” risk by the MFHP tool (“false negatives”) and 14 pedigrees for which the MFHP tool rated risk as “elevated” and GC review rated risk as “not elevated” (“false positives”). The sources of the discrepancies were diverse ( Table 1 ). The “false-negative” pedigrees shared the characteristic of a lack of CRC cases or colorectal polyps in the proband or relatives. For one of the pedigrees, the error source was a failure to enter endometrial cancer as a structured term due to misspelling. The “false-positive” pedigrees often appeared to arise from differing assignment of risk arising from personal or family histories of colon polyps or a history of CRC or CRC-associated cancers in second-degree relatives.

Table 1 Potential sources of heritable risk for CRC present in pedigrees yielding “false-negative” and “false-positive” results with the MFHP in comparison with a gold standard of pedigree review by three expert GCs
Full size table

Discussion

This study provides preliminary evidence supporting the analytic validity of an automated, consumer-oriented screening algorithm for heritable CRC risk assessment that is compatible with the US Surgeon General’s MFHP. To our knowledge, it is the first study to formally evaluate the predictive value of an FH risk algorithm designed specifically to work with the MFHP tool. It is important to note that the “gold standard” in this study is not development of disease but rather the risk prediction arrived at by a group of three expert cancer GCs. The PPV values for the MFHP and the FHT exceed the 10% threshold that is considered an acceptable value for PPV for a genomic screening test.18 Because only a fraction of individuals with elevated CRC will actually develop disease, the PPV regarding development of clinical disease is likely to be lower than the values observed in this study, and the NPV is likely to be higher, for both the MFHP and FHT tools.

The MFHP algorithm was not designed to be diagnostic of CRC syndromes; rather, it was designed to be a potential component of an educational tool that helps individuals and their health-care providers stratify risk for heritable CRC in order to have an informed conversation regarding further evaluation. As a screening tool intended for use by the general public, it was designed to have a higher sensitivity at the cost of specificity, based on the rationale that missing elevated risk of colon cancer incurs a greater penalty than encouraging additional patients to discuss their risk of CRC with a health-care provider. This choice could be debated because available evidence suggests FH is not a strong predictor of absolute risk for CRC in the population.19

Evaluation of the pedigrees for which the risk ratings were discordant between GC review and the MFHP tool suggests that the MFHP tool assigns “elevated” risk to pedigrees with second-degree relatives having CRC or multiple Lynch syndrome–related cancers, or a personal history or FH of colon polyps. The GCs might not attribute much risk to histories of colon polyps lacking details of pathology. The FHT also does not fully incorporate personal and FH of colon polyps when assigning risk. Assuming such polyps are benign may underestimate risk, and at least one counselor desired additional information.20 Two of three instances for which the MFHP tool did not assign elevated risk represent somewhat gray areas of risk assessment for CRC. The lack of clarity in guidelines for such gray areas is reflected in the inconsistency in GC risk assignment; for two of three of the “false-negative” pedigrees, at least one expert GC rated the risk as “weak” and another rated the risk as “strong.”

This study has a number of important limitations. Measured internal agreement between the counselors suggests that expert GCs do not always concur when independently assigning CRC risk to pedigrees. The discrepancies between risk estimates provided by GC review and the MFHP algorithm highlight that, currently, there is no universally accepted “gold standard” for the detection of CRC risk in FHs in routine clinical care. Similar to the 2005 study by Qureshi et al.16, this study uses review by expert GCs to arrive at a proxy gold standard for CRC risk. Additional limitations of this study include the small sample size, the nature of the ClinSeq population (which may not represent the full diversity of the US population), and that the study was only able to examine the analytic validity (and not the clinical utility) of the MFHP CRC algorithm. Future studies should seek to replicate these findings and define the clinical validity and clinical utility of the MFHP tool in additional patient populations.

Disclosure

The authors declare no conflict of interest.