Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

Overestimation of the classification accuracy of a biomarker for assessing heavy alcohol use

Molecular Psychiatry volume 23, pages 2114–2115 (2018)Cite this article

Reports of alcohol intake by subjects or other informants may be unreliable, which limits the ability to identify individuals at risk for an alcohol use disorder or related health problems such as liver disease, heart disease and cancer. To improve the assessment of alcohol intake/consumption, Liu et al.1 proposed a blood biomarker based on methylation information from 144 CpG sites. The 144 CpGs were identified by screening a set of ~440 000 CpGs in a discovery sample of 6,926 subjects. To evaluate the ability of these CpGs to identify heavy alcohol drinking, they calculated the area under the curve (AUC) in four replication cohorts of 920–2003 subjects. The 144 CpGs in combination with age, sex and body mass index (BMI) produced AUCs that ranged from 0.90–0.99 for heavy drinkers versus nondrinkers and 0.85–0.99 for heavy versus light drinkers. The AUC is a measure of discriminatory power with a random classifier having an AUC of 0.5 and a perfect classifier an AUC of 1. Thus, their results suggested excellent to almost perfect classification accuracy and the authors concluded that the 144 CpGs performed better than commonly used clinical variables and biomarkers in discriminating current heavy alcohol drinking.

We have a methodological concern related to the estimation of the AUC in the replication cohorts. Although these cohorts were not used to select the 144 CpGs, the authors did not use the coefficients from the discovery set to determine classification accuracy. Instead, they re-estimated these coefficients by fitting a logistic regression model in each replication cohort. This carries the risk of overestimating the AUC because these cohorts can no longer yield a truly independent assessment of the classification accuracy. This risk increases with the number of CpGs, as models with more variables are more likely to capitalize on idiosyncrasies of individual data sets, thereby resulting in 'overfitting'.2

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1

References

  1. Liu C, Marioni RE, Hedman AK, Pfeiffer L, Tsai PC, Reynolds LM et al. A DNA methylation biomarker of alcohol consumption. Mol Psychiatry 2016 e-pub ahead of print15 November 2016; doi: https://doi.org/10.1038/mp.2016.192.

    Article  Google Scholar 

  2. Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Verlag: New York, NY, USA, 2001.

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for Biomarker Research and Precision Medicine, Virginia Commonwealth University, Richmond, VA, USA

    M W Hattab, S L Clark & E J C G van den Oord

Authors
  1. M W Hattab
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S L Clark
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. E J C G van den Oord
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to E J C G van den Oord.

Ethics declarations

Conflict of Interest

The authors declare no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hattab, M.W., Clark, S.L. & van den Oord, E.J.C.G. Overestimation of the classification accuracy of a biomarker for assessing heavy alcohol use. Mol Psychiatry 23, 2114–2115 (2018). https://doi.org/10.1038/mp.2017.181

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/mp.2017.181

This article is cited by

Search

Advanced search

Quick links