Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review
  • Published:

Assessing causal relationships between treatments and clinical outcomes: always read the fine print

Bone Marrow Transplantation volume 47, pages 626–632 (2012)Cite this article

Abstract

Changes in clinical practice should be driven by relevant and reliable evidence. Hence, adoption of a new therapy requires demonstrating that it provides (causes) benefit. Such evidence is generally obtained from intent-to-treat analyses of randomized clinical trials (RCTs). In this paper, we review other approaches to assessing the causal relationship between treatments and outcomes: (1) inference from non-randomized (observational) studies, (2) analysis of randomized studies where patients received treatments other than those to which they were randomized and (3) analysis of studies where the outcome of interest is sometimes unobservable because of a competing event (competing risks). We conclude that for the practice-changing demonstration of a favorable benefit-to-risk ratio, the gold standard is the intent-to-treat analysis of RCTs. At the same time, we illustrate how careful application of special statistical methods for assessment of treatment–outcome causation can be instrumental in complementing existing randomized evidence and guiding design of future research.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

References

  1. Johnson JR, Bross P, Cohen M, Rothmann M, Chen G, Zajicek A et al. Approval summary: imatinib mesylate capsules for treatment of adult patients with newly diagnosed Philadelphia chromosome-positive chronic myelogenous leukemia in chronic phase. Clin Cancer Res 2003; 9: 1972–1979.

    CAS  PubMed  Google Scholar 

  2. Korn EL . Comment: causal inference in medical area. Stat Sci 2006; 21: 310–312.

    Article  Google Scholar 

  3. Korn EL, Freidlin B . Causal inference for definitive clinical end points in a randomized clinical trial with intervening nonrandomized treatments. J Clin Oncol 2010; 28: 3800–3802.

    Article  CAS  Google Scholar 

  4. Green SB, Byar DP . Using observational data from registries to compare treatments: the fallacy of omnimetrics. Stat Med 1984; 3: 361–373.

    Article  CAS  Google Scholar 

  5. Byar DP, Simon RM, Friedewald WT, Schlesselman JJ, DeMets DL, Ellenberg JH et al. Randomized clinical trials. Perspectives on some recent ideas. N Engl J Med 1976; 295: 74–80.

    Article  CAS  Google Scholar 

  6. Canellos GP . Selection bias in trials of transplantation for metastatic breast cancer: have we picked the apple before it was ripe? J Clin Oncol 1997; 15: 3169–3170.

    Article  CAS  Google Scholar 

  7. MacNeil M, Eisenhauer EA . High-dose chemotherapy: is it standard management for any common solid tumor? Ann Oncol 1999; 10: 1145–1161.

    Article  CAS  Google Scholar 

  8. Mayer M . Listen to all the voices: an advocate's perspective on early access to investigational therapies. Clin Trials 2006; 3: 149–153.

    Article  Google Scholar 

  9. Mello MM, Brennan TA . The controversy over high-dose chemotherapy with autologous bone marrow transplant for breast cancer. Health Aff (Millwood) 2001; 20: 101–117.

    Article  CAS  Google Scholar 

  10. Antman KH, Souhami RL . High-dose chemotherapy in solid tumours. A review of published data in selected tumours with a commentary. Ann Oncol 1993; 4 (Suppl 1): 29–44.

    Article  Google Scholar 

  11. Crump M, Goss PE, Prince M, Girouard C . Outcome of extensive evaluation before adjuvant therapy in women with breast cancer and 10 or more positive axillary lymph nodes. J Clin Oncol 1996; 14: 66–69.

    Article  CAS  Google Scholar 

  12. Rahman ZU, Frye DK, Buzdar AU, Smith TL, Asmar L, Champlin RE et al. Impact of selection process on response rate and long-term survival of potential high-dose chemotherapy candidates treated with standard-dose doxorubicin-containing chemotherapy in patients with metastatic breast cancer. J Clin Oncol 1997; 15: 3171–3177.

    Article  CAS  Google Scholar 

  13. García-Carbonero R, Hidalgo M, Paz-Ares L, Calzas J, Gómez H, Guerra JA et al. Patient selection in high-dose chemotherapy trials: relevance in high-risk breast cancer. J Clin Oncol 1997; 15: 3178–3184.

    Article  Google Scholar 

  14. Zujewski J, Nelson A, Abrams J . Much ado about not enough data: high-dose chemotherapy with autologous stem cell rescue for breast cancer. J Natl Cancer Inst 1998; 90: 200–209.

    Article  CAS  Google Scholar 

  15. Antman KH, Rowlings PA, Vaughan WP, Pelz CJ, Fay JW, Fields KK et al. High-dose chemotherapy with autologous hematopoietic stem-cell support for breast cancer in North America. Clin Oncol 1997; 15: 1870–1879.

    Article  CAS  Google Scholar 

  16. Chabot JA, Tsai WY, Fine R, Chen C, Kumah CK, Antman KA et al. Pancreatic proteolytic enzyme therapy compared with gemcitabine-based chemotherapy for the treatment of pancreatic cancer. J Clin Oncol 2010; 28: 2058–2063.

    Article  CAS  Google Scholar 

  17. Levine MN . Conventional and complementary therapies: a tale of two research standards? J Clin Oncol 2010; 28: 1979–1981.

    Article  CAS  Google Scholar 

  18. Grothey A, Sugrue MM, Purdie DM, Dong W, Sargent D, Hedrick E et al. Bevacizumab beyond first progression is associated with prolonged overall survival in metastatic colorectal cancer: results from a large observational cohort study (BRiTE). J Clin Oncol 2008; 26: 5326–5334.

    Article  CAS  Google Scholar 

  19. Kazazi-Hyseni F, Beijnen JH, Schellens JH . Bevacizumab. Oncologist 2010; 15: 819–825.

    Article  CAS  Google Scholar 

  20. Giantonio BJ, Catalano PJ, Meropol NJ, O’Dwyer PJ, Mitchell EP, Alberts SR et al. Bevacizumab in combination with oxaliplatin, fluorouracil, and leucovorin (FOLFOX4) for previously treated metastatic colorectal cancer: results from the Eastern Cooperative Oncology Group Study E3200. J Clin Oncol 2007; 25: 1539–1544.

    Article  CAS  Google Scholar 

  21. Korn EL, Freidlin B, Abrams JS . Overall survival as the outcome for randomized clinical trials with effective subsequent therapies. J Clin Oncol 2011 (e-pub ahead of print 9 May 2011; doi:10.1200/JCO.2011.34.6056).

    Article  Google Scholar 

  22. Van Cutsem E, Peeters M, Siena S, Humblet Y, Hendlisz A, Neyns B et al. Open-label phase III trial of panitumumab plus best supportive care compared with best supportive care alone in patients with chemotherapy-refractory metastatic colorectal cancer. J Clin Oncol 2007; 25: 1658–1664.

    Article  CAS  Google Scholar 

  23. Motzer RJ, Escudier B, Oudard S, Hutson TE, Porta C, Bracarda S et al. Efficacy of everolimus in advanced renal cell carcinoma: a double-blind, randomised, placebo-controlled phase III trial. Lancet 2008; 372: 449–456.

    Article  CAS  Google Scholar 

  24. Sobrero AF, Maurel J, Fehrenbacher L, Scheithauer W, Abubakr YA, Lutz MP et al. EPIC: phase III trial of cetuximab plus irinotecan after fluoropyrimidine and oxaliplatin failure in patients with metastatic colorectal cancer. J Clin Oncol 2008; 26: 2311–2319.

    Article  CAS  Google Scholar 

  25. Sternberg CN, Davis ID, Mardiak J, Szczylik C, Lee E, Wagstaff J et al. Pazopanib in locally advanced or metastatic renal cell carcinoma: results of a randomized phase III trial. J Clin Oncol 2010; 28: 1061–1068.

    Article  CAS  Google Scholar 

  26. Yao JC, Shah MH, Ito T, Bohas CL, Wolin EM, Van Cutsem E et al. Everolimus for advanced pancreatic neuroendocrine tumors. N Engl J Med 2011; 364: 514–523.

    Article  CAS  Google Scholar 

  27. Escudier B, Eisen T, Stadler WM, Szczylik C, Oudard S, Siebels M et al. Sorafenib in advanced clear-cell renal-cell carcinoma. N Engl J Med 2007; 356: 125–134.

    Article  CAS  Google Scholar 

  28. Escudier B, Eisen T, Stadler WM, Szczylik C, Oudard S, Staehler M et al. Sorafenib for treatment of renal cell carcinoma: final efficacy and safety results of the phase III treatment approaches in renal cancer global evaluation trial. J Clin Oncol 2009; 27: 3312–3318.

    Article  CAS  Google Scholar 

  29. Dodd LE, Korn EL, Freidlin B, Jaffe CC, Rubinstein LV, Dancey J et al. Blinded independent central review of progression-free survival in phase III clinical trials: important design element or unnecessary expense? J Clin Oncol 2008; 26: 3791–3796.

    Article  Google Scholar 

  30. Robbins JM, Tsiatis AA . Correcting for non-compliance in randomized trials using rank preserving structural failure time models. Commun Stat Theory Meth 1991; 20: 2609–2631.

    Article  Google Scholar 

  31. Branson M, Whitehead J . Estimating a treatment effect in survival studies in which patients switch treatment. Stat Med 2002; 21: 2449–2463.

    Article  Google Scholar 

  32. Yamaguchi T, Ohashi Y . Adjusting for differential proportions of second-line treatment in cancer clinical trials. Part II. An application in a clinical trial or unresectable non-small-cell lung cancer. Stat Med 2004; 23: 2005–2022.

    Article  Google Scholar 

  33. National Institute for Health and Clinical Excellence. Sunitinib for the treatment of gastrointestinal stromal tumours. National Institute for Health and Clinical Excellence technology appraisal document 179. http://guidance.nice.org.uk/TA179.

  34. Robins JM, Finkelstein DM . Correcting for noncompliance and dependent censoring in an AIDS Clinical Trial with inverse probability of censoring weighted (IPCW) log-rank tests. Biometrics 2000; 56: 779–788.

    Article  CAS  Google Scholar 

  35. BIG 1-98 Collaborative Group. Letrozole therapy alone or in sequence with tamoxifen in women with breast cancer. N Engl J Med 2009; 361: 766–776.

    Article  Google Scholar 

  36. Colleoni M, Giobbie-Hurder A, Regan MM, Thürlimann B, Mouridsen H, Mauriac L et al. Analyses adjusting for selective crossover show improved overall survival with adjuvant letrozole compared with tamoxifen in the BIG 1-98 Study. Clin Oncol 2011; 29: 1117–1124.

    Article  CAS  Google Scholar 

  37. Finkelstein DM, Schoenfeld DA . Correcting for discretionary treatment crossover in an analysis of survival in the Breast International Group BIG 1-98 Trial by using the inverse probability of censoring weighted method. J Clin Oncol 2011; 29: 1093–1095.

    Article  Google Scholar 

  38. London WB, Frantz CN, Campbell LA, Seeger RC, Brumback BA, Cohn SL et al. Phase II randomized comparison of topotecan plus cyclophosphamide versus topotecan alone in children with recurrent or refractory neuroblastoma: a Children's Oncology Group study. J Clin Oncol 2010; 28: 3808–3815.

    Article  CAS  Google Scholar 

  39. Klein JP, Rizzo JD, Zhang MJ, Keiding N . Statistical methods for the analysis and presentation of the results of bone marrow transplants. Part I: unadjusted analysis. Bone Marrow Transplant 2001; 28: 909–915.

    Article  CAS  Google Scholar 

  40. Korn EL, Dorey FJ . Applications of crude incidence curves. Stat Med 1992; 11: 813–829.

    Article  CAS  Google Scholar 

  41. Scrucca L, Santucci A, Aversa F . Competing risk analysis using R: an easy guide for clinicians. Bone Marrow Transplant 2007; 40: 381–387.

    Article  CAS  Google Scholar 

  42. Kim HT . Cumulative incidence in competing risks data and competing risks regression analysis. Clin Cancer Res 2007; 13: 559–565.

    Article  Google Scholar 

  43. Freidlin B, Korn EL . Testing treatment effects in the presence of competing risks. Stat Med 2005; 24: 1703–1712.

    Article  Google Scholar 

  44. Alyea EP, Kim HT, Ho V, Cutler C, Gribben J, DeAngelo DJ et al. Comparative outcome of nonmyeloablative and myeloablative allogeneic hematopoietic cell transplantation for patients older than 50 years of age. Blood 2005; 105: 1810–1814.

    Article  CAS  Google Scholar 

  45. Concato J, Shah N, Horwitz RI . Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med 2000; 342: 1887–1892.

    Article  CAS  Google Scholar 

  46. Benson K, Hartz AJ . A comparison of observational studies and randomized, controlled trials. N Engl J Med 2000; 342: 1878–1886.

    Article  CAS  Google Scholar 

  47. Pocock SJ, Elbourne DR . Randomized trials or observational tribulations? N Engl J Med 2000; 342: 1907–1909.

    Article  CAS  Google Scholar 

  48. Wieand S, Murhy K . A commentary on treatment at random: the ultimate science or the betrayal of Hippocrates? J Clin Oncol 2004; 24: 5009–5011.

    Article  Google Scholar 

  49. Gale RP, Eapen M, Logan B, Zhang MJ, Lazarus HM . Are there roles for observational database studies and structured quantification of expert opinion to answer therapy controversies in transplants? Bone Marrow Transplant 2009; 43: 435–446.

    Article  CAS  Google Scholar 

  50. Stampfer MJ, Colditz GA . Estrogen replacement therapy and coronary heart disease: a quantitative assessment of the epidemiologic evidence. Prev Med 1991; 20: 47–63.

    Article  CAS  Google Scholar 

  51. Manson JE, Hsia J, Johnson KC, Rossouw JE, Assaf AR, Lasser NL et al. Estrogen plus progestin and the risk of coronary heart disease. N Engl J Med 2003; 349: 523–534.

    Article  CAS  Google Scholar 

  52. International Agency for Research of Cancer IARC. Handbook of Cancer Prevention 2 Carotenoids. International Agency for Research of Cancer: Lyon, 1998, 64–103.

  53. The Alpha-Tocopherol, Beta Carotene Cancer Prevention Study Group. The effect of vitamin E and beta carotene on the incidence of lung cancer and other cancers in male smokers. N Engl J Med 1994; 330: 1029–1035.

    Article  Google Scholar 

  54. Omenn GS, Goodman GE, Thornquist MD, Balmes J, Cullen MR, Glass A et al. Effects of a combination of beta carotene and vitamin A on lung cancer and cardiovascular disease. N Engl J Med 1996; 334: 1150–1155.

    Article  CAS  Google Scholar 

  55. Yusuf S, Dagenais G, Pogue J, Bosch J, Sleight P . Vitamin E supplementation and cardiovascular events in high-risk patients. The Heart Outcomes Prevention Evaluation Study Investigators. N Engl J Med 2000; 342: 154–160.

    Article  CAS  Google Scholar 

  56. Heart Protection Study Collaborative Group. MRC/BHF Heart Protection Study of antioxidant vitamin supplementation in 20,536 high-risk individuals: a randomised placebo-controlled trial. Lancet 2002; 360: 23–33.

    Article  Google Scholar 

  57. Stampfer MJ, Hennekens CH, Manson JE, Colditz GA, Rosner B, Willett WC . Vitamin E consumption and the risk of coronary disease in women. N Engl J Med 1993; 328: 1444–1449.

    Article  CAS  Google Scholar 

  58. Rimm EB, Stampfer MJ, Ascherio A, Giovannucci E, Colditz GA, Willett WC . Vitamin E consumption and the risk of coronary heart disease in men. N Engl J Med 1993; 328: 1450–1456.

    Article  CAS  Google Scholar 

  59. Moseley JB, O’Malley K, Petersen NJ, Menke TJ, Brody BA, Kuykendall DH et al. A controlled trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med 2002; 347: 81–88.

    Article  Google Scholar 

  60. Kirkley A, Birmingham TB, Litchfield RB, Giffin JR, Willits KR, Wong CJ et al. A randomized trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med 2008; 359: 1097–1107.

    Article  CAS  Google Scholar 

  61. Heart Protection Study Collaborative Group. MRC/BHF Heart Protection Study of antioxidant vitamin supplementation in 20 536 high-risk individuals: a randomised placebo-controlled trial. Lancet 2002; 360: 23–33.

    Article  Google Scholar 

  62. Khaw KT, Bingham S, Welch A, Luben R, Wareham N, Oakes S et al. Relation between plasma ascorbic acid and mortality in men and women in EPIC-Norfolk prospective study: a prospective population study. European Prospective Investigation into Cancer and Nutrition. Lancet 2001; 357: 657–663.

    Article  CAS  Google Scholar 

  63. Hébert PC, Wells G, Blajchman MA, Marshall J, Martin C, Pagliarello G et al. A multicenter, randomized, controlled clinical trial of transfusion requirements in critical care. Transfusion Requirements in Critical Care Investigators, Canadian Critical Care Trials Group. N Engl J Med 1999; 340: 409–417.

    Article  Google Scholar 

  64. Hébert PC, Schweitzer I, Calder L, Blajchman M, Giulivi A . Review of clinical practice on allogeneic red blood cell transfusion. Can Med Assoc J 1997; 156 (11S): S9–S26.

    Google Scholar 

  65. Eknoyan G, Beck GJ, Cheung AK, Daugirdas JT, Greene T, Kusek JW et al. Effect of dialysis dose and membrane flux in maintenance hemodialysis. N Engl J Med 2002; 347: 2010–2019.

    Article  Google Scholar 

  66. Greene T . Randomized and observational studies in nephrology: how strong is the evidence? Am J Kidney Dis 2009; 53: 377–388.

    Article  Google Scholar 

  67. The EC/IC Bypass Study Group. Failure of extracranial-intracranial arterial bypass to reduce the risk of ischemic stroke. Results of an international randomized trial. N Engl J Med 1985; 313: 1191–1200.

    Article  Google Scholar 

  68. Weinstein PR, Rodriguez y Baena R, Chater NL . Results of extracranial-intracranial arterial bypass for intracranial internal carotid artery stenosis: review of 105 cases. Neurosurgery 1984; 15: 787–794.

    CAS  PubMed  Google Scholar 

  69. Fisher RI, Gaynor ER, Dahlberg S, Oken MM, Grogan TM, Mize EM et al. Comparison of a standard regimen (CHOP) with three intensive chemotherapy regimens for advanced non-Hodgkin's lymphoma. N Engl J Med 1993; 328: 1002–1006.

    Article  CAS  Google Scholar 

  70. Liu PY, Anderson G, Crowley JJ . Observational studies and randomized trials. N Engl J Med 2000; 343: 1195.

    CAS  PubMed  Google Scholar 

  71. Sacks H, Chalmers TC, Smith Jr H . Randomized versus historical controls for clinical trials. Am J Med 1982; 72: 233–240.

    Article  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Division of Cancer Treatment and Diagnosis, Biometric Research Branch, National Cancer Institute, Bethesda, MD, USA

    B Freidlin & E L Korn

Authors
  1. B Freidlin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. E L Korn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to B Freidlin.

Ethics declarations

Competing interests

The authors declare no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Freidlin, B., Korn, E. Assessing causal relationships between treatments and clinical outcomes: always read the fine print. Bone Marrow Transplant 47, 626–632 (2012). https://doi.org/10.1038/bmt.2011.119

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/bmt.2011.119

Keywords

Search

Advanced search

Quick links