Abstract
Artificial intelligence techniques, specifically deep learning, have already affected daily life in a wide range of areas. Likewise, initial applications have been explored in rheumatology. Deep learning might not easily surpass the accuracy of classic techniques when performing classification or regression on low-dimensional numerical data. With images as input, however, deep learning has become so successful that it has already outperformed the majority of conventional image-processing techniques developed during the past 50 years. As with any new imaging technology, rheumatologists and radiologists need to consider adapting their arsenal of diagnostic, prognostic and monitoring tools, and even their clinical role and collaborations. This adaptation requires a basic understanding of the technical background of deep learning, to efficiently utilize its benefits but also to recognize its drawbacks and pitfalls, as blindly relying on deep learning might be at odds with its capabilities. To facilitate such an understanding, it is necessary to provide an overview of deep-learning techniques for automatic image analysis in detecting, quantifying, predicting and monitoring rheumatic diseases, and of currently published deep-learning applications in radiological imaging for rheumatology, with critical assessment of possible limitations, errors and confounders, and conceivable consequences for rheumatologists and radiologists in clinical practice.
Key points
-
The number of research studies on deep learning in rheumatological imaging has grown rapidly during the past 5 years, but they mainly consist of pilot studies that require external validation.
-
Confounding factors and errors in deep-learning methods need to be ruled out before deep learning can be applied in clinical practice, for which the intended use should be strictly defined.
-
Deep-learning techniques, together with mapping to explain their reasoning, will enable hypothesis-free image analysis and could identify new imaging biomarkers.
-
Deep learning might assist rheumatologists and radiologists in interpreting rheumatological images, increasing their diagnostic, prognostic and monitoring accuracy, and decreasing workloads and costs.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Kingsmore, K. M., Puglisi, C. E., Grammer, A. C. & Lipsky, P. E. An introduction to machine learning and analysis of its use in rheumatic diseases. Nat. Rev. Rheumatol. 17, 710–730 (2021).
Christodoulou, E. et al. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J. Clin. Epidemiol. 110, 12–22 (2019).
Calivà, F. et al. Studying osteoarthritis with artificial intelligence applied to magnetic resonance imaging. Nat. Rev. Rheumatol. 18, 112–121 (2022).
Cipolletta, E. et al. Artificial intelligence for ultrasound informative image selection of metacarpal head cartilage. a pilot study. Front. Med. 8, 589197 (2021).
Prasoon, A. et al. Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. Med. Image Comput. Comput. Assist. Interv. 16, 246–253 (2013).
Banerjee, S., Bhunia, S. & Schaefer, G. Osteophyte detection for hand osteoarthritis identification in X-ray images using CNNs. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2011, 6196–6199 (2011).
Folle, L. et al. Advanced neural networks for classification of MRI in psoriatic arthritis, seronegative, and seropositive rheumatoid arthritis. Rheumatology 61, 4945–4951 (2022).
Hassanzadeh, T. et al. AB0205 RA treatment effects in wrist MRIs, determined by deep learning. Ann. Rheum. Dis. 82, 1286 (2023).
Abedin, J. et al. Predicting knee osteoarthritis severity: comparative modeling based on patient’s data and plain X-ray images. Sci. Rep. 9, 5761 (2019).
Leung, K. et al. Prediction of total knee replacement and diagnosis of osteoarthritis by using deep learning on knee radiographs: data from the osteoarthritis initiative. Radiology 296, 584–593 (2020).
Jia, J. et al. Automatic pulmonary function estimation from chest CT scans using deep regression neural networks: the relation between structure and function in systemic sclerosis. IEEE Access 11, 135272–135282 (2023).
Chang, G. H. et al. Assessment of knee pain from MR imaging using a convolutional Siamese network. Eur. Radiol. 30, 3538–3548 (2020).
Ras, G., Xie, N., Gerven, M. V. & Doran, D. Explainable deep learning: a field guide for the uninitiated. J. Artif. Int. Res. 73, 68 (2022).
National Institutes of Health. The Osteoarthritis Initiative. NIMH Data Archive https://nda.nih.gov/oai (2023).
Chen, N. et al. A fully automatic target detection and quantification strategy based on object detection convolutional neural network YOLOv3 for one-step X-ray image grading. Anal. Methods 15, 164–170 (2023).
Chen, P., Gao, L., Shi, X., Allen, K. & Yang, L. Fully automatic knee osteoarthritis severity grading using deep neural networks with a novel ordinal loss. Comput. Med. Imaging Graph. 75, 84–92 (2019).
Liu, B., Luo, J. & Huang, H. Toward automatic quantification of knee osteoarthritis severity using improved Faster R-CNN. Int. J. Comput. Assist. Radiol. Surg. 15, 457–466 (2020).
Norman, B., Pedoia, V., Noworolski, A., Link, T. M. & Majumdar, S. Applying densely connected convolutional neural networks for staging osteoarthritis severity from plain radiographs. J. Digit. Imaging 32, 471–477 (2019).
Tiulpin, A., Thevenot, J., Rahtu, E., Lehenkari, P. & Saarakkala, S. Automatic knee osteoarthritis diagnosis from plain radiographs: a deep learning-based approach. Sci. Rep. 8, 1727 (2018).
Tolpadi, A. A., Lee, J. J., Pedoia, V. & Majumdar, S. Deep learning predicts total knee replacement from magnetic resonance images. Sci. Rep. 10, 6371 (2020).
Tiulpin, A. et al. Multimodal machine learning-based knee osteoarthritis progression prediction from plain radiographs and clinical data. Sci. Rep. 9, 20038 (2019).
Hirvasniemi, J. et al. The KNee OsteoArthritis Prediction (KNOAP2020) challenge: an image analysis challenge to predict incident symptomatic radiographic knee osteoarthritis from MRI and X-ray images. Osteoarthritis Cartilage 31, 115–125 (2023).
Jansen, M. P. et al. Artificial intelligence in osteoarthritis: repair by knee joint distraction shows association of pain, radiographic and immunological outcomes. Rheumatology 62, 2789–2796 (2022).
Ambellan, F., Tack, A., Ehlke, M. & Zachow, S. Automated segmentation of knee bone and cartilage combining statistical shape knowledge and convolutional neural networks: data from the Osteoarthritis Initiative. Med. Image Anal. 52, 109–118 (2019).
Cheng, R. et al. Fully automated patellofemoral MRI segmentation using holistically nested networks: implications for evaluating patellofemoral osteoarthritis, pain, injury, pathology, and adolescent development. Magn. Reson. Med. 83, 139–153 (2020).
Gaj, S., Yang, M., Nakamura, K. & Li, X. Automated cartilage and meniscus segmentation of knee MRI with conditional generative adversarial networks. Magn. Reson. Med. 84, 437–449 (2020).
Liu, F. et al. Deep convolutional neural network and 3D deformable approach for tissue segmentation in musculoskeletal magnetic resonance imaging. Magn. Reson. Med. 79, 2379–2391 (2018).
Norman, B., Pedoia, V. & Majumdar, S. Use of 2D U-net convolutional neural networks for automated cartilage and meniscus segmentation of knee MR imaging data to determine relaxometry and morphometry. Radiology 288, 177–185 (2018).
Panfilov, E., Tiulpin, A., Nieminen, M. T., Saarakkala, S. & Casula, V. Deep learning-based segmentation of knee MRI for fully automatic subregional morphological assessment of cartilage tissues: data from the Osteoarthritis Initiative. J. Orthop. Res. 40, 1113–1124 (2022).
Razmjoo, A. et al. T2 analysis of the entire osteoarthritis initiative dataset. J. Orthop. Res. 39, 74–85 (2021).
Chang, G. H. et al. Subchondral bone length in knee osteoarthritis: a deep learning-derived imaging measure and its association with radiographic and clinical outcomes. Arthritis Rheumatol. 73, 2240–2248 (2021).
Pedoia, V. et al. 3D convolutional neural networks for detection and severity staging of meniscus and PFJ cartilage morphological degenerative changes in osteoarthritis and anterior cruciate ligament subjects. J. Magn. Reson. Imaging 49, 400–410 (2019).
Liu, F. et al. Deep learning approach for evaluating knee MR images: achieving high diagnostic performance for cartilage lesion detection. Radiology 289, 160–169 (2018).
Astuto, B. et al. Automatic deep learning-assisted detection and grading of abnormalities in knee MRI studies. Radiol. Artif. Intell. 3, e200165 (2021).
Namiri, N. K. et al. Deep learning for large scale MRI-based morphological phenotyping of osteoarthritis. Sci. Rep. 11, 10915 (2021).
Brui, E. et al. Deep learning-based fully automatic segmentation of wrist cartilage in MR images. NMR Biomed. 33, e4320 (2020).
Üreten, K. et al. Detection of hip osteoarthritis by using plain pelvic radiographs with deep learning methods. Skeletal Radiol. 49, 1369–1374 (2020).
Xue, Y., Zhang, R., Deng, Y., Chen, K. & Jiang, T. A preliminary examination of the diagnostic value of deep learning in hip osteoarthritis. PLoS ONE 12, e0178992 (2017).
von Schacky, C. E. et al. Development and validation of a multitask deep learning model for severity grading of hip osteoarthritis features on radiographs. Radiology 295, 136–145 (2020).
Radke, K. L. et al. Adaptive IoU thresholding for improving small object detection: a proof-of-concept study of hand erosions classification of patients with rheumatic arthritis on X-ray images. Diagnostics 13, 104 (2022).
Murakami, S., Hatano, K., Tan, J., Kim, H. & Aoki, T. Automatic identification of bone erosions in rheumatoid arthritis from hand radiographs based on deep convolutional neural network. Multimed. Tools Appl. 77, 10921–10937 (2018).
Hirano, T. et al. Development and validation of a deep-learning model for scoring of radiographic finger joint destruction in rheumatoid arthritis. Rheumatol. Adv. Prac. 3, rkz047 (2019).
Chaturvedi, N. DeepRA: predicting joint damage from radiographs using CNN with attention. Preprint at https://doi.org/10.48550/arXiv.2102.06982 (2021).
Rohrbach, J., Reinhard, T., Sick, B. & Dürr, O. Bone erosion scoring for rheumatoid arthritis with deep convolutional neural networks. Comput, Electr. Eng. 78, 472–481 (2019).
Üreten, K., Erbay, H. & Maraş, H. H. Detection of rheumatoid arthritis from hand radiographs using a convolutional neural network. Clin. Rheumatol. 39, 969–974 (2020).
Izumi, K. et al. Detecting hand joint ankylosis and subluxation in radiographic images using deep learning: a step in the development of an automatic radiographic scoring system for joint destruction. PLoS ONE 18, e0281088 (2023).
Fiorentino, M. C., Moccia, S., Cipolletta, E., Filippucci, E. & Frontoni, E. A learning approach for informative-frame selection in US rheumatology images. in: Cristani, M., Prati, A., Lanz, O., Messelodi, S., Sebe, N. (eds) New Trends in Image Analysis and Processing — ICIAP 2019. ICIAP 2019. Lecture Notes in Computer Science, vol 11808. https://doi.org/10.1007/978-3-030-30754-7_23 (Springer, Cham, 2019).
Tang, J. et al. Enhancing convolutional neural network scheme for rheumatoid arthritis grading with limited clinical data. Chin. Phys. B 28, 038701 (2019).
Hemalatha, R. J., Vijaybaskar, V. & Thamizhvani, T. R. Automatic localization of anatomical regions in medical ultrasound images of rheumatoid arthritis using deep learning. Proc. Inst. Mech. Eng. H. 233, 657–667 (2019).
Christensen, A. B. H., Just, S. A., Andersen, J. K. H. & Savarimuthu, T. R. Applying cascaded convolutional neural network design further enhances automatic scoring of arthritis disease activity on ultrasound images from rheumatoid arthritis patients. Ann. Rheum. Dis. 79, 1189–1193 (2020).
Fiorentino, M. C. et al. A deep-learning framework for metacarpal-head cartilage-thickness estimation in ultrasound rheumatological images. Comput. Biol. Med. 141, 105117 (2022).
Folle, L. et al. Deep learning methods allow fully automated segmentation of metacarpal bones to quantify volumetric bone mineral density. Sci. Rep. 11, 9697 (2021).
Folle, L. et al. Deep learning-based classification of inflammatory arthritis by identification of joint shape patterns — how neural networks can tell us where to “Deep Dive” clinically. Front. Med. 9, 850552 (2022).
Wong, L. M., Shi, L., Xiao, F. & Griffith, J. F. Fully automated segmentation of wrist bones on T2-weighted fat-suppressed MR images in early rheumatoid arthritis. Quant. Imaging Med. Surg. 9, 579–589 (2019).
Shamonin, D. P. et al. POS0920 quantification of tenosynovitis from wrist MRIs, based on deep learning. Ann. Rheum. Dis. 82, 770–771 (2023).
Li, Y. et al. OP0002 exploring the use of artificial intelligence in predicting rheumatoid arthritis, based on extremity MR scans in early arthritis and clinically suspect arthralgia patients. Ann. Rheum. Dis. 82, 1–2 (2023).
Hassanzadeh, T. et al. A deep learning-based comparative MRI model to detect inflammatory changes in rheumatoid arthritis. Biomed. Signal. Process. Control. 88, 105612 (2024).
Hepburn, C. E. et al. Towards deep learning-assisted quantification of inflammation in spondyloarthritis: intensity-based lesion segmentation. Preprint at https://doi.org/10.48550/arXiv.2106.11343 (2021).
Lin, K. Y. Y., Peng, C., Lee, K. H., Chan, S. C. W. & Chung, H. Y. Deep learning algorithms for magnetic resonance imaging of inflammatory sacroiliitis in axial spondyloarthritis. Rheumatology 61, 4198–4206 (2022).
Han, Q. et al. Automatic quantification and grading of hip bone marrow oedema in ankylosing spondylitis based on deep learning. Mod. Rheumatol. 32, 968–973 (2022).
Bressem, K. K. et al. Deep learning detects changes indicative of axial spondyloarthritis at MRI of sacroiliac joints. Radiology 305, 655–665 (2022).
Lee, K. H., Choi, S. T., Lee, G. Y., Ha, Y. J. & Choi, S. I. Method for diagnosing the bone marrow edema of sacroiliac joint in patients with axial spondyloarthritis using magnetic resonance image analysis based on deep learning. Diagnostics 11, 1156 (2021).
Koo, B. S. et al. A pilot study on deep learning-based grading of corners of vertebral bodies for assessment of radiographic progression in patients with ankylosing spondylitis. Ther. Adv. Musculoskelet. Dis. 14, 1759720x221114097 (2022).
Bressem, K. K. et al. Deep learning for detection of radiographic sacroiliitis: achieving expert-level performance. Arthritis Res. Ther. 23, 106 (2021).
Üreten, K., Maraş, Y., Duran, S. & Gök, K. Deep learning methods in the diagnosis of sacroiliitis from plain pelvic radiographs. Mod. Rheumatol. 33, 202–206 (2023).
Grob, A. et al. External validation of the deep learning system “SpineNet” for grading radiological features of degeneration on MRIs of the lumbar spine. Eur. Spine J. 31, 2137–2148 (2022).
Smerilli, G. et al. Development of a convolutional neural network for the identification and the measurement of the median nerve on ultrasound images acquired at carpal tunnel level. Arthritis Res. Ther. 24, 38 (2022).
Fabry, V. et al. A deep learning tool without muscle-by-muscle grading to differentiate myositis from facio-scapulo-humeral dystrophy using MRI. Diagn. Interv. Imaging 103, 353–359 (2022).
Wang, F. et al. Assessment of idiopathic inflammatory myopathy using a deep learning method for muscle T2 mapping segmentation. Eur. Radiol. 33, 2350–2357 (2022).
Burlina, P., Billings, S., Joshi, N. & Albayda, J. Automated diagnosis of myositis from muscle ultrasound: exploring the use of machine learning and deep learning methods. PLoS ONE 12, e0184059 (2017).
Roncato, C. et al. Colour Doppler ultrasound of temporal arteries for the diagnosis of giant cell arteritis: a multicentre deep learning study. Clin. Exp. Rheumatol. 38, 120–125 (2020).
Garaiman, A. et al. Vision transformer assisting rheumatologists in screening for capillaroscopy changes in systemic sclerosis: an artificial intelligence model. Rheumatology 62, 2492–2500 (2022).
Gurunath Bharathi, P. et al. A deep learning system for quantitative assessment of microvascular abnormalities in nailfold capillary images. Rheumatology 62, 2325–2329 (2023).
Mohajer, B. et al. Role of thigh muscle changes in knee osteoarthritis outcomes: osteoarthritis initiative data. Radiology 305, 169–178 (2022).
Adebayo, J. et al. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems 31 (NeurIPS, 2018).
Dakkak, Y. J., Jansen, F. P., DeRuiter, M. C., Reijnierse, M. & van der Helm-van Mil, A. H. M. Rheumatoid arthritis and tenosynovitis at the metatarsophalangeal joints: an anatomic and MRI study of the forefoot tendon sheaths. Radiology 295, 146–154 (2020).
Maleki, F. et al. Generalizability of machine learning models: quantitative evaluation of three methodological pitfalls. Radiol. Artif. Intell. 5, e220028 (2023).
Zech, J. R. et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Med. 15, e1002683 (2018).
Geirhos, R. et al. Shortcut learning in deep neural networks. Nat. Mach. Intell. 2, 665–673 (2020).
Reinke, A. et al. Understanding metric-related pitfalls in image analysis validation. Preprint at https://doi.org/10.48550/arXiv.2302.01790 (2023).
Abdalla, M. & Fine, B. Hurdles to artificial intelligence deployment: noise in schemas and “Gold” labels. Radiol. Artif. Intell. 5, e220056 (2023).
Meszaros, J., Minari, J. & Huys, I. The future regulation of artificial intelligence systems in healthcare services and medical research in the European Union. Front. Genet. 13, 927721 (2022).
Pesapane, F., Volonté, C., Codari, M. & Sardanelli, F. Artificial intelligence as a medical device in radiology: ethical and regulatory issues in Europe and the United States. Insights Imaging 9, 745–753 (2018).
Mangnus, L., van Steenbergen, H. W., Reijnierse, M. & van der Helm-van Mil, A. H. Magnetic resonance imaging-detected features of inflammation and erosions in symptom-free persons from the general population. Arthritis Rheumatol. 68, 2593–2602 (2016).
Boer, A. C. et al. Using a reference when defining an abnormal MRI reduces false-positive MRI results-a longitudinal study in two cohorts at risk for rheumatoid arthritis. Rheumatology 56, 1700–1706 (2017).
Boeren, A. M. P. et al. Towards a simplified fluid-sensitive MRI protocol in small joints of the hand in early arthritis patients: reliability between modified Dixon and regular Gadolinium enhanced TSE fat saturated MRI-sequences. Skeletal Radiol. 52, 1193–1202 (2023).
Hassanzadeh, T. et al. A deep learning model to locate inflammatory changes in rheumatoid arthritis. Ann. Rheum. Dis. 82, 298–299 (2023).
Mongan, J., Moy, L. & Kahn, C. E. Jr Checklist for artificial intelligence in medical imaging (CLAIM): a guide for authors and reviewers. Radiol. Artif. Intell. 2, e200029 (2020).
Acknowledgements
The work presented in Fig. 5 has been funded by the Dutch Research Council (NWO) Applied and Engineering Sciences (project number 17970).
Author information
Authors and Affiliations
Contributions
B.C.S. researched data for the article. All authors contributed substantially to discussion of the content. B.C.S., M.S. and A.H.M.v.d.H.-v.M. wrote the article. All authors reviewed and/or edited the manuscript before submission.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Reviews Rheumatology thanks Reza Forghani and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Related links
CS230 Deep Learning tutorial: https://cs230.stanford.edu/
List of commercial AI products for radiology: https://grand-challenge.org/aiforradiology/
Machine Learning Glossary: https://developers.google.com/machine-learning/glossary
Glossary
- Cross-entropy
-
A measure of the difference between two probability distributions, where the one distribution is from the ground truth and the other from the model’s output. Used as a loss function, it penalizes errors especially when the model is confident, but wrong.
- Data augmentation
-
Artificially expanding the number and diversity of training examples by performing random transformations, or adding noise or simulated objects (such as lesions) to existing image data.
- Dice coefficient
-
A statistic that is used to quantify the similarity between two samples; in image segmentation, it measures the overlap between the ground truth and the model-produced segmentation3.
- Loss function
-
A mathematical entity for quantifying how well a machine-learning algorithm models the data. Higher values indicate poorer modelling ability. During training, the loss function is coupled with an optimizer, which is used to tune the parameters of the machine-learning or deep-learning algorithm to minimize the loss function and ultimately maximize algorithm performance3.
- Saliency maps
-
Derived images usually displayed as heat maps that show the locations in the input image that contributed most to the model’s output.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Stoel, B.C., Staring, M., Reijnierse, M. et al. Deep learning in rheumatological image interpretation. Nat Rev Rheumatol 20, 182–195 (2024). https://doi.org/10.1038/s41584-023-01074-5
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41584-023-01074-5