Abstract
The monitoring of physiological parameters is a crucial topic in promoting human health and an indispensable approach for assessing physiological status and diagnosing diseases. Particularly, it holds significant value for patients who require long-term monitoring or with underlying cardiovascular disease. To this end, Visual Contactless Physiological Monitoring (VCPM) is capable of using videos recorded by a consumer camera to monitor blood volume pulse (BVP) signal, heart rate (HR), respiratory rate (RR), oxygen saturation (SpO2) and blood pressure (BP). Recently, deep learning-based pipelines have attracted numerous scholars and achieved unprecedented development. Although VCPM is still an emerging digital medical technology and presents many challenges and opportunities, it has the potential to revolutionize clinical medicine, digital health, telemedicine as well as other areas. The VCPM technology presents a viable solution that can be integrated into these systems for measuring vital parameters during video consultation, owing to its merits of contactless measurement, cost-effectiveness, user-friendly passive monitoring and the sole requirement of an off-the-shelf camera. In fact, the studies of VCPM technologies have been rocketing recently, particularly AI-based approaches, but few are employed in clinical settings. Here we provide a comprehensive overview of the applications, challenges, and prospects of VCPM from the perspective of clinical settings and AI technologies for the first time. The thorough exploration and analysis of clinical scenarios will provide profound guidance for the research and development of VCPM technologies in clinical settings.
Similar content being viewed by others
Introduction
Visual Contactless Physiological Monitoring (VCPM) is an emerging technology that can measure the vital signs based on videos. It has been proven that VCPM is highly effective in monitoring blood volume pulse (BVP) signal, heart rate (HR), respiratory rate (RR), oxygen saturation (SpO2), and blood pressure (BP)1,2,3,4. More significantly, VCPM’s contactless characteristic offers clinical benefits such as user-friendliness, full automation, long-term monitoring, zero skin damage, improved clinical workflow efficiency and the greatly reduced risk of cross-infection. Particularly, VCPM can also play a critical role in combating cardiovascular diseases (CVDs) and offer full-cycle personal health management5,6.
As illustrated in Fig. 1, the basic physiological principles of VCPM are established on the cardiopulmonary and circulatory systems. As shown in Fig. 1a, the cardiopulmonary system facilitates the transportation of blood between the heart and lungs, whereas the blood moves from the aorta through the systemic arteries. In blood circulation theory, blood is ejected out of the heart and propagates along the arterial tree, and the BVP waveform takes on typical morphological components corresponding to landmark events (e.g., the contraction of left ventricle and the dicrotic notch) in the cardiac cycle7. Since blood flow is regulated by cardiac and respiratory interactions, it is theoretically possible to extract various physiological parameters through the analysis of a photoplethysmography (PPG) signal7.
Figure 1 illustrates remote PPG (rPPG) technology, approaches of physiological parameter measurement, and the solution of cardiopulmonary status assessment, which is broadly defined as VCPM technologies in this paper. As shown in Fig. 1b, owing to the fact that BVP waveform can be detected by the camera, rPPG technology is capable of extracting PPG signals from videos of the skin. Furthermore, PPG signals are employed to infer HR, RR, SpO2 and BP (Fig. 1c–e). SpO2 monitoring requires the measurement of at least two PPG wavelengths for SpO2 calibration. Additionally, BP can be measured by multi-site pulse transit time (PTT) (inferred from PPG waveforms from two distinct body sites), multi-wavelength PTT (from different skin layers), and morphological features. Moreover, in Fig. 1f, the vital signs can be employed to assess the wellness of cardiopulmonary system.
In 2000, Wu et al. proposed the first prototype of PPG imaging that uses an NIR light and black/white camera8. In 2007, researchers discovered that consumer RGB cameras can detect PPG waveforms in ambient light.9,10. After a decade, various camera-based rPPG algorithms, which were developed based on conventional computer vision and signal processing technology, have made a vigorous progress. Classic and popular algorithms include but are not limited to: CHROM11, PBV12, POS13, S2R14. In 2017, which can be called “the first year of deep learning for rPPG technology", scholars from University of Oxford and Taipei University of Science and Technology respectively presented their research achievements on newborn and adult subjects at the International Conference on Automatic Face and Gesture Recognition and the International Joint Conference on Biometric Recognition15,16. From 2021 to 2023, a variety of VCPM algorithms have emerged for the continuous monitoring of premature infants, babies, ICU patients, elderly people, etc17,18,19,20. Meanwhile, the number of the studies based on AI technologies with healthy subjects/laboratory environments has increased exponentially. Furthermore, PPG waveforms, derived from videos20,21,22, can be employed to infer HR23,24, SpO225,26, RR18,27, BP28,29 and disease analysis30,31,32.
The COVID-19 pandemic over the past three years (2020–2022) has expedited the revolution of digital medicine33,34,35,36,37. The utilization of telemedicine systems experienced an exponential growth in numerous countries in the Organization for Economic Co-operation and Development (OECD) throughout 202038. Compared to 2019, the number of Medicare fee-for-service beneficiary telehealth visits increased 63-fold in 2020, reaching nearly 52.7 million in the United States39. Similarly, in Germany, there were almost 1.4 million video consultations conducted during the first half of 202040. In the second quarter of 2020 alone, patients consulted with doctors or psychotherapists via video almost 1.2 million times40.
Most importantly, COVID-19 has changed the context of digital medicine, and promoted the development of telemedicine and Primary Health Care (PHC) system34,35,37,38,41. Governments paid increasing attention to digital medicine and telemedicine systems, and patients gradually accepted this treatment approach37,38,42. During the COVID-19 outbreak, we have noticed that many countries and regions were suffering from a shortage of essential vital-sign monitoring equipment, particularly blood oxygen level monitors. Given that blood saturation is a critical biomarker that can be utilized to infer the likelihood of being infected with the COVID-19 disease, patients under home quarantine can judge whether they are developing lung infections or experiencing severe illness by these parameters35,43,44,45. Because of the utilization of off-the-shelf devices such as webcams or smartphone cameras for measuring vital signals, VCPM technologies can potentially solve the aforementioned challenges of medical equipment shortages. Based on these factors, VCPM technologies offer a natural and cost-effective approach to establishing digital medicine or PHC systems.
VCPM technologies based on deep learning have made tremendous progress in recent years, but the majority of these studies are limited to laboratory settings or healthy subjects. To apply these technologies to clinical medicine, there is a large space of improvement. Therefore, the motivation of this review paper is to re-examine the application of VCPM technologies in clinical healthcare monitoring, summarize the encountered challenges and issues, and enhance the fundamental theory of VCPM in clinical settings. Moreover, the prospect of developing a VCPM algorithm based on the state-of-the-art (SOTA) artificial intelligence technologies is depicted. Overall, this review offers the guidelines for the future development of VCPM algorithms toward clinical-grade applications.
The rest of the paper is structured as four sections. First, the search results of existing relevant works and study characteristics will be elaborated. Then, we will discuss the revolution of digital medicine, the merits of VCPM technology and the necessity of clinical settings in general. Next, the main challenges in clinical study will be illustrated in detail. Finally, the future directions and prospects of VCPM will be presented at length.
Results
In this section, the future directions of VCPM will be summarized from three perspectives: the adoption of SOTA deep learning technologies and the breakthrough in the current limitations and clinical application challenges. The framework of this section is organized as Fig. 2. The establishment of a national and international standard of VCPM system is of top importance, and other parts can be divided into AI technology, clinical application and other aspects.
Unsupervised learning
The unsupervised learning technique can be employed to establish AI models for the VCPM task without relying on ground truth vital signs during the training stage. Furthermore, unsupervised learning methods are typically more robust to noise and variations in data, making them ideal for real-world applications. In fact, not only the vital signs hidden in the skin video are weak, but the spatio-temporal features are intertwined46. Hence, it is a significant challenge to explicitly design neural network structures or loss functions to effectively decouple these spatio-temporal vital-sign features. Nevertheless, it is feasible to construct a reasonable strategy of unsupervised learning that enables the model to learn on its own and disentangle the intertwined features. For instance, recent research has demonstrated that unsupervised learning technologies are capable of extracting rPPG signals from unlabeled video data47,48,49,50,51,52,53. Moreover, the performance of those algorithms48,49,50,51,54 is comparable to or even better than that of supervised approaches.
Federated learning
Federated Learning (FL) is a distributed machine learning paradigm or framework, and it is proposed to solve the data island problem of privacy protection. FL is capable of joint modeling without sharing participants’ data. The training dataset is stored in the local storage of participants, ensuring user privacy and complying with data usage standards55,56. In addition, FL technologies have arguably become the most widely used privacy preservation technique in AI-based medical applications56,57,58,59,60,61,62. Overall, FL technology is a promising solution to privacy protection, which can promote the R&D of VCPM for multi-centric clinical application studies. For instance, Liu et al. firstly developed a mobile FL camera-based PPG signal monitoring system with non-clinical public databases and showed that it can perform competitively with traditional state-of-the-art supervised learning methods63.
Skin segmentation and temporal consistency
Due to the common occurrence of face occlusion and lateral face orientation in clinical settings, skin segmentation is a suitable solution that can effectively alleviate these unfavorable conditions. ROI extraction or skin segmentation is a critical preprocessing step for the VCPM task as only the skin surface can offer information of blood volume changes. In the earlier studies, various facial landmark detectors64,65,66 have been utilized to locate the ROI67. Nevertheless, it proves that these methods are ineffective in scenarios involving head movement or face occlusion, etc17,68,69. Additionally, Ouzar et al. demonstrated that face detectors64,65,66 might fail to detect ROI in the MMSE-HR dataset70, whereas this issue can be resolved by adopting a face segmentation algorithm69,71.
It is important to accurately segment the skin ROI to extract vital signs effectively. Furthermore, skin segmentation has the ability to reduce noise and variation in original data, thereby improving the accuracy of VCPM algorithms and enabling vital signs extraction even under non-ideal conditions. In particular, skin segmentation is greatly crucial in clinical settings where the occlusion of the face of ICU patients is more prevalent. As shown in Fig. 3a, in the real-world scenario in ICU, face occlusion not only introduces additional noise, but also may cause failure in locating the patient’s face ROI if existing feature point detection algorithms are used. Therefore, by utilizing the SOTA semantic segmentation algorithms, we can obtain skin segmentation input at the pixel level, effectively mitigating interference from non-skin regions and enhancing the accuracy and robustness of the VCPM algorithm. The skin segmentation results, generated by the segment anything model (SAM) online demo (https://segment-anything.com/demo), on simulated clinical dataset are presented in Fig. 3.
Despite the introduction of segmentation algorithms by current pioneering researchers, the temporal consistency of continuous frames across video has not been taken into consideration22,69. Unlike single-image segmentation, temporal consistency is a critical metric that can significantly improve the performance of VCPM algorithms. Temporal consistency guarantees that the segmentation of each frame remains consistent with that of previous frames, which is crucial for accurate tracking of skin regions in videos over time. If the temporal consistency of the skin segmentation approach is suboptimal, the segmentation algorithm may introduce extra noise, ultimately leading to a decline in the performance of the VCPM algorithm. The AI-based optical flow is a potential research direction that can improve temporal consistency of video skin segmentation.
In 2023, various revolutionary segmentation tools72,73 were published, which provided great prospects for further improvement of VCPM algorithm performance. In Fig. 3, the results of facial skin segmentation using SAM72 in ICU patients under various complicated clinical conditions are presented. The segmentation results demonstrate that the background area is completely eliminated and the skin area is well preserved at a pixel-level precision. Thus, with the aid of advanced segmentation tools, the clinical VCPM algorithm can be trained with less background noise and more effective data, thereby increasing the feasibility of practical application.
Establishment of the national and international standard of the VCPM system
VCPM technology is an accessible, comfortable, and convenient approach for physiological monitoring. To prevent the potential abuse or misuse of VCPM, it is essential to establish national and international standards and guidelines for its use in digital medicine. In terms of algorithm performance and data security, the standards should at least encompass the following aspects. Additionally, the recommended settings for the clinical application of VCPM technologies are listed in Table 1.
-
Video capture software and hardware settings. The coded format of recording video is a crucial parameter for VCPM algorithm to extract vital signs. If the compression ratio of the collected video is too high, it may result in the loss of weak physiological signals implied in the video. The vital signs are time-domain information, therefore a stable and consistent sampling rate of the video is required. In addition, the resolution of the video is a crucial factor that ensures video quality and minimizes white noise. Hence, it is imperative to develop specialized video recording software to configure camera settings that can ensure the optimal performance of the VCPM algorithm.
-
Standard operating procedure (SOP). The SOP includes the lowest ambient light intensity, allowable subject motion magnitude, the shortest video duration and other considerations.
-
Data privacy protection. Due to the fact that video data commonly cover both facial information and vital signs of subjects, protecting privacy is a crucial issue and a primary requirement. Formulate the criteria for video data accessibility based on the purposes and occupational categories. The related occupations include public individuals, physicians, researchers, pertinent government staffs and policy makers.
Disease analysis, diagnosis and cardiopulmonary status assessment
With the aid of the AI technology boom, PPG, HR and heart rate variability can serve as biomarkers for disease analysis, diagnosis, and assessment of cardiopulmonary status30,31,32,74,75,76. Recently, numerous studies have demonstrated the high sensitivity and specificity of VCPM technologies in detecting atrial fibrillation75,77,78,79,80,81. Additionally, in literature32, a novel AI algorithm, which leverages PPG and ECG generated by PPG, has been successfully developed for CVD detection, including coronary artery disease, congestive heart failure, myocardial infarction (MI), and hypotension (HOTN). It can be seen that the utilization of VCPM technologies for monitoring vital signs and capturing their variations over days and weeks holds great potential in enabling early disease prediction and diagnosis35,82.
Multiple vital-sign measurement
Despite the verification of VCPM technology in measuring HR, RR, SpO2, and BP, a multi-task AI model proficient in simultaneous detection of the four vital signs has not emerged. However, in clinical settings, it is imperative to concurrently monitor multiple vital signs to ensure comprehensive monitoring of the patient’s physiological status. In clinical patient monitoring, HR, RR, SpO2 and BP are the four essential parameters that comprehensively reflect the cardiopulmonary status of patients, and they are the fundamental indicators of the traditional multi-parameter patient monitors. If the VCPM framework can monitor multiple vital parameters simultaneously, it will be closer to the application of fully non-contact monitoring of patients in highly acute settings. Therefore, the study of multi-parametric measurement of AI models will make a crucial breakthrough in real-world clinical applications. Currently, the majority of researchers are primarily focused on developing contactless measurement algorithms for a single physiological parameter or two parameters with strong correlation, such as HR and RR. For instance, Villarroel and Jorge et al. have developed two AI models capable of monitoring HR and RR in clinical conditions18,68.
Fortunately, the VCPM technology has demonstrated the ability to simultaneously measure vital signs including HR, RR, SpO2 and BP3,83,84,85. Firstly, HR and RR can be derived from PPG signals17,18,23,27,68,86,87,88,89; Secondly, by analyzing two distinct PPG waveforms at the same measurement site, SpO2 can be computed25,26,90,91,92,93,94; Finally, utilizing two different PPG waveforms extracted from separate body sites enables the inference of both diastolic and systolic BP1,2,4,95,96,97,98,99. Therefore, the development of a large-scale and multi-task AI model, which has the capacity to simultaneously measure multiple physiological signals, holds significant clinical application potential and represents a promising direction for future research. In conclusion, it is greatly promising to establish a unified AI model incorporating multiple physiological parameters in clinical scenarios.
Establishment of public health early warning and decision system based on VCPM
As illustrated in Fig. 4, the VCPM-based telemedicine system will not only be used for personal health monitoring and disease diagnosis, but also serves as an AI tool in response to public health issues, such as CVD in the elderly and the COVID-19 pandemic. Firstly, The vital signs of individuals measured by the VCPM-based telemedicine system can be utilized for personalized healthcare and disease diagnosis. Moreover, during an epidemic, the large-scale basic data of the public collected by the VCPM telemedicine system can be employed to establish a public health decision-making system, and offer crucial technical support for the government in formulating timely response strategies. For instance, numerous AI prediction models have been developed to predict the infected population and the mortality100,101,102.
Integration into telemedicine system for clinical application
In telemedicine or telehealth system, video consultation is one of the must-have functions, which is a subjective approach of disease counseling. After the COVID-19 pandemic from 2020 to 2022, many telemedicine/telehealth systems have implemented this method to mitigate cross-infection risks between healthcare providers and patients during treatment for COVID-19 or other illnesses103,104,105,106. Therefore, VCPM can be easily integrated into those existing medical systems to support objective physiological information during video consultations with physicians. The implementation of this measure will further enhance the functionality of the telemedicine system and provide an exceptional user experience for those utilizing the remote system.
Other recent research directions
Database synthesis method
It is a significantly challenging task to collect a large-scale and multi-centric database representing a range of environments, body movements, illumination conditions and physiological states. However, establishing a simulation video database integrated with physiological signals is a feasible solution for VCPM tasks84,107,108. For instance, in 2022, Daniel et al. released a synthetic database, named SCAMPS, which comprised 2,800 videos featuring synchronized cardiac and respiratory signals as well as facial action intensities109. Moreover, the synthetic data have the merits of noiselessness and precise synchronization. SCAMPS was successfully utilized to train AI models to develop the VCPM algorithms for healthy subjects52,110. Thus, developing a simulation database for clinical settings would be an invaluable future direction as it can mitigate the challenges of collecting extensive clinical data while safeguarding medical data privacy. It should be noted that the mathematical modeling of hemodynamics and oxygen saturation is a challenging task, which currently precludes the incorporation of oxygen and blood pressure information into simulation data.
Domain adaptive
Due to the bias (e.g., illumination, the bias of distinct clinical centers) between the training source and testing target domain, the generalization ability of deep learning-based methods should be introduced. To improve the generalization ability of rPPG models, Du et al. proposed a domain adaptive method that aligns intermediate domains and synthesizes target noise in the source domain to achieve superior noise reduction by reducing domain discrepancy110. We deem that the adaptive domain approach can be extended to effectively mitigate the disparity between laboratory scenario data and clinical data.
Transformer-based VCPM technologies
Transformer was first proposed in the field of Natural Language Processing (NLP)111,112. Then, another milestone event is the successful adaptation of Transformer for computer vision (CV) tasks, known as Vision Transformer (ViT)113. Nowadays, the Transformer module is renowned for achieving a unified architecture that utilizes self-attention mechanism to extract spatial (e.g., CV task) and temporal (e.g., NLP task) features simultaneously. Another characteristic of Transformer is its ability to handle various forms of input data fed into an embedded encoder. Overall, it is a promising direction to explore a Transformer-based VCPM framework which can extract spatial-temporal features and monitor multiple vital parameters in clinical settings. For instance, Wang et al. proposed a Transformer-based unsupervised learning model for remote HR measurement53.
GAN-based VCPM technologies
Generative adversarial network (GAN) is an unsupervised learning framework for estimating generative models via adversarial training114. GANs are widely utilized in the fields of data generation, data augmentation, style transfer, etc. Recently, GAN has been introduced to improve the performance and generalization of VCPM technologies110,115,116,117. Particularly, GAN is used to generate adversarial noise to improve the generalization ability of PPG signals’ prediction models110,115. Although some achievements have been made in studies on healthy subjects and laboratory settings, it also has significant value in clinical scenarios. Owing to the complex clinical scenarios and its distinction in different clinical centres, the generalization performance of AI-based VCPM algorithms might be degraded when applied to other clinical scenarios. Therefore, GAN-based VCPM technology is a potential approach to alleviating the generalization difficulty in multiple centres.
Discussion
In the section, we will discuss the topics on (A) digital medicine revolution; (B) the merits of VCPM technologies; (C) The necessity of clinical settings; and (D) Main challenges in clinical study.
(A) Digital medicine revolution
Key information
-
Digital Medicine. Digital medicine is a comprehensive concept that encompasses the use of digital technologies, such as biotechnology, health technology, and biomedical engineering, to enhance healthcare delivery and improve patient outcomes through signal processing, artificial intelligence, machine learning, and big data analysis.
-
Telehealth. Telehealth encompasses remote clinical healthcare, patient professional health education, as well as public health and healthcare administration. Usually, telehealth covers a significant proportion of digital health solutions.
-
Telemedicine. Currently, there is no universally accepted definition of telemedicine. Generally, it is the utilization of telecommunications to remotely provide healthcare services, encompassing a wide range of applications such as video consultations, diagnosis and patient monitoring. It can be implemented through video conferencing, photo calling or special telemedicine software. Telemedicine is a component of the broader field of telehealth.
-
Remote Patient Monitoring (RPM). RPM, a comprehensive technology solution, involves the utilization of sensors and other devices to remotely gather data on a patient’s health status, which can then be transmitted to healthcare providers for analysis and intervention. VCPM can be considered as one of the RPM techniques. This technology is applicable to monitor various conditions such as heart failure, diabetes, COVID-1935, and interstitial lung disease33.
-
Self-monitoring and home-based monitoring. Self-monitoring refers to the utilization of digital tools and devices by patients to track their own health data, such as blood pressure monitors, glucose meters, and fitness trackers. This practice enables patients to proactively manage their health and detect potential diseases at an early stage. Home-based monitoring involves leveraging digital technologies to deliver healthcare services directly to patients in their residences.
The measurement physiological signals is a fundamental procedure for monitoring the body’s status, which is widely employed in clinical settings and daily health surveillance. VCPM, as a contactless measurement method, offers the benefits of user-friendly monitoring, passive monitoring and cost-effectiveness etc. Therefore, it has the significant potential for application in clinical settings or home-based monitoring, and is poised to revolutionize traditional medical devices, telemedicine, intelligent monitoring, and medicine industry.
As illustrated in Fig. 5, the current trend in the development of neonatal physiological signal monitoring instruments is shifting from wired contact to wireless contact monitoring118,119,120, and ultimately towards contactless measurement. In 2022, an AI-based contactless physiological monitoring algorithm was developed for post-operative patients in ICU settings. In the study, the VCPM algorithm measured the HR with a mean absolute error (MAE) of 2.5 beats/min in comparison to two reference HR sensors, and measured the RR with a MAE of 2.4 breaths/min against the reference value computed from the chest impedance pneumogram18.
(B) The merits of VCPM technologies
Firstly, VCPM possesses a greater number of inherent and potential advantages. As illustrated in Fig. 6a, the approach of VCPM presents numerous merits, including contactless and non-invasive monitoring, passive measurement, user-friendliness, comfort and convenience, as well as suitability for long-term monitoring. Then, it leverages the ubiquitous devices and internet infrastructure at hand, including smartphones, webcams, and telecommunications systems. Therefore, the VCPM is a more natural method to establish telemedicine or home-based monitoring systems, and has the potential to yield significant economic and social benefits, including but not limited to preventing cross-infection among individuals/patients, reducing patients’ costs121, and promoting equitable distribution of medical resources.
The merits of clinical applications
Due to the prevalence of camera devices and the convenient monitoring manner, the VCPM technologies have the potential to flexibly record public large-scale disease data and an individual’s physiological information. Meanwhile, big data and AI technologies have played a significant role in studying and recognizing brand-new diseases (e.g., predict infection rate and mortality) by utilizing large-scale vital signs from the public41. On one hand, VCPM establishes horizontal relationships between patients and providers, and makes multinational collaborations more feasible41. Moreover, as illustrated in Fig. 7, VCPM technologies have broad applications in various clinical scenarios, such as elderly care, newborn monitoring, ICU patient healthcare, rehabilitation training, and so on.
On the other hand, in terms of individuals, VCPM can be easily implemented on a large scale to track longitudinal changes, which are crucial medical indicators of their physiological status. Individuals undergo their own daily, weekly, and seasonal fluctuations in a variety of physiological parameters and activities. The earliest deviations from the norm can be detected only by establishing an individual’s baseline when they are healthy44. Therefore, owing to its flexible and passive manner, VCPM is significant for monitoring the physiological parameters of patients whether they are at home or in the hospital. For example, patients can transmit their skin video to the AI physiological signal monitoring system, and a physician can work remotely based on the vital signs. These advancements may encourage patients and their families to take greater ownership of their own healthcare. This system has the ability to reduce medical costs, decrease reliance on specialized equipment and physicians, promote equal distribution of medical resources, and improve the quality of healthcare services.
The advantages of AI-based VCPM approaches
Firstly, it is an undisputed trend that utilizes SOTA AI technologies to develop VCPM algorithms. According to incomplete statistics, the vast majority of SOTA VCPM algorithms that were released from 2020 to 2023 are based on AI techniques. What’s more, the performance of deep learning methods has far exceeded that of signal processing methods. Secondly, AI technologies can not only be employed to develop the approaches of estimating multiple vital parameters, but also are capable of advancing post-process solutions, such as disease diagnosis with individual longitudinal analysis and other similar patient horizontal comparison44. In addition, the AI-based VCPM solution with the power of privacy protection presents a highly appealing option for clinical applications, such as utilizing AI-based approaches for protecting privacy55,57,58. Generally, the AI-based VCPM solution holds immense potential and offers significant advantages in the fields of clinical applications and digital medicine.
The opportunities of VCPM-based digital medicine system
The VCPM will significantly expand the application range and scenarios of telemedicine and telehealth systems. A telemedicine/telehealth system typically encompasses the fundamental capabilities of biosignal measurement and video consultation. As illustrated in Fig. 6a, the VCPM approach utilizes existing infrastructures, such as webcams and the Internet, to establish a telemedicine/telehealth system without specialized medical equipment. Therefore, as shown in Fig. 6b, VCPM is particularly well-suited for establishing a telemedicine system that integrates remote patient monitoring, home-based monitoring, and video consultation simultaneously. It can be regarded as one of the essential underlying technologies for telemedicine systems, especially in supporting remote patient monitoring and home-based healthcare.
Ultimately, VCPM technology offers an unprecedented opportunity to self-health monitoring, PHC41 and telemedicine system due to the distinctive merits of VCPM, which include contactless operation, user-friendly interface, low cost and non-requirement for medical professionals. VCPM can be applied across the entire spectrum of prevention, diagnosis, and treatment. It is competent method to facilitate self-physiological signal monitoring and health status assessment in the stage of disease prevention. Furthermore, it can be integrated into PHC and telemedicine systems with fundamental physiological data for diagnosis. It serves as a tool to monitor the body’s physiological state, and can be applied widely as illustrated in Fig. 7.
(C) The necessity of clinical settings
Firstly, multi-parameter monitoring is a necessary approach to maintaining the life and health of preterm / newborn infants. In 2020, World Health Organization reported that approximately 35% of all under-5 deaths occurred within the first week of birth122. For newborn or preterm infants, vital-sign monitoring is a fundamental clinical requirement because the fetal-to-neonatal transition after birth is a complex physiological process that affects all organ systems123,124,125,126. Moreover, it is also an indispensable procedure in the neonatal intensive care unit (NICU) environment. However, traditional contact-based methods are uncomfortable even harmful over the long-term contact of sensors. Thus, the visual contactless pipeline provides a notable competitive advantage in vital-sign monitoring by providing a convenient and contactless approach17. For instance, some pioneering studies have been conducted on hospitalized neonates based on deep learning17,68,127,128.
Furthermore, the majority of clinical patients require vital-sign monitoring, particularly those who are critically ill, have had surgery or suffered from CVD129 or hypertension. For patients who require long-term monitoring, traditional contact monitors have obvious clinical disadvantages. If the sensor probe is too tight, it can cause skin damage during extended use. Conversely, if the probe is too loose, it may easily detach due to the patient’s movement and necessitate professional reattachment. The primary unmet need being addressed by non-contact monitoring solutions is the mitigation of patient discomfort caused by contact or wearable monitoring technology130. For instance, wearable sensors are difficult to use in some patients with cognitive impairment (e.g., Alzheimer’s disease)75.
(D) Main challenges in clinical study
Compared with the studies based on healthy subjects or laboratory scenarios, the clinical application of VCPM faces a multitude of unique challenges and the number of clinical studies is extremely limited. Therefore, the study of VCPM techniques is highly valuable in addressing digital healthcare challenges in real-world clinical scenarios131. Certainly, the following disadvantages of the VCPM technologies must be taken into consideration when applied in clinical settings: (1) Privacy protection; (2) Requiring substantial clinical validation; (3) Not suitable for dark environment unless using an infrared camera; (4) The performance susceptible to disturbance, such as head movement. In addition to the aforementioned issues, the primary obstacles that VCPM faces in clinical study are drawn out in this section.
I. A shortage of public clinical database
The primary challenge lies in the absence of a publicly available database of clinical scenarios. To date, some pioneering studies have been conducted on clinical patients, but none of those data is available due to the patients’ privacy protection. Villarroel et al. conducted research on the application of VCPM algorithms in post-operative patients18 and preterm infants68 in the intensive care unit (ICU) respectively. However, those corresponding database are not publicly accessible. Moreover, there are only 15 ICU patients and 30 preterm infants recruited in literature68 and18 respectively. The limited amount of clinical data from a single center are insufficient to support further research and optimization of AI algorithms, as well as the clinical application of VCPM algorithms.
The scarcity of a clinical public database seriously impedes the algorithmic and application innovation in the VCPM research community. First, due to the unavailability of clinical data, the barriers to the clinical research of VCPM are increased. Thus, a significant proportion of scholars fail to carry out research smoothly. Next, there is no unified benchmark for comparing algorithms developed by various researchers. Last, it would hinder the healthy and sustainable development of the research community. Overall, it is imperative and opportune to establish a public database on its clinical scenarios.
The main deterrent to releasing clinical data results from safeguarding patients’ privacy132,133. Generally, the data recorder for VCPM includes the facial video of patients and multiple physiological information. Hence, providing access to the data for researchers in need while ensuring privacy presents a tricky issue. To this end, it is necessary to establish new standards for privacy and disclosure of clinical databases by collaborating with the government, academia, and medical community. These guidelines will revolutionize the development and application of AI technologies in digital medicine. From the perspective of technology, there are at least two potential solutions to achieve this objective: establish an AI-based privacy protection system or simulated database for clinical studies.
On the one hand, the primary concept behind federated learning systems is to construct machine learning models utilizing database that are distributed across multiple devices, while simultaneously preventing any potential data leakage134,135. Recently, federated learning have been widely applied in healthcare and clinical systems57,58,136,137,138, and just one VCPM study leverages federated learning at present63. On the other hand, simulation dataset leverages the concept of digital twins, utilizing both the original clinical video and corresponding physiological signal data to construct a simulation database. Thus, some researchers just need to access the simulation dataset to develop their AI algorithms, and then adopt the transfer learning to optimize the model trained on simulated data.
II. Complex clinical scenarios
Due to the extremely weak vital signs hidden in facial videos, they are susceptible to interference from subjects’ status and surroundings. For instance, face occlusion and lateral face videos can weaken physiological signals, while head motion and illumination changes will enhance disturbances. Ultimately, these negative factors increase the challenges in developing a robust VCPM algorithm.
Face occlusion and lateral face orientation
Oxygen therapy is commonly applied to ICU patients, but it will obscure parts of the face due to the presence of oxygen tubes and fixed coated fabric. Moreover, the oxygen tubes are situated in various regions of the face. Thus, it is a time-consuming and laborious task to segment them from each frame of the facial videos. In addition, unlike healthy subjects in laboratory settings, clinical patients can not be instructed to face the camera and are typically confined to their sickbeds with a lateral orientation. Moreover, to minimize background interference and maximize the retention of skin that contains physiological signal information, it is necessary to eliminate non-skin region as much as possible before we feed the skin regions into a VCPM algorithm. For videos of the healthy subjects’ face, a facial landmark tool is commonly utilized to extract face ROI, but the tool is not usually applicable to subjects with occluded or laterally oriented face139,140.
Head motion and illumination changes
Furthermore, the current bottleneck of VCPM solution lies in their algorithmic performance, which fails to meet clinical measurement accuracy requirements when subjects experience head motion or surrounding illumination changes. The essential reason is that the amplitude of weak vital signs concealed in facial videos is significantly lower than the noise caused by head motion and illumination changes. There have been a few pioneering scholars attempting to tackle these hot-potato issues, yet much work remains for VCPM technologies to attain their full potential, particularly in medical field.
III. Confidence evaluation of algorithms
Due to head movements or illumination changes, the performance of VCPM algorithms may become worse. Therefore, it is reasonable to introduce a confidence evaluation to assess results. The real-time presentation of the confidence coefficient indicates the level of confidence in the measured vital signs. The confidence level can be regarded as a metric of evaluating the algorithm’s adaptability to clinical scenarios. Furthermore, it will not only facilitate physicians in assessing patients’ conditions, but also provide guidance for further algorithmic improvement to researchers.
IV. Pathological feasibility analysis
In clinical practice, there is a high prevalence of hypertension and CVDs among the elderly population, resulting in various abnormal PPG signals. As shown in Fig. 8, the morphological characteristics of abnormal PPG signals are greatly different from that of normal PPG waveforms. As illustrated in Fig. 1, the fundamental principle of VCPM algorithm is based on the current normal PPG signal. Therefore, it is a tough task to develop a clinical VCPM algorithm that can adapt to abnormal PPG signals and further infer HR, RR, BP, and SpO2, which is also urgently needed validation in clinical studies.
However, there has been no study investigating the impact of abnormal pathological PPG signals on the performance of VCPM algorithms so far. All studies assume that subjects have normal PPG waveforms. Hence, comprehensive research guarantee that the effectiveness and robustness of VCPM algorithm is applicable to abnormal PPG signals in clinical situations, which is a challenging task and an innovative future direction to develop an effective VCPM pipeline. Furthermore, VCPM-based technologies can be developed for the diagnosis of CVDs.
Figure 8 displays five PPG waveforms, including four commonly seen abnormal PPG signals in clinical settings. Similarly, the abnormal PPG waveforms were detected in our clinical studies uses a finger-clip sensor. The waterhammer PPG is characterized by a sudden increase in the amplitude of the PPG signal, followed by a gradual decrease. The slow-rising PPG is identified by a prolonged rise time, which refers to the duration from the onset of the blood volume change to the peak of the signal. Specifically, the slow-rising PPG has a longer rise time compared with normal signals.
Pulsus bisferiens, meaning “beating twice", is a type of arterial pulse characterized by two distinct systolic peaks resulting from a rapid rise in blood pressure during systole, followed by a brief fall and then a second rise. This phenomenon is most commonly associated with aortic regurgitation, which stems from the incomplete closure of the aortic valve during diastole, leading to retrograde blood flow into the left ventricle, causing an increase in stroke volume consequently.
Pulsus alternans is a condition distinguished by alternating strength of the arterial pulse between beats due to variations in stroke volume. A decrease in stroke volume leads to weaker pulses, while an increase results in stronger ones. This phenomenon is most commonly related to the left ventricular dysfunction like that observed in heart failure.
Methods
Search results
We retrieved a total of 381, 1279 and 1243 records from three databases (Pubmed, Web of Science (WOS), and IEEE) respectively (Fig. 9). Initially, we applied time filters due to the commencement of VCPM research in 2007, which resulted in a remaining total of 345, 1026, and 1139 records respectively. Subsequently, specific built-in filter tools of the three databases were employed: (1) Exclude 28 records with not full text filter in Pubmed; (2) After filtering out literature types that include review papers, unspecified material, books, abstracts only and letter, 730 papers remain in the search results of WOS; (3) Utilizing the filter of publication topics (patient monitoring, medical image processing, medical signal processing, cardiology, diseases, health care, biomedical optical imaging, telemedicine, medical signal detection or cardiovascular system) and 993 records remain in the search results of IEEE. Next, after eliminating the duplicates, there were 1943 items remaining. Finally, after screening by title, abstract or full text, studies conducted in laboratory settings or using radar sensors (e.g., MMW radar) were excluded. Only research papers related to clinical settings, digital medicine, telemedicine or healthcare were selected for final analysis, resulting in a total of 43 papers.
Study characteristics
The 43 research papers are listed in reverse chronological order in Table 2. There were 24 papers in which studies were based on neonates or premature infants, the subjects of another 18 papers were adult patients, and the subjects of the last one included newborns and children141. Besides, Batbayar et al. developed a rapid preliminary COVID-19 screening system integrated with a stereo depth, an RGB and a thermal camera to measure RR, HR, and body temperature (BT) respectively142. The six studies (Villarroel et al., 2020, 2019, 2014; Chaichulee et al. 2019, 2018; Jorge et al., 2022)18,68,143,144,145,146 were from the same team at Oxford University. Among these studies, four papers focused on neonates while the remaining two were for adult patients.
In terms of the implemented algorithm, there were 29 research papers (29/43, 67.4%) that conducted classical methods, and only nine studies (10/43, 23.3%) implemented AI-based methods. Besides, the remaining four articles (4/43, 9.3%) did not explicitly state the used methods. Among the ten AI-based papers, five studies were from the team of Oxford University (UK)18,68,143,145,146, two from Beihang University (China)17,147, and the remaining three from RWTH Aachen University (Germany)148, Indian Institute of Technology Madras (India)127, Institute of Computer Science FORTH (Greece)149 respectively.
Additionally, the relationship between the number of subjects and total videos’ length are presented in Fig. 10 based on the data resources shown in Table 2. Despite the 23 pairs of data may not be entirely statistically significant, Fig. 10 presents that the data scale (video total length or number of subject) of AI-based methods are commonly larger than non-AI approaches when excluding studies inside the blue ellipse from the same team. Generally, the performance of AI-based approaches is dependent on the scale of database, while classical methods only require fewer data samples. In fact, no matter AI-based approaches or classical ones, a large-scale and diverse database is indispensable to assess the comprehensive performance of VCPM approach toward complex real-world clinical settings.
In addition, although contactless studies in clinical scenarios have been emerging, some critical physiological indicators such as SpO2 and BP have not been researched so far. The research contrast of physiological parameters between laboratories and clinical scenarios is illustrated in Table 3. Depending on the selected clinical studies and the SOTA research trends of VCPM, our findings are summarized below:
-
All clinical studies are based on their own database. The vast majority of these studies focus on the assessment of clinical applications, rather than dealing with real-world clinical scenarios, researching novel paradigm or analyzing clinicopathology.
-
As shown in Tables 2 and 3, all clinical studies concentrate on the measurement of HR or RR. Even though AI-based research developed for SpO2 or BP measurement on healthy subjects has been growing explosively from 202190,96, none of them are intended for monitoring SpO2 or BP in clinical settings.
-
The VCPM algorithms, developed for clinical settings, are increasingly favored by researchers at present. In particular, the research on VCPM algorithms showed an exponential growth in 2022.
-
There is a great gap between healthy/laboratory scenarios and clinical settings on the research of AI-based methods. A great many SOTA AI-based methods have been developed on healthy/laboratory settings20, but none of them has been applied in clinical settings.
Phenomenon: the gap between the laboratory and clinical settings
-
(1)
The studies of AI-based VCPM algorithm have been growing exponentially on healthy subjects/laboratory settings from 2021 to 2023. As shown in Table 3, the SOTA algorithms have demonstrated outstanding performance, but have rarely been generalized to clinical application.
-
(2)
The studies of VCPM algorithms have been soaring on patients/clinical settings from 2021 to 2023, yet only a limited number of studies have incorporated AI-based algorithms. In Table 1, only 6 papers (6/21, 28.6%) utilized AI technologies (2021–2023).
-
(3)
From the perspective of the novelty of approaches based on AI technologies, unsupervised learning47,48,51, Transformer49,150,151,152, GANs115,116,117, meta-learning153, and Graph Neural Networks154,155 have been developed for non-clinical scenarios. However, these technologies are rarely utilized for clinical settings.
-
(4)
As illustrated in Table 3, although the AI-based study about contactless SpO2 and BP estimation has become a hot topic in recent two years, all the studies concentrated on healthy people rather than the patients in the hospital.
Reason: thinking and inference
-
(1)
The SOTA AI-based VCPM algorithms developed in laboratory settings still face significant challenges in clinical application. Further verification is required to confirm the performance of these algorithms when generalized to clinical scenarios.
-
(2)
Because of the privacy protection of patients, there is still a shortage of large-scale and accessible clinical databases for the researchers of computer vision and AI. This seriously hinders the development and application of AI-based VCPM algorithms in the clinical environment.
-
(3)
Particularly, the performance of SpO2 and BP measurement algorithm in clinical settings is urgently need to be validated. It has crucial guiding significance for subsequent clinical studies of VCPM technologies. The performance evaluation of measuring SpO2 should cover the range of blood oxygen levels from 70% to 100%. To comprehensively evaluate the performance of the BP measurement algorithm, it is necessary to recruit sufficient hypertensive and hypotensive patients respectively.
Prospect: possible solutions
-
(1)
It will represent a significant milestone to establish a large-scale publicly accessible clinical database and usage standard for VCPM researchers. The database includes video, PPG, HR, RR, SpO2, and BP information. The greatest obstacle to achieving public access to the clinical database is privacy protection. Therefore, the exploration of a publicly available clinical database that protects privacy is one of the crucial future research directions.
-
(2)
The database will promote the study of VCPM algorithms in clinical settings, achieve a fair comparison of algorithm performance, and facilitate the sustainable development of the VCPM community.
-
(3)
Moreover, the database will attract plenty of excellent researchers in the field of computer vision and AI to join the clinical VCPM community. Ultimately, it will bridge the current great research gap between laboratory and clinical settings, and accelerate the clinical applications of VCPM technologies.
Reporting summary
Further information on research design is available in the Nature Research Reporting Summary linked to this article.
References
Curran, T., Liu, X., McDuff, D., Patel, S. & Yang, E. Camera-based remote photoplethysmography to measure heart rate and blood pressure in ambulatory patients with cardiovascular disease: Preliminary analysis. J. Am. Coll. Cardiol. 81, 2301–2301 (2023).
Wu, B.-F., Wu, B.-J., Tsai, B.-R. & Hsu, C.-P. A facial-image-based blood pressure measurement system without calibration. IEEE Trans. Instrum. Measur. 71, 1–13 (2022).
Wang, W. & Wang, X. Contactless vital signs monitoring (Academic Press, 2021).
Frey, L., Menon, C. & Elgendi, M. Blood pressure measurement using only a smartphone. npj Digital Med. 5, 86 (2022).
Shabaan, M. et al. Survey: smartphone-based assessment of cardiovascular diseases using ecg and ppg analysis. BMC Med. Inform. Decision Making 20, 1–16 (2020).
Krittanawong, C. et al. Integration of novel monitoring devices with machine learning technology for scalable cardiovascular management. Nat. Rev. Cardiol. 18, 75–91 (2021).
Pereira, T. et al. Photoplethysmography based atrial fibrillation detection: a review. NPJ Digital Med. 3, 3 (2020).
Wu, T., Blazek, V. & Schmitt, H. J. Photoplethysmography imaging: a new noninvasive and noncontact method for mapping of the dermal perfusion changes. In Optical techniques and instrumentation for the measurement of blood composition, structure, and dynamics, vol. 4163, 62–70 (SPIE, 2000).
Takano, C. & Ohta, Y. Heart rate measurement based on a time-lapse image. Med. Eng. Phys. 29, 853–857 (2007).
Verkruysse, W., Svaasand, L. O. & Nelson, J. S. Remote plethysmographic imaging using ambient light. Optics express 16, 21434–21445 (2008).
De Haan, G. & Jeanne, V. Robust pulse rate from chrominance-based rppg. IEEE Trans. Biomed. Eng. 60, 2878–2886 (2013).
De Haan, G. & Van Leest, A. Improved motion robustness of remote-ppg by using the blood volume pulse signature. Physiol. Measur. 35, 1913 (2014).
Wang, W., Den Brinker, A. C., Stuijk, S. & De Haan, G. Algorithmic principles of remote ppg. IEEE Trans. Biomed. Eng. 64, 1479–1491 (2016).
Wang, W., Stuijk, S. & De Haan, G. A novel algorithm for remote photoplethysmography: spatial subspace rotation. IEEE Trans. Biomed. Eng. 63, 1974–1984 (2015).
Chaichulee, S. et al. Multi-task convolutional neural network for patient detection and skin segmentation in continuous non-contact vital sign monitoring. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), 266–272 (IEEE, 2017).
Hsu, G.-S., Ambikapathi, A. & Chen, M.-S. Deep learning with time-frequency representation for pulse estimation from facial videos. In 2017 IEEE international joint conference on biometrics (IJCB), 383–389 (IEEE, 2017).
Huang, B. et al. A neonatal dataset and benchmark for non-contact neonatal heart rate monitoring based on spatio-temporal neural networks. Eng. Appl. Artificial Intell. 106, 104447 (2021).
Jorge, J. et al. Non-contact physiological monitoring of post-operative patients in the intensive care unit. NPJ Digital Med. 5, 4 (2022).
Huang, P.-W., Wu, B.-J. & Wu, B.-F. A heart rate monitoring framework for real-world drivers using remote photoplethysmography. IEEE J. Biomed. Health Inform. 25, 1397–1408 (2020).
Zhao, C., Zhou, M., Zhao, Z., Huang, B. & Rao, B. Learning spatio-temporal pulse representation with global-local interaction and supervision for remote prediction of heart rate. IEEE J. Biomed. Health Inform. (2023).
Lu, H., Yu, Z., Niu, X. & Chen, Y.-C. Neuron structure modeling for generalizable remote physiological measurement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 18589–18599 (2023).
Li, B., Zhang, P., Peng, J. & Fu, H. Non-contact ppg signal and heart rate estimation with multi-hierarchical convolutional network. Pattern Recognit. 139, 109421 (2023).
Alnaggar, M., Siam, A. I., Handosa, M., Medhat, T. & Rashad, M. Video-based real-time monitoring for heart rate and respiration rate. Expert Syst. Appli. 225, 120135 (2023).
Zhao, C., Wang, H., Chen, H., Shi, W. & Feng, Y. Jamsnet: a remote pulse extraction network based on joint attention and multi-scale fusion. IEEE Trans Circuits Syst Video Technology (2022).
Mathew, J. et al. Remote blood oxygen estimation from videos using neural networks. IEEE J. Biomed. Health Inform. (2023).
Gupta, A., Ravelo-García, A. G. & Dias, F. M. Availability and performance of face based non-contact methods for heart rate and oxygen saturation estimations: A systematic review. Comput. Method. Programs Biomed. 219, 106771 (2022).
Vatanparvar, K., Gwak, M., Zhu, L., Kuang, J. & Gao, A. Respiration rate estimation from remote ppg via camera in presence of non-voluntary artifacts. In 2022 IEEE-EMBS International Conference on Wearable and Implantable Body Sensor Networks (BSN), 1–4 (IEEE, 2022).
Slapničar, G., Wang, W. & Luštrek, M. Feasibility of remote blood pressure estimation via narrow-band multi-wavelength pulse transit time. ACM Trans Sensor Netw. early access, 1–20, (Association for Computing Machinery, New York, NY, USA, 2023).
Wu, B.-J., Wu, B.-F. & Hsu, C.-P. Camera-based blood pressure estimation via windkessel model and waveform features. IEEE Trans Instrum. Measur. (2022).
Charlton, P. H. et al. Wearable photoplethysmography for cardiovascular monitoring. Proc. IEEE 110, 355–381 (2022).
Sadad, T. et al. Detection of cardiovascular disease based on ppg signals using machine learning with cloud computing. Comput. Intell. Neurosci. 2022, 1-11 (2022).
Lan, E. Performer: A novel ppg-to-ecg reconstruction transformer for a digital biomarker of cardiovascular disease detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 1991–1999 (2023).
Wijsenbeek, M. S. et al. Home monitoring in interstitial lung diseases. Lancet Respir. Med. 11, 97–110 (2023).
Hollander, J. E. & Carr, B. G. Virtually perfect? telemedicine for covid-19. N. Engl. J. Med. 382, 1679–1681 (2020).
Alboksmaty, A. et al. Effectiveness and safety of pulse oximetry in remote patient monitoring of patients with covid-19: a systematic review. Lancet Digital Health 4, e279–e289 (2022).
Talukdar, D., De Deus, L. F. & Sehgal, N. Evaluation of a camera-based monitoring solution against regulated medical devices to measure heart rate, respiratory rate, oxygen saturation, and blood pressure. Cureus 14, 1–13 (2022).
Omboni, S. et al. Evidence and recommendations on the use of telemedicine for the management of arterial hypertension: an international expert position paper. Hypertension 76, 1368–1383 (2020).
OECD. The COVID-19 Pandemic and the Future of Telemedicine https://www.oecd-ilibrary.org/content/publication/ac8b0a27-en (2023).
Samson, L. W., Tarazi, W., Turrini, G. & Sheingold, S. Medicare beneficiaries’ use of telehealth in 2020: Trends by beneficiary characteristics and location. Washington, DC: Office of the Assistant Secretary for Planning and Evaluation (2021).
Bundesvereinigung, K. Immer mehr praxen greifen zur kamera-zahl der videosprechstunden auf über eine million gestiegen. Kassenärztliche Bundesvereinigung. https://www.kbv.de/html/1150_50419. php [accessed 2022-03-28] (2021).
Hanson, K. et al. The lancet global health commission on financing primary health care: putting people at the centre. Lancet Global Health 10, e715–e772 (2022).
Whitelaw, S., Mamas, M. A., Topol, E. & Van Spall, H. G. Applications of digital technology in covid-19 pandemic planning and response. Lancet Digital Health 2, e435–e440 (2020).
Organization, W. H. et al. Interim guidance for member states-on the use of pulse oximetry in monitoring covid-19 patients under home-based isolation and care 2021 (2021).
Radin, J. M., Quer, G., Jalili, M., Hamideh, D. & Steinhubl, S. R. The hopes and hazards of using personal health technologies in the diagnosis and prognosis of infections. Lancet Digital Health 3, e455–e461 (2021).
Greenhalgh, T. et al. Remote management of covid-19 using home pulse oximetry and virtual ward support. bmj 372, 1-13 (2021).
Narayanswamy, G. et al. Bigsmall: Efficient multi-task learning for disparate spatial and temporal physiological measurements. Preprint at https://arxiv.org/abs/2303.11573 (2023).
Gideon, J. & Stent, S. The way to my heart is through contrastive learning: Remote photoplethysmography from unlabelled video. In Proceedings of the IEEE/CVF international conference on computer vision, 3995–4004 (2021).
Yue, Zijie, Miaojing Shi, and Shuai Ding. Facial Video-based Remote Physiological Measurement via Self-supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45, 13844–13859 (2023).
Park, S., Kim, B.-K. & Dong, S.-Y. Self-supervised rgb-nir fusion video vision transformer framework for rppg estimation. IEEE Trans. Instrum. Measur. 71, 1–10 (2022).
Sun, Z. & Li, X. Contrast-phys: Unsupervised video-based remote physiological measurement via spatiotemporal contrast. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XII, 492–510 (Springer, 2022).
Speth, J., Vance, N., Flynn, P. & Czajka, A. Non-contrastive unsupervised learning of physiological signals from video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14464–14474 (2023).
Yang, Y. et al. Simper: Simple self-supervised learning of periodic targets. Preprint at https://arxiv.org/abs/2210.03115 (2022).
Wang, R.-X., Sun, H.-M., Hao, R.-R., Pan, A. & Jia, R.-S. Transphys: Transformer-based unsupervised contrastive learning for remote heart rate measurement. Biomed.Signal Process. Control 86, 105058 (2023).
Liu, X. et al. rppg-mae: Self-supervised pre-training with masked autoencoders for remote physiological measurement. Preprint at https://arxiv.org/abs/2306.02301 (2023).
Li, J. et al. A federated learning based privacy-preserving smart healthcare system. IEEE Transactions on Industrial Informatics 18, 2021–2031 (2021).
Kaissis, G. A., Makowski, M. R., Rückert, D. & Braren, R. F. Secure, privacy-preserving and federated machine learning in medical imaging. Nat. Machine Intell. 2, 305–311 (2020).
Rieke, N. et al. The future of digital health with federated learning. NPJ Digital Med. 3, 119 (2020).
Nguyen, D. C. et al. Federated learning for smart healthcare: A survey. ACM Comput. Surveys. 55, 1–37 (2022).
Gupta, A., Misra, S., Pathak, N. & Das, D. Fedcare: Federated learning for resource-constrained healthcare devices in iomt system. IEEE Trans. Comput. Soc. Syst. (2023).
Kaissis, G. et al. End-to-end privacy preserving deep learning on multi-institutional medical imaging. Nat. Machine Intell. 3, 473–484 (2021).
Couderc, J.-P., Page, A., Lutz, M., Tsouri, G. R. & Hall, B. Assessment of facial video-based detection of atrial fibrillation across human complexion. Cardiovasc. Digital Health J. 3, 305–312 (2022).
Sun, Z., Junttila, J., Tulppo, M., Seppänen, T. & Li, X. Non-contact atrial fibrillation detection from face videos by learning systolic peaks. IEEE J. Biomed. Health Inform. 26, 4587–4598 (2022).
Liu, X., Zhang, M., Jiang, Z., Patel, S. & McDuff, D. Federated remote physiological measurement with imperfect data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2155–2164 (2022).
Viola, P. & Jones, M. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001, vol. 1, I–I (Ieee, 2001).
King, D. E. Dlib-ml: a machine learning toolkit. J Machine Learning Rese. 10, 1755–1758 (2009).
Zhang, K., Zhang, Z., Li, Z. & Qiao, Y. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23, 1499–1503 (2016).
Huang, B., Lin, C.-L., Chen, W., Juang, C.-F. & Wu, X. A novel one-stage framework for visual pulse rate estimation using deep neural networks. Biomed. Signal Process. Control 66, 102387 (2021).
Villarroel, M. et al. Non-contact physiological monitoring of preterm infants in the neonatal intensive care unit. NPJ Digital Med. 2, 128 (2019).
Ouzar, Y., Djeldjli, D., Bousefsaf, F. & Maaoui, C. X-ippgnet: a novel one stage deep learning architecture based on depthwise separable convolutions for video-based pulse rate estimation. Comput. Biol. Med. 154, 106592 (2023).
Zhang, Z. et al. Multimodal spontaneous emotion corpus for human behavior analysis. In Proceedings of the IEEE Conference On Computer Vision And Pattern Recognition, 3438–3446 (2016).
Nirkin, Y., Masi, I., Tuan, A. T., Hassner, T. & Medioni, G. On face segmentation, face swapping, and face perception. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), 98–105 (IEEE).
Kirillov, A. et al. Segment anything. Preprint at https://arxiv.org/abs/2304.02643 (2023).
Zou, X. et al. Segment everything everywhere all at once. Preprint at https://arxiv.org/abs/2304.06718 (2023).
Li, X. et al. The obf database: A large face video database for remote physiological signal measurement and atrial fibrillation detection. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), 242–249 (IEEE).
Saner, H., Knobel, S. E. J., Schuetz, N. & Nef, T. Contact-free sensor signals as a new digital biomarker for cardiovascular disease: chances and challenges. Eur. Heart J. Digital health 1, 30–39 (2020).
Dautov, C. P., Dautov, R., Couderc, J.-P. & Tsouri, G. R. Machine learning approach to detection of atrial fibrillation using high quality facial videos. In 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), 1–4 (IEEE, 2021).
Sun, Y. et al. Contactless facial video recording with deep learning models for the detection of atrial fibrillation. Sci. Rep. 12, 281 (2022).
Liu, X. et al. Vidaf: A motion-robust model for atrial fibrillation screening from facial videos. IEEE J. Biomed. Health Inform. 26, 1672–1683 (2021).
Wu, B.-F., Wu, B.-J., Cheng, S.-E., Sun, Y. & Chung, M.-L. Motion-robust atrial fibrillation detection based on remote-photoplethysmography. IEEE J. Biomed. Health Inform. (2022).
Liu, X., Yang, X., Song, R., Wang, D. & Li, L. Pfdnet: A pulse feature disentanglement network for atrial fibrillation screening from facial videos. IEEE J. Biomed. Health Inform. (2022).
Yan, B. P. et al. High-throughput, contact-free detection of atrial fibrillation from video with deep learning. JAMA Cardiol. 5, 105–107 (2020).
Diao, J. A., Marwaha, J. S. & Kvedar, J. C. Video-based physiologic monitoring: promising applications for the icu and beyond. npj Digital Med. 5, 26 (2022).
Molinaro, N. et al. Contactless vital signs monitoring from videos recorded with digital cameras: an overview. Front. Physiol. 13, 160 (2022).
McDuff, D. Camera measurement of physiological vital signs. ACM Comput. Surveys 55, 1–40 (2023).
Zhang, X., Hu, M., Zhang, Y., Zhai, G. & Zhang, X.-P. Recent progress of optical imaging approaches for noncontact physiological signal measurement: A review. Adv. Intell. Syst. 5, 1–21 (2023).
Scebba, G., Da Poian, G. & Karlen, W. Multispectral video fusion for non-contact monitoring of respiratory rate and apnea. IEEE Trans. Biomed. Eng. 68, 350–359 (2020).
Molinaro, N., Schena, E., Silvestri, S. & Massaroni, C. Multi-roi spectral approach for the continuous remote cardio-respiratory monitoring from mobile device built-in cameras. Sensors 22, 2539 (2022).
Maurya, L., Kaur, P., Chawla, D. & Mahapatra, P. Non-contact breathing rate monitoring in newborns: a review. Comput. Biol. Med. 132, 104321 (2021).
Lorato, I. et al. Towards continuous camera-based respiration monitoring in infants. Sensors 21, 2268 (2021).
Hoffman, J. S. et al. Smartphone camera oximetry in an induced hypoxemia study. NPJ Digital Med. 5, 146 (2022).
Shao, D. et al. Noncontact monitoring of blood oxygen saturation using camera and dual-wavelength imaging system. IEEE Trans. Biomed. Eng. 63, 1091–1098 (2015).
Stogiannopoulos, T., Cheimariotis, G.-A. & Mitianoudis, N. A study of machine learning regression techniques for non-contact spo2 estimation from infrared motion-magnified facial video. Information 14, 301 (2023).
Hu, M. et al. Contactless blood oxygen estimation from face videos: a multi-model fusion method based on deep learning. Biomed. Signal Process. Control 81, 104487 (2023).
Tian, X., Wong, C.-W., Ranadive, S. M. & Wu, M. A multi-channel ratio-of-ratios method for noncontact hand video based spo _2 monitoring using smartphone cameras. IEEE J. Selected Top. Signal Process. 16, 197–207 (2022).
González, S., Hsieh, W.-T. & Chen, T. P.-C. A benchmark for machine-learning based non-invasive blood pressure estimation using photoplethysmogram. Sci. Data 10, 149 (2023).
Zhao, L. et al. Emerging sensing and modeling technologies for wearable and cuffless blood pressure monitoring. NPJ Digital Med. 6, 93 (2023).
Xuan, Y. et al. Ultra-low-cost mechanical smartphone attachment for no-calibration blood pressure measurement. Sci. Rep. 13, 8105 (2023).
Bousefsaf, F. et al. Estimation of blood pressure waveform from facial video using a deep u-shaped network and the wavelet representation of imaging photoplethysmographic signals. Biomed. Signal Process. Control 78, 103895 (2022).
Huang, B., Chen, W., Lin, C.-L., Juang, C.-F. & Wang, J. Mlp-bp: a novel framework for cuffless blood pressure measurement with ppg and ecg signals based on mlp-mixer neural networks. Biomed. Signal Process. Control 73, 103404 (2022).
Yadaw, A. S. et al. Clinical features of covid-19 mortality: development and validation of a clinical prediction model. Lancet Digital Health 2, e516–e525 (2020).
Wynants, L. et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. bmj 369, 1–22 (2020).
Moulaei, K., Shanbehzadeh, M., Mohammadi-Taghiabad, Z. & Kazemi-Arpanahi, H. Comparing machine learning algorithms for predicting covid-19 mortality. BMC Med. Inform. Decision Making 22, 1–12 (2022).
Li, J.-P. O. et al. Safety of video-based telemedicine compared to in-person triage in emergency ophthalmology during covid-19. EClinicalMedicine 34, 100818 (2021).
Salvatore, C. M. et al. Neonatal management and outcomes during the covid-19 pandemic: an observation cohort study. Lancet Child Adoles. Health 4, 721–727 (2020).
Shaffer, K. M. et al. Digital health and telehealth in cancer care: a scoping review of reviews. Lancet Digital Health 5, e316–e327 (2023).
Palmer, K. R. et al. Widespread implementation of a low-cost telehealth service in the delivery of antenatal care during the covid-19 pandemic: an interrupted time-series analysis. Lancet 398, 41–52 (2021).
McDuff, D., Curran, T. & Kadambi, A. Synthetic data in healthcare. Preprint at https://arxiv.org/abs/2304.03243 (2023).
Wang, Z. et al. Synthetic generation of face videos with plethysmograph physiology. In Proceedings of the IEEE/CVF Conference On Computer Vision And Pattern Recognition, 20587–20596 (2022).
McDuff, D. et al. Scamps: Synthetics for camera measurement of physiological signals. Adv. Neural Inform. Process. Syst. 35, 3744–3757 (2022).
Du, J., Liu, S.-Q., Zhang, B. & Yuen, P. C. Dual-bridging with adversarial noise generation for domain adaptive rppg estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10355–10364 (2023).
Vaswani, A. et al. Attention is all you need. Advances in neural information processing systems 30, 1-11 (2017).
Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint at https://arxiv.org/abs/1810.04805 (2018).
Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. Preprint at https://arxiv.org/abs/2010.11929 (2020).
Goodfellow, I. et al. Generative adversarial networks. Commun. ACM 63, 139–144 (2020).
Lu, H., Han, H. & Zhou, S. K. Dual-gan: Joint bvp and noise modeling for remote physiological measurement. In Proceedings of the IEEE/CVF Conference On Computer Vision And Pattern Recognition, 12404–12413 (2021).
Wu, B.-F., Chiu, L.-W., Wu, Y.-C., Lai, C.-C. & Chu, P.-H. Contactless blood pressure measurement via remote photoplethysmography with synthetic data generation using generative adversarial network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2130–2138 (2022).
Song, R. et al. Pulsegan: learning to generate realistic pulse waveforms in remote photoplethysmography. IEEE J. Biomed. Health Inform. 25, 1373–1384 (2021).
Xu, S. et al. Wireless skin sensors for physiological monitoring of infants in low-income and middle-income countries. Lancet Digital Health 3, e266–e273 (2021).
Zhu, Y. et al. Skin-interfaced electronics: a promising and intelligent paradigm for personalized healthcare. Biomaterials, 296, 1–26 (2023).
Chung, H. U. et al. Skin-interfaced biosensors for advanced wireless physiological monitoring in neonatal and pediatric intensive-care units. Nat. Med. 26, 418–429 (2020).
Atmojo, J. T., Sudaryanto, W. T., Widiyanto, A., Ernawati, E. & Arradini, D. Telemedicine, cost effectiveness, and patients satisfaction: a systematic review. J Health Policy Manag. 5, 103–107 (2020).
W.H.O. Newborns: improving survival and well-being. https://www.who.int/news-room/fact-sheets/detail/newborns-reducing-mortality, (World Health Organization, 19 Sep., 2020).
Brocklehurst, P. et al. Computerised interpretation of fetal heart rate during labour (infant): a randomised controlled trial. Lancet 389, 1719–1729 (2017).
Anton, O. et al. Heart rate monitoring in newborn babies: a systematic review. Neonatology 116, 199–210 (2019).
Wyllie, J. et al. Part 11: neonatal resuscitation: 2010 international consensus on cardiopulmonary resuscitation and emergency cardiovascular care science with treatment recommendations. Resuscitation 81, e260–e287 (2010).
Wyllie, J. et al. European resuscitation council guidelines for resuscitation 2015: Section 7. resuscitation and support of transition of babies at birth (2015).
Sahoo, N. N. et al. Deep learning based non-contact physiological monitoring in neonatal intensive care unit. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 1327–1330 (IEEE, 2022).
Lyra, S. et al. Camera fusion for real-time temperature monitoring of neonates using deep learning. Medi. Biol. Eng. Comput. 60, 1787–1800 (2022).
Kuan, P. X. et al. Efficacy of telemedicine for the management of cardiovascular disease: a systematic review and meta-analysis. Lancet Digital Health 4, e676–e691 (2022).
Reports, F. S. M. R. Advanced non-contact patient monitoring technologies: A new paradigm in healthcare monitoring. https://www.marketresearch.com/Frost-Sullivan-v383/Advanced-Non-contact-Patient-Monitoring-1293554/, (Frost & Sullivan, May, 2020).
Saeed, U. et al. Machine learning empowered covid-19 patient monitoring using non-contact sensing: an extensive review. J. Pharm. Analysis (2022).
Zhao, R. et al. Visual content privacy protection: A survey. Preprint at https://arxiv.org/abs/2303.16552 (2023).
Sun, Z. & Li, X. Privacy-phys: facial video-based physiological modification for privacy protection. IEEE Signal Process. Lett. 29, 1507–1511 (2022).
Konečny`, J., McMahan, H. B., Ramage, D. & Richtárik, P. Federated optimization: Distributed machine learning for on-device intelligence. Preprint at https://arxiv.org/abs/1610.02527 (2016).
Yang, Q., Liu, Y., Chen, T. & Tong, Y. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol. 10, 1–19 (2019).
Dayan, I. et al. Federated learning for predicting clinical outcomes in patients with covid-19. Nat. Med. 27, 1735–1743 (2021).
Monitoring, P. Guest editorial federated learning for privacy preservation of healthcare data in internet of medical things and patient monitoring. IEEE J. Biomed. Health Inform. 27, 648–651 (2023).
Pati, S. et al. Federated learning enables big data for rare cancer boundary detection. Nat. Commun. 13, 7346 (2022).
Grooby, E. et al. Neonatal face and facial landmark detection from video recordings. Preprint at https://arxiv.org/abs/2302.04341 (2023).
Dosso, Y. S., Kyrollos, D., Greenwood, K. J., Harrold, J. & Green, J. R. Nicuface: robust neonatal face detection in complex nicu scenes. IEEE Access 10, 62893–62909 (2022).
Rehouma, H., Noumeir, R., Essouri, S. & Jouvet, P. Quantitative assessment of spontaneous breathing in children: evaluation of a depth camera system. IEEE Transa. Instrum. Measur. 69, 4955–4967 (2019).
Unursaikhan, B. et al. Development of a novel vital-signs-based infection screening composite-type camera with truncus motion removal algorithm to detect covid-19 within 10 seconds and its clinical validation. Front. Physiol. 13, 1–10 (2022).
Villarroel, M. et al. Non-contact vital-sign monitoring of patients undergoing haemodialysis treatment. Sci. Rep. 10, 1–21 (2020).
Villarroel, M. et al. Continuous non-contact vital sign monitoring in neonatal intensive care unit. Healthcare Technol. Lett. 1, 87–91 (2014).
Chaichulee, S. et al. Cardio-respiratory signal extraction from video camera data for continuous non-contact vital sign monitoring using deep learning. Physiol. Measur. 40, 115001 (2019).
Chaichulee, S. et al. Localised photoplethysmography imaging for heart rate estimation of pre-term infants in the clinic. In Optical diagnostics and sensing XVIII: toward point-of-care diagnostics, vol. 10501, 146–159 (SPIE, 2018).
Liu, Z. et al. Contactless respiratory rate monitoring for icu patients based on unsupervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6004–6013 (2023).
Lyra, S. et al. A deep learning-based camera approach for vital sign monitoring using thermography images for icu patients. Sensors 21, 1495 (2021).
Pediaditis, M. et al. Contactless respiratory rate estimation from video in a real-life clinical environment using eulerian magnification and 3d cnns. In 2022 IEEE International Conference on Imaging Systems and Techniques (IST), 1–6 (IEEE, 2022).
Liu, L. et al. Information-enhanced network for noncontact heart rate estimation from facial videos. IEEE Transactions on Circuits and Systems for Video Technology (2023).
He, H., Yuan, Y., Chen, Y.-C., Cao, P. & Katabi, D. Contactless oxygen monitoring with gated transformer. NeurIPS 2022 Workshop on Learning from Time Series for Health. 1–19 (2022).
Qian, W., Guo, D., Li, K., Tian, X. & Wang, M. Dual-path tokenlearner for remote photoplethysmography-based physiological measurement with facial videos. Preprint at https://arxiv.org/abs/2308.07771 (2023).
Lee, E., Chen, E. & Lee, C.-Y. Meta-rppg: Remote heart rate estimation using a transductive meta-learner. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16, 392–409 (Springer, 2020).
Yue, Z., Ding, S., Yang, S., Wang, L. & Li, Y. Multimodal information fusion approach for noncontact heart rate estimation using facial videos and graph convolutional network. IEEE Trans. Instrum. Measur. 71, 1–13 (2021).
Shao, H., Luo, L., Chen, S., Hu, C. & Yang, J. Hyperbolic embedding steered spatiotemporal graph convolutional network for video-based remote heart rate estimation. Eng. Appl. Artificial Intell. 124, 106642 (2023).
Barde, K. et al. Measurements of pulse rate using facial video cameras from smart devices in patients diagnosed with atrial fibrillation. J. Electrocardiol. 78, 58–64 (2023).
Svoboda, L. et al. Contactless heart rate measurement in newborn infants using a multimodal 3d camera system. Front. Pediatri. 10, 1–11 (2022).
Allado, E. et al. Accurate and reliable assessment of heart rate in real-life clinical settings using an imaging photoplethysmography. J. Clin.Med. 11, 6101 (2022).
Allado, E. et al. Remote photoplethysmography is an accurate method to remotely measure respiratory rate: a hospital-based trial. J. Clin. Med. 11, 3647 (2022).
Zeng, Y., Song, X., Chen, H., Huang, W. & Wang, W. A multi-modal clinical dataset for critically-ill and premature infant monitoring: Eeg and videos. In 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), 1–5 (IEEE, 2022).
Wang, H., Huang, J., Wang, G., Lu, H. & Wang, W. Surveillance camera-based cardio-respiratory monitoring for critical patients in icu. In 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), 1–4 (IEEE, 2022).
Ottaviani, V. et al. Contactless monitoring of breathing pattern and thoracoabdominal asynchronies in preterm infants using depth cameras: A feasibility study. IEEE J. Transl. Eng. Health Med. 10, 1–8 (2022).
Hajj-Ali, Z., Greenwood, K., Harrold, J. & Green, J. R. Towards depth-based respiratory rate estimation with arbitrary camera placement. In 2022 IEEE International Symposium on Medical Measurements and Applications (MeMeA), 1–6 (IEEE, 2022).
Nagy, Á., Földesy, P., Jánoki, I., Siket, M. & Zarándy, Á. Automatic annotation to train roi detectionalgorithm for premature infant respirationmonitoring in nicu. Available at SSRN 4166553.
Varma, M. et al. Contactless monitoring of respiratory rate (rr) and heart rate (hr) in non-acuity settings: a clinical validity study. BMJ open 12, e065790 (2022).
Chen, Q. et al. Camera-based heart rate estimation for hospitalized newborns in the presence of motion artifacts. BioMed. Eng. OnLine 20, 1–16 (2021).
Kyrollos, D. G., Tanner, J. B., Greenwood, K., Harrold, J. & Green, J. R. Noncontact neonatal respiration rate estimation using machine vision. In 2021 IEEE Sensors Applications Symposium (SAS), 1–6 (IEEE, 2021).
Khanam, F.-T.-Z., Perera, A. G., Al-Naji, A., Gibson, K. & Chahl, J. Non-contact automatic vital signs monitoring of infants in a neonatal intensive care unit based on neural networks. J. Imaging 7, 122 (2021).
Laurie, J., Higgins, N., Peynot, T., Fawcett, L. & Roberts, J. An evaluation of a video magnification-based system for respiratory rate monitoring in an acute mental health setting. Int. J. Med. Inform. 148, 104378 (2021).
Yu, X., Laurentius, T., Bollheimer, C., Leonhardt, S. & Antink, C. H. Noncontact monitoring of heart rate and heart rate variability in geriatric patients using photoplethysmography imaging. IEEE J. Biomed. Health Inform. 25, 1781–1792 (2020).
Malafaya, D., Domingues, S. & Oliveira, H. P. Domain adaptation for heart rate extraction in the neonatal intensive care unit. In 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 1082–1086 (IEEE, 2020).
Chen, Q. et al. Non-contact heart rate monitoring in neonatal intensive care unit using rgb camera. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 5822–5825 (IEEE, 2020).
Imano, W. et al. Non-contact respiratory measurement using a depth camera for elderly people. Sensors 20, 6901 (2020).
Paul, M. et al. Non-contact sensing of neonatal pulse rate using camera-based imaging: a clinical feasibility study. Physiol. Measur. 41, 024001 (2020).
Negishi, T. et al. Contactless vital signs measurement system using rgb-thermal image sensors and its clinical screening test on patients with seasonal influenza. Sensors 20, 2171 (2020).
Slapnicar, G., Dovgan, E., Cuk, P. & Lustrek, M. Contact-free monitoring of physiological parameters in people with profound intellectual and multiple disabilities. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 0–0 (2019).
Antognoli, L. et al. Enanced video heart rate and respiratory rate evaluation: standard multiparameter monitor vs clinical confrontation in newborn patients. In 2019 IEEE International Symposium on Medical Measurements and Applications (MeMeA), 1–5 (IEEE, 2019).
Rasche, S. et al. Remote photoplethysmographic assessment of the peripheral circulation in critical care patients recovering from cardiac surgery. Shock 52, 174–182 (2019).
Antognoli, L., Marchionni, P., Nobile, S., Carnielli, V. P. & Scalise, L. Assessment of cardio-respiratory rates by non-invasive measurement methods in hospitalized preterm neonates. In 2018 IEEE International Symposium on Medical Measurements and Applications (MeMeA), 1–5 (IEEE, 2018).
Trumpp, A. et al. Camera-based photoplethysmography in an intraoperative setting. Biomed. Eng. Online 17, 1–19 (2018).
Cobos-Torres, J.-C., Abderrahim, M. & Martínez-Orgado, J. Non-contact, simple neonatal monitoring by photoplethysmography. Sensors 18, 4362 (2018).
Blanik, N. et al. Remote vital parameter monitoring in neonatology–robust, unobtrusive heart rate detection in a realistic clinical scenario. Biomed. Eng./Biomedizinische Technik 61, 631–643 (2016).
Sikdar, A., Behera, S. K., Dogra, D. P. & Bhaskar, H. Contactless vision-based pulse rate detection of infants under neurological examinations. In 2015 37th annual international conference of the IEEE engineering in medicine and biology society (EMBC), 650–653 (IEEE, 2015).
Aarts, L. A. et al. Non-contact heart rate monitoring utilizing camera photoplethysmography in the neonatal intensive care unit-a pilot study. Early Human Dev. 89, 943–948 (2013).
Scalise, L., Bernacchia, N., Ercoli, I. & Marchionni, P. Heart rate measurement in neonatal patients using a webcamera. In 2012 IEEE International Symposium on Medical Measurements and Applications Proceedings, 1–4 (IEEE, 2012).
Xi, L., Wu, X., Chen, W., Wang, J. & Zhao, C. Weighted combination and singular spectrum analysis based remote photoplethysmography pulse extraction in low-light environments. Med. Eng. Phys. 105, 103822 (2022).
Speth, J., Vance, N., Flynn, P. & Czajka, A. Non-contrastive unsupervised learning of physiological signals from video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14464–14474 (2023).
Yu, Z. et al. Physformer++: Facial video-based physiological measurement with slowfast temporal difference transformer. Int. J. Comput.Vision 131, 1307–1330 (2023).
Gwak, M., Vatanparvar, K., Kuang, J. & Gao, A. Motion-based respiratory rate estimation with motion artifact removal using video of face and upper body. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 1961–1967 (IEEE, 2022).
Cheng, J. et al. Motion-robust respiratory rate estimation from camera videos via fusing pixel movement and pixel intensity information. IEEE Transactions on Instrumentation and Measurement (2023).
Xing, W., Shi, Y., Wu, C., Wang, Y. & Wang, X. Predicting blood pressure from face videos using face diagnosis theory and deep neural networks technique. Computers in Biology and Medicine 107112 (2023).
Acknowledgements
This work is supported by the National Natural Science Foundation of China (62203037 and 62271241), the National Key R&D Program of China (2022YFC2407800), Guangdong Basic and Applied Basic Research Foundation (2023A1515012983), and Shenzhen Fundamental Research Program (JCYJ20220530112601003). The authors would like to thank Freepik for providing some figures.
Author information
Authors and Affiliations
Contributions
B.H., W.W., Z.L. conceived and designed the study. B.H., S.H., C.-L.L., J.S., L.W. and W.W. discussed the clinical application. B.H. and Z.L. conducted the literature search with support from S.H., C.Z., J.S., C.Z., and L.W.. B.H., S.H., Z.L., C.-L.L., J.S., C.Z., L.W. and W.W. wrote and revised the paper. All authors approved the final manuscript.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Huang, B., Hu, S., Liu, Z. et al. Challenges and prospects of visual contactless physiological monitoring in clinical study. npj Digit. Med. 6, 231 (2023). https://doi.org/10.1038/s41746-023-00973-x
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41746-023-00973-x