The problem

The need to accurately detect and extract the key features of targets, such as edges and contours, in dim lighting poses a formidable challenge for imaging technologies. This challenge is particularly pronounced when the light intensity contrast between the target and its surroundings is minimal. To robustly recognize and track targets in low-contrast environments, these features need to be precisely identified. Imaging dim targets with traditional silicon-based complementary metal–oxide–semiconductor (CMOS) or charge-coupled device (CCD) image sensors involves extending exposure times or globally increasing the sensitivity or gain of all pixels1. Unfortunately, such measures decrease the sensor speed and introduce more noise into the image. Moreover, CMOS- and CCD-based image sensors are unable to selectively amplify the weak features of targets in low-contrast scenarios directly within the sensor. Instead, each pixel has a fixed photoresponsivity and is independently operated. Therefore, all the pixels respond uniformly to light, which limits the adaptability and performance of conventional image sensors in complex lighting conditions.

The solution

In-sensor computing, which refers to efficient information processing within the sensor, can be used to solve this challenge. Our goal was to develop a computational optoelectronic sensor with interpixel correlation to effectively detect dim targets. We created computational units composed of adjacent intercorrelated pixels, including peripheral passive pixels and a central active pixel (Fig. 1a). The passive pixels have a fixed and identical responsivity, whereas the responsivity of the active pixel is modulated according to the photocurrent output from its neighbours. The correlation between the passive and active pixels allows the responsivity of the unit to be dynamically tuned by the local image gradient. Thus, the computational unit can also be regarded as a dynamic kernel. For the active pixel we used a photosensitive graphene–Ge heterostructure with a top and bottom dual-gate architecture (Fig. 1b). The basic responsivity of the active pixel is set by the back gate while the top gate is used for the feedback regulation of the responsivity in accordance with the local image gradient.

Fig. 1: A conceptual illustration and experimental demonstration of pixel-correlated computing.
figure 1

a, A schematic of pixel-correlated computing using passive and active optoelectronic pixels. b, A schematic of the graphene–Ge heterostructure used to form the active pixel. VTG and VBG are the top-gate and back-gate voltages, respectively. c, Images of a person standing in a dim corridor obtained with a camera (left) and processed by the proposed computational sensor (right). Iout is the total photocurrent output from the active and passive pixels. The red dashed box indicates the location of the person in the image, demonstrating that the in-sensor dynamic computing approach can extract the edge profile. h-BN, hexagonal boron nitride. © 2024, Yang, Y. et al.

Full size image

We then tested the ability of this sensor to extract the edge features of a dim target — a person standing in a dark corridor (Fig. 1c). We compared these results with another processing technique that uses in-sensor optoelectronic convolutional processing with responsivity-fixed filter kernels2. The in-sensor pixel-correlated computing approach could extract the profile of the person, whereas the optoelectronic convolutional approach could not, demonstrating the ability of our proposed approach to selectively amplify edge features. This selective enhancement enables the neural network backends to robustly recognize targets with high precision. Our computational sensor was also able to accurately track an unmanned aerial vehicle in dim conditions, demonstrating its ability to robustly extract features from images with varying contrast and complex visual environments.

Future directions

Our interpixel correlation approach not only addresses the long-standing scientific challenge of robustly recognizing and tracking dim targets but also has broader implications. For example, by enabling enhanced feature extraction directly within the sensor, our method facilitates robust and high-precision recognition with fast convergence rates in backend processors, particularly neural networks. The dynamic control and correlated programming of the active pixels has the potential to allow convolutional neural network backpropagation approaches to be incorporated into in-sensor dynamic computing. However, this study does not delve into the specifics of how such incorporation would be achieved. Additionally, the CMOS compatibility of graphene–Ge optoelectronic devices could enable large-scale on-chip integration3,4; however, further work is needed to explore the practical implementation and scalability of the device for real-world applications, as well as potential challenges that could arise for such applications.

The next steps for this research will involve validating the scalability of our approach through large-scale on-chip integration. Extending the visible detection wavelength to 1,550 nm or even mid-infrared wavelengths is another logical progression, to broaden the applicability of our proposed technology to various low-contrast scenarios, including medical imaging, remote sensing, early warning systems, monitoring and security. We also hope to explore the potential of using intrinsic physical characteristics, such as the polarization of light, for dim target detection5.

Yuekun Yang, Shi-Jun Liang & Feng Miao

Nanjing University, Nanjing, China.

Expert opinion

“This paper demonstrates an advanced form of in-sensor processing technology. It applies an algorithm that interacts with the displayed view to further enhance the processing outcomes. This technique exhibits superior precision in edge detection, particularly for dim images, compared to the non-interactive traditional in-sensor processing methods. Moreover, the dynamic control inherent in this approach holds the potential to incorporate backpropagation into in-sensor processing, making it a highly scalable technique.” Houk Jang, Brookhaven National Laboratory, Upton, NY, USA.

Behind the paper

Since 2020, in-sensor computing has attracted increasing research interest. This approach enhances the functionality and capability of sensors by increasing the nonlinearity, adaptability and interpixel correlation of the physical process of photonic-to-electric conversion.

Throughout his time as a doctoral student at the Shanghai Institute of Microsystem and Information Technology, Y.Y. worked on nanomaterial preparation and optoelectronic device design and fabrication. In 2020, Y.Y. joined F.M.’s group at Nanjing University as a postdoctoral researcher where he has continued to explore functional photosensors and intelligent vision systems. Y.Y. had the initial idea for this work when he studied convolutional neural networks. He wanted to find out what would happen if the weight of the convolutional kernel is varied according to the local image greyscale. Based on this question, he devoted his efforts to achieving the hardware implementation and exploring suitable applications. S.-J.L. & F.M.

From the editor

“This work shows that dynamic in-sensor computing can selectively enhance local light intensity gradients, allowing the edge features of images to be enhanced. It is a clever use of the tunable optoelectronic properties of mixed-dimensional heterostructures. The technology can be used to track targets in low-contrast scenarios.” Katharina Zeissler, Associate Editor, Nature Electronics.