Abstract
Accurate detection and tracking of targets in low-light and complex scenarios is essential for the development of intelligent machine vision. However, such capabilities are difficult to achieve using conventional static optoelectronic convolutional processing. Here we show that in-sensor dynamic computing can be used for accurate detection and robust tracking of dim targets. The approach uses multiple-terminal mixed-dimensional graphene–germanium heterostructure device arrays and relies on the dynamic correlation of adjacent optoelectronic devices in the array. The photoresponse of the devices can range from positive to negative depending on the drain–source voltage polarity and can be further tailored using the back-gate and top-gate voltage. The correlation characteristic of the device array can be used to selectively amplify small differences in light intensity and to accurately extract edge features of dim targets. We show that the approach can provide robust tracking of dim targets in complex environments.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
Source data are provided with this paper. All other data that support the plots within this paper and other findings of this study are available from the corresponding authors upon reasonable request.
Code availability
The source codes used for simulation and data plotting are available from the corresponding authors upon reasonable request.
References
Wang, X. et al. Infrared dim target detection based on visual attention. Infrared Phys. Technol. 55, 513–521 (2012).
Dudek, P. et al. Sensor-level computer vision with pixel processor arrays for agile robots. Sci. Robot. 7, eabl7755 (2022).
Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).
Ma, C. et al. Intelligent infrared sensing enabled by tunable moiré quantum geometry. Nature 604, 266–272 (2022).
Zhou, F. & Chai, Y. Near-sensor and in-sensor computing. Nat. Electron. 3, 664–671 (2020).
Chai, Y. In-sensor computing for machine vision. Nature 579, 32–33 (2020).
Liang, S. & Miao, F. Lego-like reconfigurable AI chips. Nat. Electron. 5, 327–328 (2022).
Zhang, Z. et al. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat. Nanotechnol. 17, 27–32 (2021).
Liao, F. et al. Bioinspired in-sensor visual adaptation for accurate perception. Nat. Electron. 5, 84–91 (2022).
Wan, T. et al. In-sensor computing: materials, devices, and integration technologies. Adv. Mater. 35, 2203830 (2023).
Choi, C. et al. Reconfigurable heterogeneous integration using stackable chips with embedded artificial intelligence. Nat. Electron. 5, 386–393 (2022).
Wang, C.-Y. et al. Gate-tunable van der Waals heterostructure for reconfigurable neural network vision sensor. Sci. Adv. 6, eaba6173 (2020).
Wang, S. et al. Networking retinomorphic sensor with memristive crossbar for brain-inspired visual perception. Natl Sci. Rev. 8, nwaa172 (2020).
Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).
Pi, L. et al. Broadband convolutional processing using band-alignment-tunable heterostructures. Nat. Electron. 5, 248–254 (2022).
Jang, H. et al. An atomically thin optoelectronic machine vision processor. Adv. Mater. 32, 2002431 (2020).
Zhu, Q.-B. et al. A flexible ultrasensitive optoelectronic sensor array for neuromorphic vision systems. Nat. Commun. 12, 1798 (2021).
Wu, P. et al. Next-generation machine vision systems incorporating two-dimensional materials: progress and perspectives. InfoMat 4, e12275 (2021).
Wang, S. et al. Nonvolatile van der Waals heterostructure phototransistor for encrypted optoelectronic logic circuit. ACS Nano 16, 4528–4535 (2022).
Choi, C. et al. Curved neuromorphic image sensor array using a MoS2-organic heterostructure inspired by the human visual recognition system. Nat. Commun. 11, 5934 (2020).
Liang, S.-J., Cheng, B., Cui, X. & Miao, F. Van der Waals heterostructures for high-performance device applications: challenges and opportunities. Adv. Mater. 32, 1903800 (2020).
Wang, C.-Y. et al. 2D layered materials for memristive and neuromorphic applications. Adv. Electron. Mater. 6, 1901107 (2020).
Koppens, F. H. L. et al. Photodetectors based on graphene, other two-dimensional materials and hybrid systems. Nat. Nanotechnol. 9, 780–793 (2014).
Akinwande, D. et al. Graphene and two-dimensional materials for silicon technology. Nature 573, 507–518 (2019).
Bae, S.-H. et al. Integration of bulk materials with two-dimensional materials for physical coupling and applications. Nat. Mater. 18, 550–560 (2019).
Liu, Y. et al. Van der Waals heterostructures and devices. Nat. Rev. Mater. 1, 16042 (2016).
Liu, C. et al. Two-dimensional materials for next-generation computing technologies. Nat. Nanotechnol. 15, 545–557 (2020).
Bao, Q. & Loh, K. P. Graphene photonics, plasmonics, and broadband optoelectronic devices. ACS Nano 6, 3677–3694 (2012).
Zeng, L.-H. et al. Monolayer graphene/germanium Schottky junction as high-performance self-driven infrared light photodetector. ACS Appl. Mater. Interfaces 5, 9362–9366 (2013).
Yang, F. et al. Ultrathin broadband germanium-graphene hybrid photodetector with high performance. ACS Appl. Mater. Interfaces 9, 13422–13429 (2017).
Chang, K. E. et al. High-responsivity near-infrared photodetector using gate-modulated graphene/germanium Schottky junction. Adv. Electron. Mater. 5, 1800957 (2019).
Pan, C. et al. Reconfigurable logic and neuromorphic circuits based on electrically tunable two-dimensional homojunctions. Nat. Electron. 3, 383–390 (2020).
Walkowski, S. et al. Quality evaluation of virtual slides using methods based on comparing common image areas. Diagnostic Pathol. 6, S14 (2011).
Acknowledgements
This work was supported in part by the National Key R&D Program of China under grant no. 2023YFF1203600, the National Natural Science Foundation of China (grant nos. 62122036, 62034004, 61921005, 62304104 and 62204119), the Natural Science Foundation of Jiangsu Province (grant no. BK20210337) and the Strategic Priority Research Program of the Chinese Academy of Sciences (grant no. XDB44000000). F.M. acknowledges the support from the AIQ foundation and e-Science Center of the Collaborative Innovation Center of Advanced Microstructures. The microfabrication center of the National Laboratory of Solid State Microstructures (NLSSM) is also acknowledged for their technical support.
Author information
Authors and Affiliations
Contributions
Y.Y., S.-J.L. and F.M. conceived the concept and designed the experiments. S.-J.L. and F.M. supervised the whole project. Y.Y., Y.L., Z.-A.L. and S.W. prepared the heterostructures and fabricated the devices. Y.Y., P.W. and W.Y. performed device characterization and optoelectronic measurements. Y.Y., P.W. and C.P. performed in-sensor dynamic processing with the devices. C.P. and X.Y. devised and prepared circuits. G.L. and Z.D. provided GeOI substrates and performed Raman characterization. Y.Y., C.P., Y.L., S.-J.L. and B.C. analysed the data and carried out simulations. Y.Y., S.-J.L. and F.M. wrote the paper with input from all other authors.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Electronics thanks Houk Jang, Walterio Mayol-Cuevas and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 Comparison of simulation results between in-sensor dynamic computing and conventional optoelectronic convolution.
For conventional optoelectronic convolution, the convolutional kernel is standard Laplace kernel: \(\left[\begin{array}{ccc}1 & 0 & 1\\ 0 & -4 & 0\\ 1 & 0 & 1\end{array}\right]\), which can be regarded as the second derivative of the space and widely utilized for edge feature extraction in image processing. In traditional optoelectronic convolution processing, the kernel is static throughout the image processing. For in-sensor dynamic computing, the center element of the kernel is dynamically changed according to the local spatial light intensity gradient. All the images exhibit same greyscale range from 0 to 255.
Extended Data Fig. 2 Overall structure for large-scale integration of dynamic kernel arrays.
Schematic of circuit structure including arrays of the dynamic kernel displayed in Fig. 4c (light blue box) and readout circuits of each column (light red box). The readout circuit comprises two subtractors for calculating photocurrents of passive devices and active device separately. The multiplexer is used for selecting the dark current of active device according to the top gate voltage. The final output Vout can be directly readout from the capacitor integration circuit and then reset the capacitor for next image capture. The time sequences of each signal in a single image capture are also provided.
Extended Data Fig. 3 Transistor level layout of one dynamic kernel and one readout circuit as displayed in the black dashed rectangle in Extended Data Fig. 2.
Voltage sources (VP1, VP3, VP6, VP8) proportional to the light intensity shed on passive devices (DP1, DP3, DP6, DP8) are used for determining the top gate voltage of active device (VTG-A). Current sources (∑Ilight-Pi and Ilight-A) representing electrical outputs of passive and active devices are subtracted by the corresponding dark currents and fed into the capacitor integration circuit for final output (Vout).
Extended Data Fig. 4 Simulation results for potential peripheral electronics in Extended Data Fig. 3.
a, The changes of VTG-A with VP3 – VP6. if |VP3 – VP6| exceeds a threshold, VTG-A will change from VL to VH. b, Iph-t and Vout-t in one image capture for two cases. The whole exposure for one row is divided into four stages: add up photocurrent of all passive devices (0-10 μs), complete photocurrent integration of the active device (10-20 μs), hold and read out Vout (20-30 μs), reset the capacitive readout circuit (30-40 μs).
Extended Data Fig. 5 Simulation results of feature maps by using in-sensor dynamic correlated processing and conventional convolutional processing.
The original images of human faces from three different persons (denote as type I, type II and type III) exhibit varying-contrast feature. It is difficult to extract the edge features from images with lower contrast based on conventional method, while the proposed in-sensor dynamic computing allows for accurate edge feature extraction regardless of the image contrast.
Supplementary information
Supplementary Information
Supplementary Figs. 1–7.
Supplementary Video 1
The video of an unmanned aerial vehicle (UAV) flying through an alley is displayed in the file ‘UAV video.mp4’. Six frames are extracted from the video as the red dashed boxes denoted. The image contrast of each frame is different. We carried out in-sensor dynamic computing and traditional optoelectronic convolution processing for the six frames. The UAV can be tracked at all times by using our proposed in-sensor dynamic computing. In contrast, the UAV cannot be tracked all the time by using traditional optoelectronic convolution processing. This indicates that our proposed computing technology is promising in the robust tracking of the moving targets in complex visual scenes.
Supplementary Data
Statistical source data.
Source data
Source Data Fig. 2
Statistical source data.
Source Data Fig. 3
Statistical source data.
Source Data Fig. 4
Statistical source data.
Source Data Fig. 5
Statistical source data.
Source Data Extended Data Fig. 4
Statistical source data.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Yang, Y., Pan, C., Li, Y. et al. In-sensor dynamic computing for intelligent machine vision. Nat Electron 7, 225–233 (2024). https://doi.org/10.1038/s41928-024-01124-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41928-024-01124-0
