A bioinspired tactile scanner for computer haptics

Abstract

Computer haptics (CH) is about integration of tactile sensation and rendering in Metaverse. However, unlike computer vision (CV) where both hardware infrastructure and software programs are well developed, a generic tactile data capturing device that serves the same role as what a camera does for CV, is missing. Bioinspired by electrophysiological processes in human tactile somatosensory nervous system, here we propose a tactile scanner along with a neuromorphically-engineered system, in which a closed-loop tactile acquisition and rendering (re-creation) are preliminarily achieved. Based on the architecture of afferent nerves and intelligent functions of mechano-gating and leaky integrate-and-fire models, such a tactile scanner is designed and developed by using piezoelectric transducers as axon neurons and thin film transistor (TFT)-based neuromorphic circuits to mimic synaptic behaviours and neural functions. As an example, the neuron-like tactile information of surface textures is captured and further used to render the texture friction of a virtual surface for “recreating” a “true” feeling of touch.

Introduction

Virtual world and space in the name of Metaverse are becoming a unique technical reality in parallel to our real environment and society. Anything we do in the real world can find a digital twin in Metaverse. Taking a virtual basketball game as an example (Fig. 1), to create such a virtual space for playing, all scenarios (information), such as visual, sound, and haptics, have to be rendered from real data in the real world. A greater challenge remains to enable virtual interaction with visual, auditory, and tactile sensations in the event of playing while players are facilitated by computing wearables such as virtual reality (VR) glasses and haptic gloves (Fig. 1a). Creations of visual and auditory sensations can be achieved by cameras, displays, and acoustic devices with the help of computer vision (CV) and computer audition (CA) technologies. For instance, today’s cameras and microphones (Fig. 1b) can capture a scene with fine details in some cases even better than our human vision and auditory systems, followed by using signal processing algorithms to rebuild a virtual gaming environment full of digital contents in both visual and audial effects. However, in order to “play” basketball, haptic rendering as a process to impose desired tactile sensory stimuli such as “holding” and “passing” the basketball is another “have-to-do” task. In addition, the player has to “feel” the virtual basketball that contains physical attributes such as shape, texture, surface roughness, etc.

Fig. 1: Role of tactile scanner in computer haptics.

a Playing basketball in Metaverse: a player is immersed in the virtual space but still uses eyes to observe dynamics of the scene, ears to hear communication among players, and hands to hold and pass the basketball. To enable this, virtual reality technology that brings visual, auditory, or tactile sensations and feedback to the player is a must, and wearable VR headsets and smart gloves are used to make human-virtual space interaction possible. b Analogous to human eye, ear, and hand, camera, microphone, and tactile scanner are fundamental “artificial sensory organs” to that collect data and help build the virtual scene with digital contents that can stimulate human sensory systems such as vision, auditory, and haptics. c Haptic glove embedding actuators are used to render tactile sensation. d Essentially, in the tactile scanner, an array of tactile sensors with a certain spatial resolution acquires tactile information from vibrotactile stimuli when scanning the surface. e Relative to tactile sensing, tactile rendering is a process to “recreate” a tactile sensation based on tactile information collected by the scanner. An array of actuators in the haptic system (e.g., haptic glove) generates vibrotactile stimuli for the payer to have a real “feeling” of the virtual basketball (Avatar).

Haptic rendering is one domain of computer haptics (CH), an emerging technological field, providing not only tactile sensation but also tactile feedback when interacting with a virtual object in Metaverse. Such a virtual interaction is generally required for a wide range of Metaverse applications, including entertainment¹, education², telemedicine³, and manufacturing⁴ to enhance the immersive experience. Since CH has a bi-directional interaction characteristic resulting in a closed-loop path including⁵: First, a tactile sensor or scanner is used to acquire tactile information of a physical object such as a basketball, and to quantify tactile information for spatiotemporal transformation and storage. Second, haptic rendering models and algorithms, including a hierarchical database for storing geometrical and material properties of objects, are developed to compute tactile sensation. Third, a haptic rendering system such as a haptic smart glove obtains tactile information from datasets, executes haptic-rendering algorithms, and feeds motor-like driving signals to an actuator embedded in the glove for generating force stimuli to realize haptic re-creation and tactile feedback.

Analogous to CV or CA, CH requires both hardware infrastructure, such as tactile data capturing devices, and software, such as rendering programs, to develop itself as a viable and mature field. As the first step, acquiring authentic skin-like tactile data is as crucial as that CV uses an eye-like camera to capture human vision-like images, and CA utilizes an ear-like audio recorder to capture sounds and voices.

To mimic both functionalities and properties of human skin, researchers have investigated a variety of artificial electronic skin-like devices and tactile sensors using various materials such as carbon nanotubes⁶, nanowires⁷, graphene⁸, and nanocomposite materials⁹, as well as micro-nano structures, such as pyramids^10,11, micro-domes^12,13, microcone^14,15 and interlocking micro-structures^16,17. Some of these devices have demonstrated sufficient sensitivity, responsivity and dynamic range as compared to human skin. However, adoption for mass production is not yet ready. Based on neuromorphic devices such as synaptic electrolyte gated transistor¹⁸, ferroelectric transistor¹⁹, memristors²⁰, artificial neuromorphic somatosensory systems have been proposed and showcased tactile perception and feedback functions even though they are still in a very early stage. For example, a bioinspired afferent nervous system integrated with pressure sensors and synaptic devices is capable of pressure and vibration detection as well as perceiving sensation and feedback on the stimuli with a closed-loop human-machine interaction architecture^{21,22,23,24,25,26} (Supplementary Table 1). The artificial neural tactile sensation of naturalistic textures in the field of hand prostheses were notable technological advances with naturalistic tactile stimuli and equipped with sensorimotor loop^27,28 (Supplementary Table 2).

However, there still lacks a generic tactile data capturing device for CH, namely “tactile scanner” (Fig. 1b), as a hand-like tool to collect neuron-like signals that feed a haptic computing system for tactile perception. In the case of “playing” virtual basketball, such a tactile scanner must be able to transform mechanical stimuli into neuron-like vibrotactile signals (Fig. 1d), followed by a wearable glove-like actuator converting them into force stimuli for the player to feel and experience while playing (Fig. 1e).

In particular, this work aims to address the design and deployment of such an artificial tactile scanner that follows neuro-physiological models to process tactile stimuli information with naturalistic tactile features. The tactile scanner employs piezoelectric transducer to mimic the mechano-gating model of mechanoreceptor, utilizes synaptic TFT to perform first-order non-linear function to preprocess tactile input, and applies the Leaky integrate-and-fire (LIF) neuron model to enable encoding dynamic tactile stimuli. Moreover, a closed-loop bioinspired interactive haptic system that includes tactile acquisition and haptic rendering has also been demonstrated. This work serves as a preparatory step toward the biomimetic tactile sensory device in touch perception for CH (Supplementary Table 3).

Results

Architecture of bioinspired neuromorphic tactile system

In order to design such a bioinspired tactile scanner, we should first examine how the human skin works at the neural level. Figure 2a presents a simplified nervous system and a signal chain of human haptics upon a mechanical stimulus from “holding” a basketball. While slow adaptation (SA) such as Merkel disk (SAI) and Ruffini endings (SAII) detect static force, fast adaptation (FA) enabled by Meissner corpuscles (FAI) and Pacinian corpuscles (FAII) are more responsive to dynamic force²⁹. When a sensory neuron receives mechanical stimuli from various mechanoreceptors, pulse-like action potentials (APs) are generated and transmitted through synapses and dendrite neurons in the forms of spikes between afferent and efferent neurons in the spinal cord³⁰. Subsequently, the intermediate neurons of the cuneate nucleus, the brain’s first level of processing for tactile perception, transmit the encoded spike signal to the thalamus and cerebral somatosensory cortex, where the tactile information is decoded into sensory perception, and an action decision is made by association cortices after spike neural network computing^31,32,33,34. Then, movement-related preparatory activity will be planned in the motor cortex, and a sequence of brain waves carrying such encoded activity information will be generated and translated into motor spikes by motor-like neurons. These spikes will be further enhanced by efferent axon neurons adjacent to skeletal muscle for taking actions of “holding” a basketball³⁵.

Fig. 2: Biological vs. artificial tactile sensory nervous systems.

a Schematic diagram of biological tactile sensory nervous system and its signal chain between organs and neurons. After pre-processed in afferent neurons, the neural signals in the form of spikes enter the initial processing in the primary somatosensory cortex and are translated into the non-noxious tactile perception, which is partially segregated into association cortices for decision-making. Haptic processing for surface texture, roughness and shape mainly occurs in parietal opercular cortex and the medial occipital cortex³¹. The lateral prefrontal area might play an important cognitive role of surface roughness, whereas the bilateral parietal operculum and insula might represent the extracted sensory information of frictional properties³². The shape information goes into the intraparietal sulcus and lateral occipital complex³³, while orientation sensing is distributed across the parietal operculum, the anterior intraparietal sulcus, and the parieto-occipital region³⁴. b Schematic diagram showing the architectural hardware design of the artificial tactile sensory nervous system starting from a tactile sensor as a sensory neuron. The final stage is haptic rendering. The dashed regions indicate key circuit models that perform the analog functionalities of the biological nervous system. c Flow of neuron-like signals in each stage where spikes are used to communicate between subsystems.

Inspired by such a biological tactile signal processing procedure, an artificial neuromorphic tactile system is designed with its architecture proposed in Fig. 2b and its corresponding signal chain shown in Fig. 2c. Such an artificial haptic nervous system can be regarded as a haptics-application specific “computer”. In contrast to a conventional signal processing and computing system (Supplementary Fig. 1), such a neuromorphic system tends to accomplish similar tasks at much lower computing costs and power consumption. To build a skin-like neuromorphic system, thin film transistor (TFT) technology is considered due to its large area form factor and mass production readiness, thanks to the display industry and consumer electronics.

Bioinspired tactile scanner

Biologically, our skin uses axon-connected sensory receptors located at the distal end of primary afferent system, such as Merkel cells, to receive mechanical stimuli and process them to generate APs that travel down to a pre-synaptic neuron from SAI terminals as illustrated in Fig. 3a. When the signal is strong enough, it is transmitted further to the next neuron in the dendrite by firing and releasing neurotransmitters to a post-synaptic cell via a synapse. To help understand this process, a descriptive mechano-gating model mimicking the mechanically activated Piezo1/Piezo2 channel is presented in Fig. 3b^36,37,38,39. The mechanotransduction occurs as the mechanical force is applied through a three-bladed element to the transduction channel, suggesting that the mechanosensitive model acts as a propeller-like processor in response to force and acceleration⁴⁰ (Methods).

Fig. 3: Design and deployment of tactile scanner.

a Schematic representation of pre-synaptic, post-synaptic neurons, and synapse. b A mechano-gating model showcasing the Piezo1/Piezo2 channels are activated by compressive force (Methods). c Leaky integrate-and-fire (LIF) circuit model of a neuron. d Illustration of the neuromorphic model for a sensor unit in the tactile scanner. e Prototype of 10 × 10 axon sensory array based on piezoelectric sensors and neuromorphic circuit array formed by neuromorphic processing units (NPUs) for the tactile scanner. The axon sensory array is flexible and conformable to fit the object’s surface (Supplementary Fig. 5). f Microscopic image of the fabricated NPU based on 2T1C neuromorphic circuit with a size of 350 μm × 350 μm, TFT N1 W/L ratio of 516 µm/3.2 µm, TFT N2 W/L ratio of 102 µm/3.2 µm, and Cs = 0.1 pF.

Electrically, the Hodgkin-Huxley model (HHM), the Izhikevich model, and the LIF have been proposed to simulate the aforementioned “fire” process (Supplementary Fig. 2, Methods and Supplementary Note 1). The HHM model is more reflective for a biological neuron but too complex for practical applications^41,42. The Izhikevich model is computationally simple with rich firing patterns as biological neurons for neuroprosthetic applications⁴³, but the Implementation circuit is complex and difficult to execute for large-scale tactile sensory nervous systems. The LIF model proposed by Thorpe & Gautrais is more electrically-favorable and easily implemented where in order to simulate spike-like behavior, a neuron is treated as an electrical circuit with an ionic resistor (R_Ions), a resting voltage source (V_Rest), a membrane capacitor (C_Mem), and a current source (I_IN) as synaptic inputs (Fig. 3c)⁴⁴.

Based on these models, we design a tactile scanner to mimic the above-mentioned sensory process in the skin and simplify it into three stages: sensing, processing, receiving, and accordingly design a neuromorphic processing unit (NPU) consisting of an artificial axon (transducer/sensor), synapse (transmitter), and dendrite (receiver) to fulfill its functions. Figure 3d presents the circuitry design of the NPU and its neuromorphic circuit model including a piezoelectric transducer as an axon, a synaptic TFT as a synapse, and a TFT-based 1T1C circuit as a dendrite (Supplementary Fig. 8).

First, to design a transducer as an axon neuron, polyvinylidene fluoride (PVDF) is employed because of its piezoelectric responsivity and flexibility⁴⁵. The piezoelectric transducer, which can be symbolized as a voltage source (V_PVDF) in series with a capacitor (C_PVDF), converts the mechanical stimuli into electrical potentials (analogous to the mechano-gating model of mechanoreceptor). Next, to design a synaptic neuron, a dual-gate TFT (TFT N1) is adopted with its top gate (TG) used for synapse weight while its bottom gate (BG) is short-connected with both its drain to simulate the electrical behavior of the synapse. As shown in Fig. 3d, the equivalent channel resistance (r_syn, r_syn = 1/g_syn) can be modeled as synapse resistance, and the equivalent gate-insulator capacitance (c_syn) is regarded as cell membrane (parasitic capacitance c_tgd, c_tgs are negligible). The TG of TFT N1 modulates the threshold voltage and varies the weight factor to imitate whether the synapse is excitatory or inhibitory. Finally, the dendrite neuron is formed by a storage capacitor (C_S) to receive the accumulative signal (V_s) from the synapse, and a single gate TFT (TFT N2) is used to generate dendritic spikes (I_SPIKE) through a firing gate voltage (V_F).

To overcome the limitations of visual and auditory identification, tactile scanners should provide contact-based information such as geometry, weight, hardness, and texture. Therefore, a tactile scanner containing a 10 × 10 piezoelectric transducer array along with a 10 × 10 NPU array has been designed and fabricated (Fig. 3e). The two arrays are connected through a flexible printed circuit (FPC) board. Figure 3f gives a micrographic photo of the fabricated NPU where the TFT N1, the TFT N2, and the Cs are integrated in one unit.

Synaptic TFT as an artificial synapse

As a “messenger” between axon and dendrite, it is a synapse that determines whether tactile information should be further transmitted and processed (Fig. 4a). As schematically illustrated in Fig. 2a, a biological synapse emits neurotransmitters from pre-synaptic cell to post-synaptic cell upon arrival of APs and subsequently, spikes are fired from the post-synaptic cell (dendrite) and travel to the somatosensory cortex where tactile perception is established^{28,29,30,31,32}. As also previously discussed, a synaptic TFT N1 is designed as an artificial synapse. In a simpler approach, we use a diode-connected dual-gate TFT, where the pre-synaptic terminal (TG) is shorted with the BG and the drain terminal (Fig. 4a). It simulates the synaptic behavior in the way that positive pulses (excitatory) increase the synaptic current (I_EPSC) while negative pulses decrease it. The leakage current of the diode-connected TFT N1 in response to negative pulses is used to mimic inhibitory action of the synapse, which are less important from neuromorphic signal processing point of view⁴⁶. Such a synaptic TFT is autonomous without external biasing to enable low-power operation. Details of fabrication processes and its device characteristics can be found in Supplementary Figs. 6 and 8, respectively.

Fig. 4: Evaluation of synaptic TFT in the scanner.

a Biomimicry design of the synaptic TFT N1 including cross-sectional schematic view and equivalent device circuit model. The synaptic TFT N1 has a W/L ratio of 516 μm/3.2 μm. b Piezoelectric-generated charges (Q_PVDF) per unit time and piezoelectric voltage of PVDF transducer in the tactile scanner as a function of applied pressure. The PVDF transducer has a size of 10 mm × 10 mm in dimension and a thickness of 110 μm. c Channel conductance of the synaptic TFT N1. When the voltage of PVDF transducer is below 1.5 V, the channel conductance is below 55 nS. d Capacitance and conductance versus stimulus frequency. e △I_EPSC, defined as |I_EPSC,180us–I_EPSC,0us|, is plotted against the stimulus frequency with 799 Pa. I_EPSC,180us and I_EPSC,0us are EPSC current of tactile stimulus at different duration time, respectively. The higher frequency of the tactile stimulus, the sooner and higher it generates a spike current. f Exponential relaxation of the synaptic current after an applied tactile stimulus on the TFT N1 (as the mechanical stimulus of axon) explains the decay of channel currents as excitatory post-synaptic currents (EPSCs). g The EPSC responses stimulated by different duration time with a constant stimulus pressure of 1091 Pa. It can be seen that the EPSC current (I_EPSC) increases abruptly, followed by an exponential decay with a time constant of 132 μs (τ = 4.4 × 10⁷ Ω × 3.0 pF, when the frequency is ~2.6 kHz (~1/385 μs), the capacitance of TFT N1 is 3.0 pF, and the resistance is 4.4 × 10⁷ Ω) to its initial state (I_EPSC,i).

To evaluate the synaptic performance, we need to measure the conductance of the synaptic TFT in response to the stimulating signals from the axon-like PVDF transducer. Figure 4b gives piezoelectric-generated charges per unit time (dQ_PVDF/dt) and piezoelectric peak voltage as a function of mechanical stimuli. Here, the dQ_PVDF/dt can be regarded as an injection current to the synapse. The sensitivity of 0.25 nAl Pa⁻¹ is extracted from the quasi-linear relationship between the stimuli and dQ_PVDF/dt. Upon stimuli, the conductance of the synaptic TFT increases exponentially in line with the LIF model (Methods). When applying V_TG = V_BG = V_PVDF from 0 to 1.5 V, the conductance (g_syn) of the synaptic TFT N1 ranges from ∼0.1 nS to ∼55 nS, the same level as that of experimental values on the dendritic spine, where most excitatory synapses reside⁴⁷.

The excitatory post-synaptic currents (EPSCs) are termed to describe how the synaptic current changes with time after the stimuli. Hypothetically, the synaptic TFT can be regarded as a first-order non-linear system, and we extract the equivalent capacitance and conductance of the TFT N1 through the LCR measurements (Methods and Supplementary Fig. 8a). The I_EPSC is given quantitatively in an exponential function as follows:

$${I}_{EPSC}(t)={I}_{EPSC,\infty }+({I}_{EPSC,i}-{I}_{EPSC,\infty })\exp (-t/\tau )$$

(1)

Here, the I_EPSC,∞ refers to the steady-state current as a function of stimulus-induced: V_PVDF in Eq. (9) (Methods). I_EPSC,i refers to initial state of EPSC. The I_EPSC depends on the t and τ, where τ=r_sync_syn, r_syn = 1/g_syn, respectively and the c_syn and r_syn versus frequency are presented in Fig. 4d. A higher stimulating frequency results in a lower synaptic capacitance and a higher conductance, and in turn, the faster the neuron fires and the sooner it generates a higher spike current till it saturates (Fig. 4e). Transient electrical measurement is also performed to assess dynamics of the synaptic TFT by applying a range of dynamic stimuli. Figure 4f shows the variation of I_EPSC with a variable tactile pressure of 360-, 799-, 1091-, 2210 Pa, respectively. It is observed that the I_EPSC increases sharply as time goes by at the initial stage (set at 192.5 μs), then an exponential decay with a time constant of 132 μs to its initial state (I_EPSC,i) follows. A higher tactile stimulus leads to a qualitatively similar but more pronounced I_EPSC. Moreover, when a stimulus with a fixed magnitude of tactile pressure (1091 Pa) but prolonged duration time (38.5-, 115.5-, 192.5-, 365.5-, and 346.5 μs, respectively) is applied, the I_EPSC also increases with the duration time (Fig. 4g), implying that the I_EPSC is correlated with the behavior of short-term potentiation (STP). The aforementioned results indicate that the PVDF transducer as an artificial axon and the TFT N1 as an artificial synapse are fundamental neuromorphic components forming the frontends of tactile scanner that interface with the external force stimuli.

NPU in the tactile scanner

As previously shown in Fig. 2a, the synaptic responses are translated into encoded spikes in the cuneate neurons, which further propagate and travel to the thalamus and neocortex for perception^48,49. To build tactile intelligence, such a neural function must be conducted by neuromorphic circuits in the NPU of the tactile scanner. In our design as illustrated in Fig. 5a, after mechanical stimuli, the primary afferent responses (V_IN) of artificial synaptic TFT N1 are stored in the capacitor (C_S) mimicking the “integrate” process. The voltage (V_S) across the C_S will be compared with the threshold voltage (V_TH) set in the comparator and followed by conversion of series of firing spikes to mimic the “fire” process in the neuron (Supplementary Fig. 10a). To verify such an integrate-and-fire process by the NPU, we intentionally apply the mimicked AP signals (V_IN) with varied waveforms and durations (Methods). The V_TH used in the simulation is 255 mV ∼5-times higher than the V_TH of a biological neuron⁴⁰. While keeping the stimuli strength constant but proportionally increasing the duration time (38.5, 192.5, and 346.5 μs, respectively), the number of spikes accordingly increases, as presented in Fig. 5b. Figure 5c shows the simulated spike generation upon different stimuli with varying magnitudes but keeping constant duration time of 192.5 μs. It is found that a higher V_IN leads to a shorter firing time and a higher spike rate, which can be ascribed to the cumulative effect resulting in a prolonged integration time for spiking. Once the V_IN reaches the set V_TH for triggering I_SPIKE, the average firing rate (f_FIRING) increases with the V_IN (Fig. 5d). These results demonstrate that the I_SPIKE responding to different stimuli are highly distinguishable among each cuneate neuron. Such characteristics allow for temporal encoding in terms of time and pattern of spikes in the somatosensory cortex and association cortices inspired by sensory neurobiology⁵⁰.

Fig. 5: Design, simulation, and performance of NPU in the scanner.

a Circuitry diagram of the TFT-based NPU. b When constant voltages with different durations (38.5, 192.5, and 346.5 μs, V_IN = 1 V pulse waves with 2.65 kHz) were applied to the NPU, the number of I_SPIKE pulses (2, 11, and 21) encodes the tactile duration time. The power consumption (P) is estimated to be ~0.39 nJ/spike (P = V_DD × I_SPIKE × t, where V_DD is the power supply, t is the spike duration time). c When different voltages with constant durations (0.1-, 0.5-, 0.7-, and 1.0 V, 192.5 μs duration time) were applied to the NPU, exhibiting faster firing time and higher the I_SPIKE current response. Both peak magnitudes and numbers of I_SPIKE pulses are used to encode the input signals. d Relation between average firing fate (f_FIRING) and voltage input (V_IN). With the input V_IN ≥ V_TH, the firing rate increases. e Variation of I_SPIKE as a function of the frequency of input applied to the NPU. The V_F applies to the TFT N2 of NPU, and the I_SPIKE is amplified and converted by DAB. The tactile scanner’s acquisition rate is 1 kHz, V_F is 110 μs, t = 1/(1 kHz)–110 μs = 890 μs. Due to temporal summation, the △I_SPIKE gets enhanced because each instantaneous force stimulus-induced charges continuously inject into the C_S, resulting in a higher output. f △I_SPIKE as a function of pressure stimulus at 5 and 50 Hz, respectively. Experimentally measured output current response versus pressure (2.170−21.7 kPa, 5/50 Hz).

We further simplify the “integrate” process of the NPU in the scanner by setting V_TH at zero to process tactile input without temporal encoding, the TFT N2 of the NPU still works as a firing switch of the dendrite neuron with its output characteristics presented in Supplementary Fig. 11. In this case, the synaptic behavior of artificial synapse and the LIF model-based neuromorphic processing function are still preserved, and the I_SPIKE exhibits an exponential growth with the V_S in line with Eq. (10) (Methods and Supplementary Note 1). If high-intensity or high-frequency tactile stimulus are imposed, more charges will be accumulated in the C_S and consequently, the amplitude of I_SPIKE elevates analogous to pressure/frequency-current response exhibited by biological neuron^51,52. It can be extracted from the spike current increment (ΔI_SPIKE) rate with the applied tactile frequency (f_TACTILE) curve, and is ∼6.7 × 10⁻³nA Hz⁻¹, given a tactile stimulus of 21.7 kPa with frequency ranging from 5 Hz up to 200 Hz (Fig. 5e, and Methods). Limited by the capability of the in-house data acquisition board (DAB), the proposed scanner can only detect tactile stimuli with a frequency up to 200 Hz, lower than the human vibration detection range (<1000 Hz)⁵³. The incremental I_SPIKE is also proportional to the increased magnitude of pressure at 5 or 50 Hz, respectively (Fig. 5f). Our study shows that as a result of temporal summation, the output of the NPU is proportional to both stimulus strength and frequency, and the NPU-integrated tactile scanner can produce reliable and repeatable tactile responses (Supplementary Fig. 13). The energy consumption (P) footprint is estimated to be ∼0.39 nJ/spike at 2.65 kHz, more than 200-fold lower than that of some LIF circuits (∼100 nJ/spike at 10 kHz)⁵⁴ (Supplementary Fig. 14). In a brief summary, the NPU is promising for tactile sensing and neuromorphic signal encoding at low power consumption.

Application of tactile scanner: tactile rendering

The tactile scanner is capable of acquiring tactile information of a touch gesture or spatiotemporal distribution of an object weighting event (Supplementary Figs. 16 and 17). Before tactile rendering (TR), taking “basketball” for instance, we must capture its physical attributes such as texture, shape, and roughness. Per our previous discussion, the axon-like transducer array in the scanner is responsible for converting mechanical stimuli to electrical charges, and the NPU is in charge of processing them into neuromorphic tactile information in the form of spikes. The pre-processed tactile information of the basketball will be stored in data library as a virtual basketball. When needed, it can be restored and reloaded to realize TR in the virtual space, with help of a motor-like efferent neural device (Fig. 6a).

Fig. 6: Application of tactile scanner.

a Schematic diagram illustrating application of tactile scanner in the basketball Metaverse. The signal path starts from tactile acquisition using the scanner and ends up with tactile rendering done by haptic glove and forming the virtual basketball (Avatar). b Frequency-domain spectra of different textures from cotton, fleece, knit, metal, and wood, respectively. c The confusion matrix between real textures and predicted textures. d The encoded TA signals of wood (0−100 ms) by the NPU, and the detailed I_SPIKE,NPU from 14.5- to 16 ms (Supplementary Fig. 20). e Comparison between tactile acquisition (TA) and tactile rendering (TR) signals. The I_SPIKE-time curves obtained by the scanner when sliding on surfaces with different textures. The texture fixture moves gently on the scanner with speed (scanning speed 15 mm s⁻¹, area of 15 × 15 mm²) to fulfill one-directional scan process. In TR, after linear interpolation, conversion and amplification, the rendered tactile data typically running at rates of up to 8 ksps, is used to drive the actuator for generating the tactile stimuli. The experimental setup and test method for vibration data acquisition are shown in Supplementary Fig. 23, Supplementary Movie 3, and Supplementary Note 3, respectively. f Spearman’s correlation coefficient of sets of TA and TR information in the frequency domain to showcase level of correlation between TA and TR signals. The coefficients of $\bar{r}$ > 0.8, 0.5 < $\bar{r}$ < 0.8, 0.3 < $\bar{r}$ ≤ 0.5, $\bar{r}$ ≤ 0.3, denote a highly, moderate, low degree, no linear correlation between two variables, respectively. The value of coefficient higher than 0.5 is regarded as a relatively-high similarity.

In terms of signal processing methodology, similar to what has been done for auditory nervous system^52,55,56, it seems that human haptic system first classifies the original vibrotactile information (TA signals) in terms of frequency by different mechanoreceptors (Supplementary Fig. 18). Then, these classified TA signals are encoded into spike trains in the cuneate neurons and finally fed into the somatosensory cortex for tactile perception^31,32,33,34.

To realize it for CH, the TA signals containing texture and roughness information are captured with characteristic surfaces by the scanner and stored in a hierarchical dataset (Fig. 6e, Supplementary Fig. 19, and Supplementary Movie 2). Five types of materials with different surfaces (cotton, fleece, knit, metal, and wood) are selected for this study. Bioinspired by the frequency-sensitive method, we adopt a fast Fourier transform (FFT) to preprocess the collected TA signals into frequency spectra. It can be seen that the fluctuation of the normalized spectra varies with the texture, and the frequency clusters from 1 Hz to 50 Hz have shown characteristic differences (Fig. 6b). For instance, the characteristic frequency clusters (CFCs) of “Cotton” start at around 1 Hz and end at about 25 Hz. After training and learning based on these spectra of textures (3 × 5 × 100 inputs with different velocities) by machine learning and artificial neural networks (ANNs), a 93% accuracy for texture recognition and classification is obtained, much higher than that based on spectra without FFT-pre-processed data (Fig. 6c, Supplementary Fig. 21, and Methods). The studies indicate that the tactile scanner is capable of collecting detailed feature information with high quality, sufficient for texture recognition and classification, and the accuracy is independent of sliding speed. After being encoded by the NPU in the tactile scanner, a spike train is formed (Fig. 6d) and carries the TA signals into the somatosensory cortex for tactile perception. Rather than the precise spike timing measured over the spatiotemporal nature of spike trains^57,58, the highest achieved decoding tactile stimuli were based on the developed amplitude demodulation in communication systems⁵⁹, the decoded TA signal are timing signal with magnitude and frequency information (Supplementary Fig. 20).

In order to “redisplay” tactile information when interacting with a virtual object, a TR process including decoding, conversion, amplification, and execution will be conducted. Here, a commercial TDK actuator (PHUB8060-35A-33-000) driven by the decoded TA signal is used to render vibrotactile displacement for reconstructing the tactile force and friction (Supplementary Fig. 22). The TR signals are benchmarked by the Polytec MSA-600 Micro System Analyzer (sampling rate: 2.5 kHz) with standard deviation of its vibration noise ± 0.18 μm and SNR of 17 dB (Fig. 6e). The spatiotemporal features of TA signals are basically similar to those TR signals, and such a similarity is also quantified in CFCs between TA and TR signals (Supplementary Fig. 24), illustrating that the TR signals can be seen as digital twins of TA signals. Further, a correlation study and a perceptual analysis are conducted to verify the effectiveness of such rendering. Given that the average spearman’s correlation coefficient of five materials is 0.68, the TR signals can be regarded as replication of TA signals with a higher-than-average level of confidence (Fig. 6f and Supplementary Note 4). More than 75% of the volunteers in the psychophysical-perceptual experiment indicated that the roughness of the vibrotactile sensation generated by the TR signals can be distinguished (Supplementary Fig. 25 and Supplementary Note 5). Thus, the closed-loop tactile sensation and feedback are well demonstrated in this work.

Discussion

Bioinspired by the architecture and functions of the human tactile somatosensory system, a tactile scanner with a 10 × 10 tactile sensing array is developed particularly for CH. The tactile scanner with piezoelectric transducer and neuromorphic circuit enables real-time acquisition of tactile stimuli where the piezoelectric transducer is used to stimulate sensory receptor of the axon, and TFT-based neuromorphic circuit is adopted to imitate the transmission mechanism of synapse and dendrite neurons. Tactile information with five types of different textures is successfully captured and perceived with an over 85% accuracy of recognition and classification. The tactile stimulus information is further rendered by using a commercial actuator to recreate the touch feelings of five surfaces with a correlation efficiency of 68%. The proposed tactile scanner and the artificial tactile neuromorphic system can, therefore, be instrumental in enabling CH.

Methods

Modeling of biological and artificial sensory receptors

Mechanically gated ion channels have long been discovered in invertebrates undergoing genetic manipulation, such as the DEG/EnaC channels in caenorhabditis elegans^36,37 and the NOMPC channels in drosophila³⁸. However, the mechanotransduction of mechanically activated piezo channels in mammals, including Piezo1 and Piezo2, have not yet been proposed until 2010 by Patapoutian³⁹. The mechano-gating model presented in Fig. 3b⁴⁰, is a general descriptive model. It depicts the mechanotransduction occurrence as the mechanical force stimulus is transmitted through a three-bladed element to the transduction channel. The gating model of mechanotransduction includes three major parts: the transmembrane (TM) gate and lateral plug gates in the channel protein, a central ion-conducting pore, and three peripheral mechanosensing blades forming a propeller-shaped structure, and the Cap domain atop the TM pore⁴⁴. When a finger touches an object, the Cap takes up much of the rotation, after that, the blades open the transmembrane pore and unplug the lateral plug gates, making cation conduction via the central transmembrane pore and the three intracellular lateral portals (Fig. 3b). Recent researches have demonstrated that touch transduction via Piezo2 works in Merkel cells⁶⁰, but the relationship between its ion channel open probability and membrane tension has not yet been widely studied (Supplementary Fig. 2a). However, some reports have manifested that the channel open probability (P₀) of Piezo1 in cells follows Boltzmann distribution function with membrane pressure (P):

$${P}_{0}=\frac{1}{1+\exp [\alpha (P-{P}_{1/2})]}$$

(2)

The channel mechanosensitivity (α) is the slope of the plot ln [P₀/(1-P₀) = α(P-P_1/2)], P_1/2 is the suction at which P₀ = 0.5. The Piezo1/Piezo2 channel converts mechanical energy to APs, which is relevant to Boltzmann distribution function.

Here, we describe an artificial mechanoreceptor based on the PVDF transducer. The vibrotactile stimulus causes the generated charge of the PVDF transducer to pump as a biologically mechano-gating processer. When the vibrotactile stimulus is applied to PVDF sensor, the piezoelectric layer squeezes and induces a polarization change, resulting in the change of the surface charges of the film, as a result, an increase in voltage between the electrode surfaces. In contrast, when the vibrotactile stimulus is removed, the output voltage is in the of opposite polarity. Since the PVDF is also a capacitor, the voltage in the PVDF transducer (V_PVDF) can be calculated using the equation (Fig. 4b):

$${V}_{PVDF}=\frac{{Q}_{PVDF}}{{C}_{PVDF}}=\frac{{I}_{PVDF}\cdot {\tau }_{tactile}\cdot {t}_{PVDF}}{{\varepsilon }_{0}\cdot \varepsilon \cdot {A}_{PVDF}}$$

(3)

where t_PVDF is the thickness of PVDF film, ε₀ is vacuum dielectric constant, ε is relative dielectric constant of PVDF material, and A_PVDF is the effective area of the PVDF transducer. Numerical analysis of V_PVDF is calculated using a dielectric constant of 113 (Supplementary Reference 10).

Although the artificial sensory receptor (PVDF transducer) is limited in its capacity for imitating electrophysiological process of Piezo1/Piezo2 channel of receptors, it is able to increase intracellular neurotransmitters and transduce tactile inputs into Na⁺/Ca²⁺-APs as mechano-gating processer.

Modeling of biological and artificial neurons

The neural HHM is an expression of excitatory cell characteristics described by four ordinary differential equations³⁹ (Supplementary Fig. 2b and Supplementary Note 1),

$$I={C}_{M}\frac{dV}{dt}+\overline{{g}_{Na}}\cdot {m}^{3}h(E-{E}_{Na})+\overline{{g}_{K}}\cdot {n}^{4}(E-{E}_{K})+\overline{{g}_{L}}(E-{E}_{L})$$

(4)

$$\frac{dm}{dt}={\alpha }_{m}(1-m)-{\beta }_{m}m$$

(5)

$$\frac{dh}{dt}={\alpha }_{h}(1-h)-{\beta }_{h}h$$

(6)

$$\frac{dn}{dt}={\alpha }_{n}(1-n)-{\beta }_{n}n$$

(7)

The transfer rate constants of activation and inactivation α_m, β_m, α_h, β_h, α_n, and β_n are expressed as membrane voltage functions using the Boltzmann equation. It is found that the HHM model is a voltage-dependent channel, and the channel ions distribution follows the Boltzmann equation (Eq. (2)). The above equations establish the fact that the ions channel action of HHM model is the closest to a biological neuron and accurately expresses electrophysiological process, but it is too complex for practical application. Instead, the LIF model is mathematically simplified for simulation and analysis²⁹ (Fig. 3c and Supplementary Fig. 2c). The membrane potential is modeled as (Supplementary Note 1):

$${V}_{Mem}(t)= ({R}_{Ions}\cdot {I}_{IN}(t)+{V}_{{{\mathrm{Re}}}st})+[{V}_{0}-({R}_{Ions}\cdot {I}_{IN}(t)+{V}_{{{\mathrm{Re}}}st})] \\ \exp \left(-\frac{1}{{R}_{Ions}{C}_{Mem}}\cdot t\right)$$

(8)

When the membrane potential (V_Mem) reaches a constant threshold voltage (V_thr), a spike is triggered and V_Rest then “resets” to a resting potential.

In this work, the neuromorphic circuit model is proposed to mimic the receiving function of sensory neurons. The output subthreshold current (I_DS) of synaptic TFT N1 and the voltage of Cs (V_S) of the NPU circuit (Fig. 3d) can be expressed as the following equation (Supplementary Note 1):

$${I}_{DS}={I}_{D0}\exp \left(\frac{q[(1+\gamma ){V}_{PVDF}-{V}_{TH1}]}{\beta \cdot kT}\right)$$

(9)

$${V}_{S}(t)={V}_{0}+\frac{{I}_{D0}}{{C}_{S}}\exp \left(\frac{q[(1+\gamma )\left(\frac{{Q}_{PVDF}}{{C}_{PVDF}}\right)-{V}_{TH1}]}{\beta \cdot kT}\right)\cdot t$$

(10)

Where q is the electron charge, k is Boltzmann’s constant, T is the absolute temperature in Kelvin, β is related to the subthreshold swing, the C_S initial voltage is V₀, γ is the dependence parameter that dictates the TG controllability over the threshold voltage of TFT N1, I_D0 is the output current when V_BG = V_TH1 and V_DS« -kT/q. The derivation and parameter can be found in Supplementary Note 1.

By comparing of Eqs. (8) and (10), a resemblant expression between time, input stimuli, and the output voltage of LIF model (V_Mem(t) = f (t,I_IN)) and NPU circuit model (V_S(t) = f (t, Q_PVDF)) can be found (Supplementary Note 1). Thus, the NPU can potentially emulate functionality and signal processing of biological synapse and neuron.

Design and fabrication of axon sensor array

The multilayered design layout of 10 × 10 PVDF axon sensor array is presented in Supplementary Fig. 3. It includes cover layer 1 (CVL1), copper wire, cover layer 2 (CVL2), patterned copper electrode array, low-temperature lamination anisotropic conductive Film (ACF), PVDF film with patterned molybdenum (Mo) electrode array, carbon conductive glue (CCG), aluminum (Al) coated (one side) polyimide film (PI). First, the CVL1/Copper Wire/CVL2 layer is prepared by Shenzhen Chipwey Innovation Technologies Co. Ltd. Second, the ACF is manually attached to the CVL1/Copper Wire/CVL2 layer at room temperature. Third, a 110-μm PVDF sheet (Measurement Specialties Inc., S06260-3 model) with preferable pressure response is selected for metallization with 100-nm Mo electrodes prepared by sputtering (50 sccm, 5 mTorr, 700 W) on PVDF film with shadow masks. Fourth, the PVDF film with a patterned electrode array and the CVL1/Copper Wire/CVL2 layer are aligned and laminated through ACF (Teamchem Materials Company) bonding. The processes detail is shown in Supplementary Fig. 4, and Supplementary Table 4. Finally, the CCG (Nisshin-em Co. Ltd) is used to laminate the other side of the PVDF film with Al covered by PI film.

Design and fabrication of neuromorphic circuit array

The corresponding fabrication process of neuromorphic circuit array is presented in Supplementary Figs. 6 and 7. The microfabrication includes a five-mask photographic process. First, a 200-nm aluminum (Al) and an 80-nm molybdenum (Mo) layers are sputtered atop the electronic-grade glass substrate and are then patterned and wet etched to define the BG electrode of synaptic TFT N1, the gate electrode of TFT N2, the bottom plate of the capacitor (C_S). Second, a thin 300-nm silicon nitride (SiNx) is deposited as dielectric layer by plasma-enhanced chemical vapor deposition (PECVD). Third, a 150-nm hydrogenated amorphous silicon (a-Si:H) active layer and 50-nm n⁺ a-Si:H are deposited by PECVD at 300 °C, consecutively. After that, the active layer is patterned by photolithography and dry etching. Fourth, the via holes are etched to realize the metal interconnection between the drain electrode and BG of TFT N1. Fifth, a 20-nm Mo, a 200-nm Al, and an 80-nm Mo are sequentially sputtered and patterned to be drain/source electrodes of TFT N1/N2 and top plate region of the capacitor. Sixth, a 300-nm SiNx dielectric layer is deposited to form the TG dielectric layer for the TFT N1. Seventh, the via holes are created by selectively dry etching. Lastly, a 75-nm Mo is sputtered and patterned as the TG of the TFT N1, which also works as a light shielding layer to prevent ambient light impurging into the circuit. Figure 3f gives the micrographic photo of a NPU in the neuromorphic circuit array, while TFT N1 W/L ratio is 516 µm/3.2 µm, TFT N2 is 102 µm/3.2 µm, and Cs = 0.1 pF.

Characterization of synaptic TFT and NPU

To test the synaptic TFT N1, the BG, the TG, and the drain are shorted. The source serves as an output terminal. The equivalent resistor and capacitor of TFT N1 are obtained by impedance measurement (Keysight LCR Meter 4980 A) through a cascade EPS150 probe station at a room temperature of 295 K (Supplementary Fig. 8a). A DC voltage of 1 V sweeps stepwise at the drain of TFT N1, which is superimposed by an AC signal with the frequency ranging from 20 Hz to 1 MHz and an amplitude of 50 mV. The measurement results are shown in Fig. 4d.

The neuromorphic circuit is simulated by Silvaco tools, and the schematic diagram of NPU and a comparator are shown in Fig. 5a. The threshold voltage of the NPU is generated by a TFT-based comparator circuit where the threshold voltage range (V_TH) is set at 255 mV. The electrical potentials generated by mechanical receptor utilize pulse waves to emulate the different strength of tactile stimuli. The relevant results and parameters of NPU are described and assigned in Supplementary Fig. 10 and Supplementary Table 5. The I_SPIKE of the fabricated NPU increases with intensity (Supplementary Movie 4). The TA signals of wood in Fig. 6e as tactile inputs of NPU are simulated in Supplementary Fig. 20, indicating that the NPU is capable of encoding tactile stimuli.

Evaluation of tactile scanner

To acquire the I_SPIKE of NPU, trans-impedance amplifier (TIA) with low input bias current is designed and optimized on the DAB (Supplementary Fig. 12), which is based on LTC6269 (Linear Technology Corporation) with 50 M feedback resistor, ±2.5 V supply voltage. The TIAs convert I_SPIKE of NPU to voltage signal (V_OUT) as V_OUT = −I_SPIKE   50 M. The V_OUT is obtained by the graphical user interface of DAB (Supplementary Fig. 15), and the numerical I_SPIKE can be simultaneously obtained. Like a MOSFET switch, the TFT N2 also exhibits non-idealities like channel charge injection and off-state leakage (Supplementary Reference 14). It will cause an undesirable drop in the I_SPIKE. When the firing signal (V_F) alternates between “ON” state and “OFF” state, there is a switching noise (0 gf), and the I_SPIKE superimposes the switching noise (Supplementary Fig. 13b). The dynamic characteristics of the tactile scanner measurement platform is shown in Supplementary Fig. 13a. The shaker drive rod (PM300 piezometer) applies dynamic pressure to the scanner and the I_SPIKE current is obtained by the DAB system. The acquisition rate of the tactile scanner is approximately 1 kHz, and the pulse duration of the V_F is 110 μs with amplitude of 5 V. Considering the sensitivity requirements of the tactile scanner, the I_SPIKE is sampled at ∼30 μs after the turn-on of V_F. When the tactile scanner is exposed to continuous pressing-releasing cycles generated by a mechanical motor with a continuous force loading-unloading cycles with 0.5 N (interval of ∼6.19 s) and 0.0 N (interval of ∼5.22 s) at 100 Hz, the result is depicted in Supplementary Fig. 13c. When the force in unloading cycle, the sensor obtains ∼3.8 nA current, and up to ∼4.3 nA in loading cycle. When a series of pressure stimuli (21.7 kPa in magnitude and 2 s in duration) with varied frequencies (5-, 10-, 50-, 100-, and 200 Hz) are applied to the scanner, the I_SPIKE increases with the frequency (Supplementary Movie 1, Supplementary Fig. 13d). Thus, the NPU-integrated tactile scanner is capable of producing repeatable tactile responses.

Surface texture recognition and material classification

Tactile data for machine learning is selected over 6 seconds collection of waveforms with a sampling rate of 1 kHz, and each dataset is composed of 3000 data points generated from one cycle of forward scanning with a scanning velocity of 10-, 15-, 20 mm s⁻¹, respectively. Tactile information datasets of surface textures with 3 × 500 scans are recorded by the tactile scanner and used for training and learning. The training and test sample ratio is 4:1. Machine learning is performed using the Classification Learner from Statistics and Machine Learning Toolbox 12.3 (MATLAB version 2022a). The artificial neutral network (ANN) in this work has a hidden layer with a size of 27, and ReLU as the activation function is used to alleviate overfitting and excessive computation, and the classic softmax function is used as an output layer. The ANN determines the textures by calculating the probability that is obtained from the hidden layer. The recognition accuracy fed with discrete-time FFT data is two-fold higher than that using tactile data without FFT preprocessing (Supplementary Fig. 21).

Ethics statement

The psychophysical-perceptual experiment was conducted in accordance with the principles of the Declaration of Helsinki and approved by the Medical Ethics Committee of Sun Yat-sen Memorial Hospital (Approval No. SYSKY-2023-911-01). All individuals who participated in the psychophysical-perceptual experiment were aware of the use of their test results and provided their written informed consent. The identifiable images of the research participants confirm that consent to publish.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.