Building an automated three-dimensional flight agent for neural network reconstruction

RoboEM, an artificial intelligence (AI)-based flight agent, automatically steers through three-dimensional electron microscopy (3D-EM) images of brain tissue to follow neurites. RoboEM substantially improves state-of-the-art automated reconstructions, eliminating manual proofreading needs in complex connectomic analysis problems and paving the way for high-throughput, cost-effective, large-scale mapping of neuronal networks — connectomes.

The problem

Mapping neuronal networks in human and other mammalian brains at synaptic resolution — synaptic-resolution connectomics — holds the promise of extracting from brain tissue both sensory processing algorithms and information stored about the world. Scientists reconstruct neural networks using three-dimensional electron microscopy (3D-EM) data¹, but automated analysis of 3D-EM image stacks is challenging. Even latest-generation artificial intelligence (AI)-based image segmentation still needs to be followed by human annotators spending thousands of work hours proofreading the reconstructions².

However, since 3D-EM data in synaptic-resolution connectomics studies are reaching petabyte scale, full automation of image segmentation is required for routine mapping of connectomes. We set out to make progress on this challenge, starting from the observation that the maximum analysis speed of human annotators occurs when presenting 3D EM data in a neurite-centered 3D flight view³. Could AI agents directly emulate this analysis approach to ‘fly along’ neuronal wires?

The solution

We reasoned that human annotation of neuronal wires — tubular structures with complex 3D trajectories — can be likened to a flight agent steering through 3D-EM image data (predicting neurite continuation from neurite-centered and neurite-aligned images). Since steering a flying object in 3D along neurites on the basis of a stream of 3D input image data has similarities to steering a car in 2D along a street on the basis of the image sequence from a front-facing video stream, we started with AI architectures and steering paradigms similar to those developed for autonomous driving, but generalized them to 3D. For supervised training, we incorporated inputs displaced from neurite centerlines in various orientations with correspondingly adjusted steering, resulting in a first version of the virtual AI-based flight agent. Solely from 3D EM data as input, the agent predicts steering, which upon execution and reiteration yields a sequence of visited points (the flight path, Fig. 1). Through iterative improvements in network architecture and training paradigms — as quantified by steering error rates — and the addition of execution safeguards, we developed a sequential decision-making AI-based ‘robot’ proofreader that emulates human flight annotation in EM data, hence RoboEM.

Fig. 1: RoboEM replaces human proofreading by emulating the flight along axons.

a, Image segmentation from three-dimensional electron microscopy (3D-EM) data of brain tissue and segment agglomeration as performed by existing AI-based analysis methods, which is followed by human proofreading (involving manual inspection and/or tracing). In flight tracing, human annotators steer while flying along neurites, yielding state-of-the-art proofreading throughput. b, The RoboEM flight agent emulates human flight tracing: solely on the basis of raw electron microscopy image data (gray), RoboEM iteratively steers yielding a flight path (arrow) used to automatically correct reconstruction errors. c, Example flight path of RoboEM through 3D EM data of brain tissue from mouse cortex along a thin axon, approximately 100 nm in diameter. © 2024, Smith, M. et al., CCBY 4.0.

RoboEM performed connectomic network analyses automatically that had previously required thousands of human work hours per dataset. RoboEM had comparable error rates to those of human annotators and could replace a large proportion of human annotation. Importantly, RoboEM also resolves 3–4 times as many split errors as state-of-the-art AI techniques for connectomic data processing^4,5 while being computationally efficient, adding approximately 20% computing cost to the most efficient state-of-the-art image segmentation techniques.

The implications

Since RoboEM can be added to any already developed segmentation and agglomeration pipeline for connectomics (Fig. 1a), it is a versatile method for replacing human annotation in reconstruction pipelines (Fig. 1b,c). We have enabled full automation of neuronal circuit reconstruction, to an accuracy required for many key medium-scale connectomic analyses. More generally, we think that directly emulating the human data interaction on this task and then training an AI agent to come close to human performance may be a better approach than training AI exclusively on the raw image voxel data. Since efficiency (at a given accuracy) of automated reconstruction has become a key challenge of connectomics, the flight agent approach of predicting low-dimensional flight paths provides a solution.

Designed to fly along neuronal wires, RoboEM does not currently detect the branch points of such wires or reconstruct other kinds of cellular processes (for example, the sheet-like structures of neuroglia in the brain). The focus on a particular annotation challenge — the long-range flight along thin axonal cables (and the continuation along very thin spine necks) — is both the strength and limitation of RoboEM. Since branch points (and glial structures) are already well detectable by current AI-based processing methods, this limitation is less severe in practice.

The success of human-emulated agents in AI research spurs the hope that the 3D flight approach will be able to incorporate many of the biases and models about neuronal data that human annotators use for successful reconstruction (and possibly others that are not accessible to the human visual system). Using reinforcement learning, self-supervised learning and similar approaches, RoboEM may be further developed towards superhuman performance on neuronal circuit reconstruction in the future.

Moritz Helmstaedter & Martin Schmidt

Max Planck Institute for Brain Research, Frankfurt, Germany.

Expert opinion

“In this work, the leading approach to human-assisted neurite proofreading (flight tracing) is replaced by a supervised machine learning approach to steering the flight along the neurite, inspired by similar approaches in autonomous car driving. The reported performance roughly meets human-level proofreading for an identical task.” Albert Cardona, MRC Laboratory of Molecular Biology, Cambridge, UK.

Behind the paper

In today’s AI research, any project lasting longer than 6 months is considered slow, if not outdated and overrun by history. RoboEM, with a concept and an approach that seem quite plausible, still took 5 years from idea to the fully evaluated tool reported here. This was only possible with the patient support of the Max Planck Society, which encourages long-term projects, and the tenacity of the first author. Details made all the difference. And, as often, dead ends had to be avoided efficiently. We had to abandon, for example, the idea of using steering variability to indicate branch points — a nice analogy to road intersections in car steering, but too far-fetched to work well enough in brain tissue data. Sometimes giving up beloved ideas is as important as following through on others. M.H.

From the editor

“Inspired by self-driving cars, the RoboEM algorithm automates tracing of neurons in electron microscopy datasets. This approach stood out to me because manual tracing of neurons is labor-intensive and the automation of this task with high accuracy is sorely needed.” Nina Vogt, Senior Editor, Nature Methods.