Virtual reality-empowered deep-learning analysis of brain cells

Kaltenecker, Doris; Al-Maskari, Rami; Negwer, Moritz; Hoeher, Luciano; Kofler, Florian; Zhao, Shan; Todorov, Mihail; Rong, Zhouyi; Paetzold, Johannes Christian; Wiestler, Benedikt; Piraud, Marie; Rueckert, Daniel; Geppert, Julia; Morigny, Pauline; Rohm, Maria; Menze, Bjoern H.; Herzig, Stephan; Berriel Diaz, Mauricio; Ertürk, Ali

doi:10.1038/s41592-024-02245-2

Download PDF

Article
Open access
Published: 22 April 2024

Virtual reality-empowered deep-learning analysis of brain cells

Nature Methods (2024)Cite this article

8454 Accesses
194 Altmetric
Metrics details

Subjects

Abstract

Automated detection of specific cells in three-dimensional datasets such as whole-brain light-sheet image stacks is challenging. Here, we present DELiVR, a virtual reality-trained deep-learning pipeline for detecting c-Fos⁺ cells as markers for neuronal activity in cleared mouse brains. Virtual reality annotation substantially accelerated training data generation, enabling DELiVR to outperform state-of-the-art cell-segmenting approaches. Our pipeline is available in a user-friendly Docker container that runs with a standalone Fiji plugin. DELiVR features a comprehensive toolkit for data visualization and can be customized to other cell types of interest, as we did here for microglia somata, using Fiji for dataset-specific training. We applied DELiVR to investigate cancer-related brain activity, unveiling an activation pattern that distinguishes weight-stable cancer from cancers associated with weight loss. Overall, DELiVR is a robust deep-learning tool that does not require advanced coding skills to analyze whole-brain imaging data in health and disease.

Dense 4D nanoscale reconstruction of living brain tissue

Article Open access 10 July 2023

Whole-brain tissue mapping toolkit using large-scale highly multiplexed immunofluorescence imaging and deep neural networks

Article Open access 10 March 2021

SimpylCellCounter: an automated solution for quantifying cells in brain tissue

Article Open access 28 July 2020

Main

Analyzing the expression of proteins is essential to understand cellular and molecular processes in physiology and disease. While standard immunohistochemistry is useful for validating protein expression on tissue sections, it does not provide a holistic view of expression patterns in larger samples and information can be lost during slicing^1,2. Tissue clearing and fluorescent imaging solve many of these restrictions and allow unbiased protein expression analysis up to the whole-organism scale^1,3,4,5.

Whole-brain analysis is essential for detecting areas involved in specific behaviors or conditions. A brain-wide snapshot of the neuronal activity of an animal can be obtained by immunostaining for the expression of immediate early genes such as c-Fos. Unbiased quantification methods for system-level examination at the single-cell resolution are essential to interpret those brain-wide findings⁶, but current automated methods for cell detection and registration to the Allen Mouse Brain Atlas^7,8,9 are difficult to apply consistently to three-dimensional (3D) whole-brain datasets. Variations in image acquisitions between samples, uneven signal-to-noise ratios across the tissue or low abundance of the target protein limit detection sensitivity and specificity. This requires manual adjustments such as setting sample and volume-specific thresholds or using conservative thresholds that will not capture all information in each sample. Deep-learning-based cell detection methods offer a promising solution to address these challenges; however, their implementation typically demands advanced coding skills, presenting a challenge for users lacking computational expertise.

Here, we developed DELiVR (deep learning and virtual reality mesoscale annotation pipeline), a virtual reality (VR)-aided deep-learning pipeline for detecting c-Fos⁺ cells in cleared mouse brains (Fig. 1a) that can be extended to other cell types. We generated high-quality annotations of light-sheet microscopy data of cleared whole mouse brains stained for c-Fos in a VR environment. Next, we trained a deep neural network on these data to identify c-Fos⁺ cells across the brain and mapped them automatically to the Allen Brain Atlas. To increase the usability of DELiVR, we packaged it into a single Docker container that runs via a plugin for the open-source software Fiji. DELiVR can also be trained with custom data via Fiji to adapt DELiVR to specific datasets. We used DELiVR to study cancer-related cachexia and found increased neuronal activity in mice with weight-stable cancer in brain areas related to sensory processing and foraging. In contrast, this increase was lost in cachectic animals, suggesting a weight-stable cancer-specific neurophysiological hyperactivation phenotype.

**Fig. 1: Virtual reality-aided annotation is faster than 2D-slice annotation.**

Results

Reference annotation is faster in VR compared to 2D slices

We used the SHANEL protocol¹⁰ for whole-brain c-Fos immunostaining, tissue clearing and light-sheet fluorescence microscopy (LSFM). To train deep-learning segmentation models in a supervised manner, substantial amounts of high-quality expert annotations are crucial. As common annotation approaches such as ITK-SNAP¹¹ rely on time-consuming sequential two-dimensional (2D) slice-by-slice annotation, we used a VR approach that allows for full immersion into 3D volumetric data (Fig. 1b,c). We used two commercial VR annotation software packages (Arivis VisionVR and syGlass¹²) to evaluate the speed and accuracy of VR in comparison to 2D slice-based annotation in ITK-SNAP.

For annotation using Arivis VisionVR, the annotator defined a region of interest (ROI) in which an adaptive thresholding function was applied, according to the annotator’s input (Fig. 1d–h and Supplementary Video 1). In syGlass, the annotation tool allowed the annotator to draw simple 3D shapes as ROIs and adjust a threshold until the annotation was acceptable to the annotator (Extended Data Fig. 1a–d and Supplementary Video 2). In ITK-SNAP, individual c-Fos⁺ cells were segmented in each plane of the image stack (Extended Data Fig. 1e and Supplementary Video 3). We evaluated the time spent by the annotators for a 100³ voxel sub-volume (depicting 83 c-Fos⁺ cells) as well as the annotation quality of cell instances using the F1 score. We found that VR annotation was significantly (P = 0.0005, two-sided Mann–Whitney U-test) faster than 2D-slice annotation (Fig. 1i) and improved annotation quality (increase in F1 score from 0.7383 to 0.8032 (Fig. 1j)). Thus, we decided to generate reference data in VR for our deep-learning algorithm for c-Fos activity mapping.

DELiVR outperforms threshold-based c-Fos segmentation

To comprehensively analyze neuronal activity across the entire brain, DELiVR detects and aligns the cells to the Allen Brain Atlas. DELiVR then visualizes the segmentation in both image and atlas space. Therefore, DELiVR consists of multiple steps (Fig. 2a). First, the pipeline downsamples the raw image stack and generates ventricle masks (Extended Data Fig. 2a–c). It then upscales the masks and uses them to mask the ventricles in the raw image input. DELiVR then utilizes a customized sliding-window inferer to identify potential cells. Afterwards, we conduct a connected component analysis¹³ to identify individual cells in the masked images and filter by size. DELiVR then aligns the previously downsampled brain to the Allen Brain Atlas (CCF3, 50 µm per voxel) with mBrainAligner¹⁴ and assigns the corresponding atlas region to each detected cell. The connected component analysis returns a set of center-point coordinates and volume for each segmented cell, which DELiVR then automatically maps to the Allen Brain Atlas with mBrainAligner.

**Fig. 2: DELiVR’s UNet outperforms current methods for c-Fos⁺ cell detection.**

To train and validate our model, we randomly sampled and VR-annotated 48 × 100³ voxel patches (referring to 5,889 cells) from a c-Fos-labeled brain. From these we trained a 3D BasicUNet (Extended Data Fig. 2d). In addition, we trained recent larger segmentation models, such as transformers¹⁵, SegResNet¹⁶ and the MONAI DynUnet¹⁷ to determine which model was best suited for our data. Assessing the instance performance by calculating the overlap between individual cells, the 3D BasicUNet architecture showed the best performance (based on F1 score) (Fig. 2b and Extended Data Fig. 2e). Therefore, we chose the 3D BasicUNet for our DELiVR pipeline.

We also compared DELiVR with previously published non-deep-learning models that are applicable to cell detection in 3D images and had code available (ClearMap⁷, ClearMap2 (ref. ¹⁸) and Ilastik¹⁹). Our performance on the test set shows an F1 score of 0.7918 (+89.03% increase), instance sensitivity of 0.8470 (+181.64% increase), instance precision of 0.7434 (+7.74% increase) and a volumetric Dice of 0.6739 (+581,39% increase) compared to the second-best performing method, ClearMap2 (Fig. 2c,d, Extended Data Fig. 2f and Supplementary Table 1). We increased the performance of ClearMap based on the F1 score to 0.65 by manually pre-processing image stacks and optimizing parameters for cell detection³; however, DELiVR still had a superior performance. These scores demonstrate a clear improvement over filter and threshold-based segmentation methods as the deep-learning model captures 84.8 times more cells (1,611 true positives) than ClearMap (19 true positives), 2.8 times more cells than ClearMap2 (572 true positives) and 31.2% more than the optimized version of ClearMap (1,228 true positives) while not over-segmenting. For visualization, DELiVR generates a whole-brain segmentation output that exists in the original image space. Here, each segmented cell corresponds to a threshold value fitting to an Area ID of the Allen Brain Atlas and was colored according to the brain region that it belongs to (Extended Data Fig. 3a,b and Supplementary Video 4). In addition, we used BrainRender²⁰ to plot and visualize the detected cells in the atlas space (Fig. 2e and Extended Data Fig. 3c).

To increase usability, the entire DELiVR pipeline, encompassing atlas alignment, cell detection and visualization, is available as a single, user-friendly Docker container for both Linux and Windows. Docker is a software platform that allows to bundle and distribute applications, along with their required components, in a uniform container format²¹. We also developed a dedicated Fiji plugin to seamlessly run the DELiVR Docker (Fig. 3a–c and Supplementary Video 5).

**Fig. 3: DELiVR runs end to end and can be adapted to other cell types.**

Moreover, we provide a Docker container for training that integrates with the DELiVR Fiji plugin (Fig. 3a,c). This feature allows users to (re-)train DELiVR on other datasets, thereby enhancing DELiVR’s precision and adaptability. Users can choose to fine-tune the existing c-Fos model or train their own model from scratch. For this, one can adjust hyperparameters such as the number of epochs and learning rate. The user-trained model can then be used in the DELiVR pipeline as the inference model. For a comprehensive guide, please consult our ‘DELiVR handbook’ (Supplementary Note).

We used our DELiVR training to annotate microglial cell bodies, the brain’s resident macrophages²². We performed whole-brain nanobody labeling in CX3CR1^GFP/+ reporter mice, clearing and LSFM, and annotated microglia somata in VR (Fig. 3d,e). Training used a dataset of 161 VR-annotated 100³ voxel patches with a total of 3,798 annotated microglia somata. The newly trained model had an F1 score of 0.92, indicating robust performance²³ (Fig. 3f). We applied this model in the DELiVR pipeline and could detect and map microglia cell bodies throughout the brain. Using DELiVR’s visualization tool, we evaluated the microglia cell body segmentation output generated by DELiVR in our original images (Fig. 3g–i) and mapped the segmented cells to region IDs of the Allen Brain Atlas (Fig. 3i). Thereby, DELiVR allows to find and confirm an anatomical or functional sub-area in the original image stack of the brain.

DELiVR identifies activation patterns in tumor-bearing mice

Cancer affects normal physiology locally in the surrounding tissue but can also lead to profound changes in the systemic metabolism of the patient. This is exemplified by the wasting syndrome cancer-associated cachexia (CAC) characterized by involuntary loss of body weight^24,25,26 and specific changes in brain activity²⁷.

To identify brain regions affecting body weight maintenance in cancer, we used DELiVR to compare the neuronal activity patterns between weight-stable cancer and CAC. We subcutaneously transplanted NC26 colon cancer cells that give rise to weight-stable cancer or C26 colon cancer cells that induce weight loss (Fig. 4a). As expected²⁸, no changes in body weight were observed in NC26 tumor-bearing mice compared to controls, whereas C26 tumor-bearing mice showed significant (P < 0.0001, one-way analysis of variance (ANOVA) with Sidak post hoc analysis) reductions (Fig. 4b). The differences in body weight were not due to differences in tumor mass (Fig. 4c). C26 tumor-bearing mice displayed reduced weights of the gastrocnemius muscle and white adipose tissue depots (Extended Data Fig. 4a–c). We observed a small but statistically significant (P = 0.0479, one-way ANOVA with Sidak post hoc analysis) decrease in brain weights of cachectic C26 versus weight-stable NC26 tumor-bearing mice (Extended Data Fig. 4d). We performed c-Fos antibody labeling, clearing and imaging of whole brains of these mice and applied DELiVR for whole-brain mapping of neuronal activity. c-Fos⁺ density maps indicated an increase in brain activity in weight-stable NC26 tumor-bearing mice compared to phosphate-buffered saline (PBS) controls, whereas this increase was not present in cachectic C26 tumor-bearing mice (Fig. 4d).

**Fig. 4: DELiVR identifies changes in neuronal activity in weight-stable cancer.**

The increase in brain activity in NC26 tumor-bearing mice was most pronounced throughout the cortical plate and in the lateral septal complex (Fig. 5a,b). Overall, we identified 19 areas in NC26 tumor-bearing mice that showed statistically significantly (Padj < 0.1, two-sided unpaired t-tests with Benjamini–Hochberg multiple-testing correction with family-wise error rate (FWER) = 0.1) increased c-Fos expression compared to PBS controls after multiple-testing correction (Fig. 5a). We found that NC26-bearing mice also have more c-Fos⁺ cells in the cortical plate, with the most pronounced differences observed in the somatomotor areas (Fig. 5a,b). NC26 tumors notably increased c-Fos⁺ density in the somatosensory cortex related to the snout, specifically in the mouth region (layer 2/3 and 4) and barrel field layer 4. Furthermore, NC26-bearing mice showed more c-Fos⁺ cells than PBS controls in the primary (layers 1 and 5) and secondary motor areas (layers 2/3 and 5; Fig. 5a,b). The primary motor cortex layer 5 is especially interesting because it contains extratelencephalic projection neurons that project as far as the spinal cord, among others²⁹.

**Fig. 5: DELiVR identifies cancer-related brain activation patterns.**

We found seven areas that were significantly altered between NC26 and cachectic C26 tumor-bearing mice, whereas we did not observe significant changes (Padj > 0.1, two-sided unpaired t-tests with Benjamini–Hochberg multiple-testing correction with FWER = 0.1) in c-Fos⁺ expression when comparing PBS and cachectic C26 tumor-bearing mice after correcting for multiple testing (Fig. 5a). When comparing NC26 to C26 tumor-bearing mice, we found that NC26 mice had more c-Fos⁺ cells overall in the cortical plate, with the differences clustering in the dorsal and agranular lateral retrosplenial cortex as well as a subset of entorhinal cortex (Fig. 5a,c). Evaluation of c-Fos⁺ density heat maps offer additional details (Fig. 5d). Evaluation of layer 2/3 retrosplenial cortex shows that activity clusters in the anterior third of retrosplenial cortex in PBS and C26, but not NC26 retrosplenial cortex, where it is both stronger and spread further to the back. The retrosplenial cortex is thought to be a site of multisensory integration, spatial integration and environment mapping³⁰ and is crucially involved in foraging behavior³¹.

Overall, our findings showed that brain activity in weight-stable NC26 cancer-bearing mice are markedly different from both cachectic C26 cancer-bearing mice and PBS controls. Specifically, we find a consistent hyperactivation phenotype in NC26 brains that is detectable at the whole-cortex level, but most pronounced in areas relevant to somatosensation at the snout and motor planning, as well as spatial navigation (all of which would be consistent with a foraging-related brain activation pattern). Thus, with DELiVR we were able to identify a neuronal activity pattern specific to the NC26 cancer model.

Discussion

Here, we present DELiVR, an end-to-end VR-enabled deep-learning-based quantification pipeline for whole-brain cell mapping in cleared mouse brains. We designed it to make deep learning accessible to most biologists via a Fiji front end, not requiring coding skills. We leveraged VR technology to generate reference annotations for training a deep-learning-based segmentation network. DELiVR improves segmentation accuracy compared to current cell detection methods and generates a registered segmentation output that can be examined in the original image and in the atlas spaces. In addition, our Fiji training feature enables users to adapt DELiVR to their specific datasets, increasing its versatility and usability.

Traditional, non-machine-learning solutions for large-scale analysis of c-Fos cell detection, such as ClearMap^7,18 rely on a sophisticated system of thresholding and filtering to detect small structures and classify them as cells. While such approaches generated valuable information⁶, their performance is limited for data with variable signal-to-noise ratios, as is the case when imaging large volumes such as the entire mouse brain. Though parameters can be adjusted, it is difficult to find a setting that accounts for all cells. Hence, the thresholds tend to be set conservatively, meaning that subtle differences may be lost during threshold-based analysis. A trained deep-learning model learns these local variances, thus providing more accurate cell number estimates than threshold-based methods, as exemplified by DELiVR’s high instance F1 score. Previous approaches for segmenting cells in mouse brains ranged from deep learning^32,33,34, random forest algorithms^19,35 and threshold-based solutions^7,18,36; however, only a subset of studies published their analysis pipeline and model weights in a working package that makes it applicable for other datasets. We found that DELiVR’s 3D BasicUNet consistently outperformed all other approaches with available code. In addition to providing highly accurate AI-based cell detection, DELiVR provides a unique and accessible open-source tool, functioning seamlessly within Fiji. It encompasses all steps of brain activity mapping, including cell detection, atlas alignment and visualization, in an easily accessible environment without the need for writing additional computer code.

Our experiments showed that VR is a superior means of annotation and data exploration for volumetric data analysis. Non-VR methods show orthogonal slices, which allows an annotator to outline the shape of individual cells in 2D; however, it obfuscates necessary volumetric information, making annotation challenging and time consuming; an annotator never sees the whole cell, only a cross-section and must scroll through slices to ensure that it is in fact a cell and not background noise. In contrast, VR allows the annotator to capture 3D structures in their entirety, enabling the fast generation of more reliable annotated data.

In future work, it will be interesting to explore the possibility of performing active learning in a VR environment. Active learning is a combined machine-learning training and annotation approach, where a model selectively chooses the most informative or uncertain data points for manual annotation, allowing for efficient model improvement with fewer labeled data points³⁷. This approach is currently limited by the possibilities of the VR annotation software application programming interfaces. Using an ensemble of networks or a test time augmentation uncertainty map as well as methods such as Monte-Carlo-based sampling using dropout layers³⁸ to highlight areas that are ambiguous to the network, the annotator can be guided to even more efficient time use in VR annotation. The annotators’ choices can then be fed into a fine-tuning step to improve the model while annotating.

We used DELiVR to profile the brain activation patterns of cancer-bearing mice that were either weight-stable or displayed CAC. A mix of reduced food intake, elevated catabolism, increased energy expenditure and inflammation drives weight loss in cancer²⁶. The brain was shown to contribute to anorexia in CAC, as it responds to inflammatory cytokines that modulate the activity of neuronal populations that regulate appetite³⁹. In addition, activation of neurons in the parabrachial nucleus was shown to suppress appetite in mouse models of CAC⁴⁰. The reduction in brain weight among cachectic C26 tumor-bearing mice aligns with prior reports of decreased brain weight in mice with cachexia-inducing pancreatic tumors⁴¹. It is currently unclear whether this this volume reduction is due to cell death, or white matter loss.

Notably, we found a substantial increase in c-Fos⁺ expression in the brains of weight-stable NC26 tumor-bearing mice, especially in motor and sensory areas, and higher-order regions such as the retrosplenial cortex. Those regions are linked to sensorimotor control, motor sequencing and foraging^30,42,43. The abundance of sensory-related regions suggests cancer-specific impairment in GABAergic inhibition⁴⁴, driving a hyperactivation phenotype via disinhibition. If and how these increases in neuronal activation in weight-stable cancer-bearing mice affect body weight maintenance will be of a high interest to explore in future studies.

In conclusion, we present DELiVR: an integrated, easy-to-use pipeline to label, scan and analyze neuronal activity markers across the entire mouse brain and show how VR increases the speed and accuracy of generating reference annotations. Using DELiVR, we find differences in c-Fos expression between cachectic and non-cachectic cancer mouse brains, pointing us to a previously unknown neurophysiological phenotype in cancer-related weight control.

Methods

Whole-brain immunolabeling and clearing

Immunostaining for c-Fos was performed using a modified version of SHANEL¹⁰. All incubation steps were carried out under moderate shaking (300 rpm). For the pretreatment, samples were dehydrated with an ethanol/water series (50%, 70% and 100% ethanol) at room temperature for 3 h per step. Next, samples were incubated in dichloromethane (DCM)/methanol (2:1 v/v) at room temperature for 1 day. Brains were rehydrated with an ethanol/water series (100%, 70% and 50% ethanol and diH₂O) at room temperature for 3 h per step. Samples were incubated in 0.5 M acetic acid at room temperature for 5 h followed by washing with diH₂O. Next, brains were incubated in 4 M guanidine HCl, 0.05 M sodium acetate, 2% v/v Triton X-100, pH 6.0, at room temperature for 5 h followed by washing with diH₂O. Brains were incubated in a mix of 10% CHAPS and 25% N-methyldiethanolamine at 37 °C for 12 h before washing with diH₂O. Blocking was performed by incubating the brains in 0.2% Triton X-100, 10% dimethylsulfoxide and 10% goat serum in PBS shaking at 37 °C for 2 days. Samples were incubated with c-Fos primary antibody (Cell Signaling Technology, 2250, 1:1,000 dilution) in primary antibody buffer (0.2% Tween-20, 5% dimethylsulfoxide, 3% goat serum and 100 µl heparin per 100 ml PBS) shaking at 37 °C for 7 days. The antibody solution was filtered (22-µm pore size) before use. Samples were washed in washing solution (0.2% Tween-20 and 100 µl heparin in 100 ml PBS) shaking at 37 °C for 1 day at which the washing solution was refreshed five times. Brains were incubated with the secondary antibody (Alexa Fluor 647 and goat anti-rabbit IgG (H + L) from Invitrogen, A-21245, 1:500 dilution) in secondary antibody buffer (0.2% Tween-20, 3% goat serum and 100 µl heparin per 100 ml PBS) shaking at 37 °C for 7 days followed by incubating in washing solution shaking at 37 °C for 1 day at which the washing solution was refreshed five times. Brains were dehydrated using 3DISCO² with a THF/H₂O series (50%, 70%, 90% and 100% THF) for 12 h per step followed by an incubation in DCM for 1 h. Tissues were incubated in benzyl alcohol/benzyl benzoate (1:2 v/v) until tissue transparency was reached (>4 h).

For microglia labeling, brains of CX3CR1^GFP/+ mice were pretreated via the modified SHANEL protocol as described above and incubated with Atto647N-conjugated anti-GFP nanobooster (Chromotek, gba647n-100, 1:1,000 dilution) with 5% 2-hydroxypropyl-β-cyclodextrin, 0.2% Tween-20 and 6% goat serum in PBS for 5 days at 37 °C. Brains were washed as described in washing solution shaking at 37 °C for 1 day at which the washing solution was refreshed five times. Brains were dehydrated with an ethanol/dH₂O series (50%, 70%, 90% and 100% ethanol) at room temperature for 2 h each step and incubated in 100% ethanol overnight. Subsequently, brains were incubated in DCM for 1 h before incubation in benzyl alcohol/benzyl benzoate until tissue transparency was reached.

Light-sheet imaging

Light-sheet imaging for c-Fos labeled brains was conducted through a ×4 objective lens (Olympus XLFLUOR 340) equipped with an immersion-corrected dipping cap mounted on an UltraMicroscope II (LaVision BioTec) coupled to a white light laser module (NKT SuperK Extreme EXW-12). The antibody signal was visualized using a 640/40 nm excitation and 690/50 nm emission filter. Tiling scans (3 × 3 tiles) were acquired with a 15–20% overlap, 60% sheet width and 0.027 NA. The images were taken in 16-bit depth and at a nominal resolution of 1.625 μm per voxel on the xy axes. In the z dimension we took images in 6-μm steps using left- and right-sided illumination. Whole-brain scans for microglia-labeled CX3CR1^GFP/+ brains were generated with the LaVision BioTec Ultramicroscope Blaze coupled with LaVision BioTec MI PLAN ×12 objective (0.53 NA (WD = 10 mm), nominal pixel size of 0.54 µm in xy). Stitching of tile scans was carried out using Fiji’s stitching plugin, using the ‘Stitch Sequence of Grids of Images’ plugin⁴⁵ and custom Python scripts.

ClearMap

ClearMap⁷ and the CellMap portion¹⁸ of ClearMap2 were used with adapted settings for thresholds and cell sizes that fitted to the higher resolution and different signal-to-noise ratios in our dataset. Segmentation masks were saved as tiff stacks by toggling the ‘save’ option in the last segmentation step. ClearMap was ported to Python (v.3.5) before use, but functioned identically⁴⁶. We only used the cell segmentation portions, no pre-processing (for example ClearMap2’s flat-field correction) or post-processing, such as atlas alignment, were performed. Both pipelines were run for an entire brain and subsequently subdivided into test patches that we used for the comparisons with DELiVR. For ‘optimized ClearMap’³, we performed the following pre-processing steps on our image stack: (1) Background equalization to homogenize intensity distribution and appearance of the c-Fos⁺ cells over different regions of the brain, using pseudo-flat-field correction function from Bio-Voxxel toolbox (https://doi.org/10.5281/zenodo.5986129). (2) Convoluted background removal, to remove all particles bigger than relevant cells. This was performed with the median option in the Bio-Voxxel toolbox. (3) A 2D median filter to remove remaining noise after background removal. (4) Unsharpen mask to amplify the high-frequency components of a signal and increase overall accuracy of the cell detection algorithm of ClearMap. (5) A z-wise removal of artifacts by manually selecting ROIs in Fiji. After pre-processing, ClearMap⁷ was applied by following the original publication and considering the threshold levels that we obtained from the pre-processing steps.

Ventricle masking

We wrote an automated pre-processing script that downsamples the image stack to an isotropic 25 × 25 × 25 µm per voxel and then applies a custom-trained random forest to identify ventricles. Specifically, we integrated Ilastik¹⁹ (v.1.4.0b8) with a 3D pixel classifier, which we trained on several downsampled brain image stacks to differentiate between ventricles and brain parenchyma. The pre-processing script then generates a 3D mask stack that our script upsamples to the original image stack dimensions, using bicubic interpolation to avoid aliasing artifacts at ventricle edges. It then masks each original z-plane image with the respective mask, pads it and returns a 16-bit image stack (saved as one big .npy file that can be read via np.memmap).

Annotation

VR annotation for c-Fos⁺ cells was carried out using Arivis VisionVR (v.3.4.0, Carl Zeiss Microscopy Software Center Rostock) or syGlass (v.1.7.2, ref. ¹²). For this purpose, the annotator was wearing a VR headset (Oculus Rift S) and carried out annotations in VR using hand controllers (Oculus Touch). Slice-by-slice annotation was carried out using ITK-SNAP (v.3.8, ref. ¹¹). For comparing VR and 2D-sliced based annotation, a 100³-voxel volume of c-Fos labeled brain was annotated by the participants and the time was recorded until the annotation task was finished. For training and testing our deep-learning network, we annotated a total of 48 × 100³ voxel patches in VR. All of our training and test patches were furthermore vetted by an expert biologist in ITK-SNAP to ensure that only cells were annotated. We evaluated the annotation quality using the formula of Dice as described below. For more details about the annotation process in VR, please see our ‘DELiVR handbook’ provided as a Supplementary Note. Microglia cell bodies were annotated in VR similar to c-Fos⁺ cells using Arivis VisionVR. Only the somata were annotated, while the microglia processes were excluded.

Deep learning

To automatically segment the cells in all brains, we trained a 3D BasicUNet⁴⁷ for DELiVR from the MONAI library⁴⁸. The annotated dataset of 48 × 100³ patches was split into nine patches for testing and 39 patches for training stratified by signal after manual ventricle masking. As an activation function, we chose Mish⁴⁹ and as optimizer Ranger21 (ref. ⁵⁰). As a loss function, we used binary cross-entropy loss¹⁷. For the training of 500 epochs, we set the initial learning rate to 1 × 10⁻³ and the batch size to four. The network was then trained on a single GPU (NVIDIA RTX8000). Instead of conducting model selection, we selected the last checkpoint after 500 epochs of training. To compare the DELiVR 3D BasicUNet with other segmentation models, we trained UNETR¹⁵, SegResNet¹⁶and MONAI DynUNET¹⁷ with similar specifications.

The microglia 3D BasicUNet model was trained in a similar fashion for 500 epochs using 161 patches containing 3,798 cells. These were split into 129 patches for training and 32 patches for testing. Training was performed on an NVIDIA A100 GPU.

Evaluation of the segmentation model

Evaluation of the deep-learning model was conducted in a twofold manner. First, we evaluated the volumetric segmentation quality by assessing, for each voxel, whether it was correctly classified as foreground or background using pymia⁵¹. A volumetric quality assessment gave us TPs, FPs, FNs and true negatives by comparing every prediction voxel with the reference annotation voxel. Additionally, we conducted an instance-wise assessment of the segmentation quality. Therefore, we assess detection rates on a single-cell (instance) level⁵². To fairly evaluate every cell irrespective of the patch, we aggregated the counts across all patches and computed the instance metrics globally⁵³.

Volumetric and instance scores were calculated according to the following equations:

$${\mathrm{Dice}}=\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}\qquad{\mathrm{Sensitvity}}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}\qquad{\mathrm{Precision}}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

Comparison with ClearMap, ClearMap2, ‘Optimized ClearMap’ and Ilastik was performed on a test brain to generate segmentations from which we cropped 100³-voxel patches to avoid artifacts that occur when the methods are applied at the patch level. These patches were then compared to our reference annotation using the same metrics as described above.

Atlas registration and statistical analysis

For atlas registration, we used mBrainAligner¹⁴, which worked well with our datasets (Supplementary Fig. 1). We manually saved the downsampled isotropic 25 × 25 × 25 µm per voxel stacks as .v3draw using Vaa3d⁵⁴. Subsequently, we wrote an automated script that aligned the image stacks to mBrainAligner’s 50 × 50 × 50 µm per voxel version of the Allen Brain Atlas CCF3 reference atlas, using the LSFM example settings with minor adaptations. Subsequently, we used mBrainAligner’s swc transformation tool to map the center-point coordinates of our c-Fos⁺ cells into atlas space.

Furthermore, we wrote a custom cell-to-atlas script (reusing parser code from VeSSAP⁵⁵ and the Allen Brain Atlas CCF3 atlas file as provided by the Scalable Brain Atlas⁵⁶) that filters the cells by size, with a user-defined upper and lower limit and returns two tables: a table with each cell as a row, including the region and Allen Brain Atlas color code, etc. and a region table with one region per row, in which the number of c-Fos⁺ cells per region is summarized. For all datasets, the post-processing script generates overview tables that contain cell counts for all regions. We used the latter for uncorrected Student’s t-tests. Finally, we implemented a level-aware multiple-testing script that compares groups at the Allen Brain Atlas’s 11 structure levels. We excluded the fiber tracts from our statistical comparisons.

Visualization

For visualizing the cells and regions in atlas space, we used BrainRender²⁰ (v.2) with a modified density plot function⁴⁶. To visualize the segmented cells in the original image space, we combined the area-wise color code from the Allen Brain Atlas with the 3D segment mask output by the connected component analysis. The result is a cell mask file with each cell being color coded according to the brain area that it belongs to, which makes overlaying with the original image data in for example Fiji easy and allows for direct visual inspection of the segmentation results. Finally, we used the Allen Institute for Brain Science’s cortical flat-map code (https://github.com/int-brain-lab/atlas) with adaptions⁴⁶ to include our heat maps.

DELiVR Docker and Fiji plugin

We packaged the DELiVR pipeline as provided in GitHub (https://github.com/erturklab/delivr_cfos) into a Docker container (base, nvidia/cuda:11.7.2-runtime-ubuntu22.04) including mBrainAligner¹⁴ (https://github.com/Vaa3D/vaa3d_tools/tree/master/hackathon/mBrainAligner), Ilastik (https://www.ilastik.org/download.html, v.1.4.0b8) and TeraStitcher portable⁵⁷ (https://github.com/abria/TeraStitcher/wiki/Binary-packages#terastitcher-portable-no-gui-only-command-line-tools, v.1.11.10). The code included Python (v.3.8), PyTorch (v.1.11), PyTorch Lightning (v.2.0.5), Nibabel (v.5.1.0), MONAI (v.1.2.0), SciPy (v.1.8.1), NumPy (v.1.24.4), Pandas (v.1.4.3), imglib2 (https://github.com/imglib/imglib2) and cc3d (https://github.com/seung-lab/connected-components-3d). For details, please see the Docker file on GitHub (https://github.com/erturklab/delivr_cfos/blob/main/Dockerfile).

We wrote the Fiji⁵⁸ (v.1.52p) plugin in Java (v.1.8, using Maven (v.3.9.5) and Jackson, https://github.com/FasterXML/jackson) as a front end. This provides a graphical user interface that compiles a config.json with path names and analysis parameters. Subsequently, the plugin calls the Docker container via a shell command and displays the progress of the pipeline. For a more detailed description, please see our ‘DELiVR handbook’ provided as a Supplementary Note.

Docker for training and Fiji plugin

We packaged the training code (https://github.com/erturklab/delivr_train) as a separate Docker container, which is also accessible via the Fiji plugin. The training plugin accepts annotated patches and trains a model specifically for this dataset. This model can then be imported into the inference pipeline for dataset-specific inference for any cell type. The Fiji training plugin compiles a config_train.json and arranges the file layout for the training Docker. It displays the training progress and shows the final test scores at the end.

Cell culture

C26 and NC26 colon cancer cells were cultured in high-glucose DMEM with pyruvate (Life Technologies, 41966052), supplemented with 10% fetal bovine serum (Sigma-Aldrich, F7524) and 1% penicillin-streptomycin (Thermo Fisher, 15140122) as described previously^28,59. Before using the cells for transplantation, cells had a confluence of 80%. Cells were trypsinized, counted and required cell numbers were suspended in Dulbecco’s PBS (Thermo Fisher, 14190250).

Animal experimentation

Experiments were carried out with male BALB/c mice aged 10–12 weeks. They were purchased from Charles River Laboratories, maintained on a 12-h light–dark cycle and fed a regular unrestricted chow diet. The set points in the animal room were set to 20–24 °C temperature and 45–65% humidity. The mice were injected with 1 × 10⁶ C26 or 1.5 × 10⁶ NC26 colon cancer cells^28,59 in 50 µl PBS subcutaneously into the right flank. Control mice were injected with 50 µl PBS. After 5 days from cell implantation, mice were monitored daily for tumor growth and body weight. Cachectic C26 tumor-bearing mice were considered cachectic when they had lost 10–15% of body weight. Mice were killed following deep anesthesia with a mix of ketamine/xylazine, followed by intracardiac perfusion with heparinized PBS (10 U ml⁻¹ heparin) and by a perfusion with 4% paraformaldehyde (PFA). Tissues and organs were dissected, weighed and post-fixed at 4 °C overnight. Animal experimentation was performed in accordance with European Union directives and the German Animal Welfare Act (Tierschutzgesetz) and approved by the state ethics committee and the Government of Upper Bavaria (ROB-55.2-2532.Vet_02-18-93).

The 6–8-week-old CX3CR1^GFP/+ (B6.129P-Cx3cr1tm1Litt/J) mice were purchased from The Jackson Laboratory (strain code 005582). They were deeply anesthetized using a combination of midazolam, medetomidine and fentanyl, intracardially perfused with 15 ml 0.01 M PBS solution (10 U ml⁻¹ heparin) and 15 ml 4% PFA solution. The brain was dissected, post-fixed in 4% PFA for 6 h, then proceeded for staining and clearing following the SHANEL protocol. CX3CR1^GFP/+ mice were killed for organ withdrawal (Tötung zu Wissenschaftlichen Zwecken/Organentnahme) in accordance with the German law for animal experiments (Tierschutzgesetz), paragraph 4, section 3.

Statistical analysis

Results from biological replicates were expressed as mean ± s.e.m. Statistical analysis was performed using GraphPad Prism (v.9). Normality was tested using Shapiro–Wilk normality tests. To compare two conditions, unpaired Student’s t-tests or Mann–Whitney U-tests were performed. A one-way ANOVA with Sidak’s post hoc test or Kruskal–Wallis tests with Dunn’s multiple comparison test were used to compare three groups. For the c-Fos⁺ density comparison between areas, we used two-sided t-tests followed by Benjamini–Hochberg multiple-testing correction with a false discovery rate (FWER) of 0.1, as implemented in SciPy statsmodels.stats.multitest.multipletests module (https://www.statsmodels.org/dev/generated/statsmodels.stats.multitest.multipletests.html).

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability

All data that support the findings of this study are available from the corresponding author. We provide the numerical source files of all figures in the supplementary material. Our training and test data as well as the trained network is available in GitHub at https://github.com/erturklab/delivr_cfos (ref. ⁶⁰) and https://github.com/erturklab/delivr_train (ref. ⁶¹). A subset of representative whole-brain scans is available at the EBI Bioimage Repository (accession code S-BIAD1019). Due to limitations to share large imaging data online, additional whole-brain scans (n = 27 whole brains, ~2 TB data) will be made available upon reasonable request. The Allen Brain Atlas (CCF3) was downloaded from the Scalable Brain Atlas repository at https://scalablebrainatlas.incf.org/mouse/ABA_v3. Source data are provided with this paper.

Code availability

All code to run our pipeline end-to-end is available in GitHub at https://github.com/erturklab/delivr_cfos (ref. ⁶⁰). Training code is available in GitHub at https://github.com/erturklab/delivr_train (ref. ⁶¹). Docker containers and the plugin can be obtained from https://discotechnologies.org/DELiVR. The code is released under the MIT license.

References

Ueda, H. R. et al. Tissue clearing and its applications in neuroscience. Nat. Rev. Neurosci. 21, 61–79 (2020).
Article CAS PubMed PubMed Central Google Scholar
Erturk, A. et al. Three-dimensional imaging of solvent-cleared organs using 3DISCO. Nat. Protoc. 7, 1983–1995 (2012).
Article CAS PubMed Google Scholar
Cai, R. et al. Panoptic imaging of transparent mice reveals whole-body neuronal projections and skull-meninges connections. Nat. Neurosci. 22, 317–327 (2019).
Article CAS PubMed Google Scholar
Belle, M. et al. Tridimensional visualization and analysis of early human development. Cell 169, 161–173 (2017).
Article CAS PubMed Google Scholar
Bhatia, H. S. et al. Spatial proteomics in three-dimensional intact specimens. Cell 185, 5040–5058 (2022).
Article CAS PubMed Google Scholar
Molbay, M., Kolabas, Z. I., Todorov, M. I., Ohn, T. L. & Erturk, A. A guidebook for DISCO tissue clearing. Mol. Syst. Biol. 17, e9807 (2021).
Article PubMed PubMed Central Google Scholar
Renier, N. et al. Mapping of brain activity by automated volume analysis of immediate early genes. Cell 165, 1789–1802 (2016).
Article CAS PubMed PubMed Central Google Scholar
Nectow, A. R. et al. Identification of a brainstem circuit controlling feeding. Cell 170, 429–442 (2017).
Article CAS PubMed Google Scholar
Topilko, T. et al. Edinger-Westphal peptidergic neurons enable maternal preparatory nesting. Neuron 110, 1385–1399 (2022).
Article CAS PubMed PubMed Central Google Scholar
Zhao, S. et al. Cellular and molecular probing of intact human organs. Cell 180, 796–812 (2020).
Article CAS PubMed PubMed Central Google Scholar
Yushkevich, P. A. & Gerig, G. ITK-SNAP: An intractive medical image segmentation tool to meet the need for expert-guided segmentation of complex medical images. IEEE Pulse 8, 54–57 (2017).
Article PubMed Google Scholar
Pidhorskyi, S., Morehead, M., Jones, Q., Spirou, G., & Doretto, G. syGlass: interactive exploration of multidimensional images using virtual reality head-mounted displays. Preprint at arXiv https://doi.org/10.48550/arXiv.1804.08197 (2018).
Silversmith, W. cc3d: Connected components on multilabel 3D & 2D images. Version 3.2.1. Zenodo https://doi.org/10.5281/zenodo.5719536 (2021).
Qu, L. et al. Cross-modal coherent registration of whole mouse brains. Nat. Methods 19, 111–118 (2022).
Article CAS PubMed Google Scholar
Hatamizadeh, A. et al. UNETR: Transformers for 3D Medical Image Segmentation. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 1748–1758 (2022)
Myronenko, A. 3D MRI brain tumor segmentation using autoencoder regularization. in Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Part II, pp. 311–320 (Springer International Publishing, 2018).
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J. & Maier-Hein, K. H. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18, 203–211 (2021).
Article CAS PubMed Google Scholar
Kirst, C. et al. Mapping the fine-scale organization and plasticity of the brain vasculature. Cell 180, 780–795 (2020).
Article CAS PubMed Google Scholar
Berg, S. et al. ilastik: interactive machine learning for (bio)image analysis. Nat. Methods 16, 1226–1232 (2019).
Article CAS PubMed Google Scholar
Claudi, F. et al. Visualizing anatomically registered data with brainrender. eLife 10, e65751 (2021).
Article CAS PubMed PubMed Central Google Scholar
Merkel, D. Docker: lightweight Linux containers for consistent development and deployment. Linux J. 2014, 2 (2014).
Salter, M. W. & Stevens, B. Microglia emerge as central players in brain disease. Nat. Med. 23, 1018–1027 (2017).
Article CAS PubMed Google Scholar
Kofler, F. et al. Approaching peak ground truth. in 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), 1–6 (IEEE, 2023).
Porporato, P. E. Understanding cachexia as a cancer metabolism syndrome. Oncogenesis 5, e200 (2016).
Article CAS PubMed PubMed Central Google Scholar
Baracos, V. E., Martin, L., Korc, M., Guttridge, D. C. & Fearon, K. C. H. Cancer-associated cachexia. Nat. Rev. Dis. Prim. 4, 17105 (2018).
Article PubMed Google Scholar
Schmidt, S. F., Rohm, M., Herzig, S. & Berriel Diaz, M. Cancer cachexia: more than skeletal muscle wasting. Trends Cancer 4, 849–860 (2018).
Article CAS PubMed Google Scholar
Argiles, J. M., Stemmler, B., Lopez-Soriano, F. J. & Busquets, S. Inter-tissue communication in cancer cachexia. Nat. Rev. Endocrinol. 15, 9–20 (2018).
Article PubMed Google Scholar
Morigny, P. et al. High levels of modified ceramides are a defining feature of murine and human cancer cachexia. J. Cachexia Sarcopenia Muscle 11, 1459–1475 (2020).
Article PubMed PubMed Central Google Scholar
Baker, A. et al. Specialized subpopulations of deep-layer pyramidal neurons in the neocortex: bridging cellular properties to functional consequences. J. Neurosci. 38, 5441–5455 (2018).
Article CAS PubMed PubMed Central Google Scholar
Alexander, A. S. et al. Egocentric boundary vector tuning of the retrosplenial cortex. Sci. Adv. 6, eaaz2322 (2020).
Article PubMed PubMed Central Google Scholar
Carstensen, L. C., Alexander, A. S., Chapman, G. W., Lee, A. J. & Hasselmo, M. E. Neural responses in retrosplenial cortex associated with environmental alterations. iScience 24, 103377 (2021).
Article PubMed PubMed Central Google Scholar
Kim, Y. et al. Mapping social behavior-induced brain activation at cellular resolution in the mouse. Cell Rep. 10, 292–305 (2015).
Article CAS PubMed Google Scholar
Jager, P. et al. Dual midbrain and forebrain origins of thalamic inhibitory interneurons. eLife https://doi.org/10.7554/eLife.59272 (2021).
Tyson, A. L. et al. A deep learning algorithm for 3D cell detection in whole mouse brain image datasets. PLoS Comput. Biol. 17, e1009074 (2021).
Article CAS PubMed PubMed Central Google Scholar
Menegas, W. et al. Dopamine neurons projecting to the posterior striatum form an anatomically distinct subclass. eLife 4, e10032 (2015).
Article PubMed PubMed Central Google Scholar
Matsumoto, K. et al. Advanced CUBIC tissue clearing for whole-organ cell profiling. Nat. Protoc. 14, 3506–3537 (2019).
Article CAS PubMed Google Scholar
Nath, V., Yang, D., Landman, B. A., Xu, D. & Roth, H. R. Diminishing uncertainty within the training pool: active learning for medical image segmentation. IEEE Trans. Med. Imaging 40, 2534–2547 (2021).
Article PubMed Google Scholar
Gal, Y. & Ghahramani, Z. Dropout as a Bayesian approximation: representing model uncertainty in deep learning. in International Conference on Machine Learning, 1050–1059. (PMLR, 2015).
Burfeind, K. G., Michaelis, K. A. & Marks, D. L. The central role of hypothalamic inflammation in the acute illness response and cachexia. Semin. Cell Dev. Biol. 54, 42–52 (2016).
Article PubMed Google Scholar
Campos, C. A. et al. Cancer-induced anorexia and malaise are mediated by CGRP neurons in the parabrachial nucleus. Nat. Neurosci. 20, 934–942 (2017).
Article CAS PubMed PubMed Central Google Scholar
Winnard, P. T. Jr. et al. Brain metabolites in cholinergic and glutamatergic pathways are altered by pancreatic cancer cachexia. J. Cachexia Sarcopenia Muscle 11, 1487–1500 (2020).
Article PubMed PubMed Central Google Scholar
Rolls, E. T., Cheng, W. & Feng, J. The orbitofrontal cortex: reward, emotion and depression. Brain Commun. 2, fcaa196 (2020).
Article PubMed PubMed Central Google Scholar
Basu, R. et al. The orbitofrontal cortex maps future navigational goals. Nature 599, 449–452 (2021).
Article CAS PubMed PubMed Central Google Scholar
Kann, O. The interneuron energy hypothesis: Implications for brain disease. Neurobiol. Dis. 90, 75–85 (2016).
Article CAS PubMed Google Scholar
Preibisch, S., Saalfeld, S. & Tomancak, P. Globally optimal stitching of tiled 3D microscopic image acquisitions. Bioinformatics 25, 1463–1465 (2009).
Article CAS PubMed PubMed Central Google Scholar
Negwer, M. et al. FriendlyClearMap: an optimized toolkit for mouse brain mapping and analysis. Gigascience 12, giad035 (2022).
Article PubMed Google Scholar
Falk, T. et al. U-Net: deep learning for cell counting, detection, and morphometry. Nat. Methods 16, 67–70 (2019).
Article CAS PubMed Google Scholar
Cardoso, J. M. et al. MONAI: an open-source framework for deep learning in healthcare. Preprint at arXiv https://doi.org/10.48550/arXiv.2211.02701 (2022).
Misra, D. A self regularized non-monotonic neural activation function. Preprint at arXiv https://doi.org/10.48550/arXiv.1908.08681 (2019).
Wright, L. & Demeure, N. Ranger21: a synergistic deep learning optimizer. Preprint at arXiv https://doi.org/10.48550/arXiv.2106.13731 (2021).
Jungo, A., Scheidegger, O., Reyes, M. & Balsiger, F. pymia: a Python package for data handling and evaluation in deep learning-based medical image analysis. Comput. Methods Prog. Biomed. 198, 105796 (2021).
Article Google Scholar
Pan, C. et al. Deep learning reveals cancer metastasis and therapeutic antibody targeting in the entire body. Cell 179, 1661–1676 (2019).
Article CAS PubMed PubMed Central Google Scholar
Kofler, F. et al. blob loss: instance imbalance aware loss functions for semantic segmentation. in Information Processing in Medical Imaging (Springer, 2023).
Peng, H., Ruan, Z., Long, F., Simpson, J. H. & Myers, E. W. V3D enables real-time 3D visualization and quantitative analysis of large-scale biological image data sets. Nat. Biotechnol. 28, 348–353 (2010).
Article CAS PubMed PubMed Central Google Scholar
Todorov, M. I. et al. Machine learning analysis of whole mouse brain vasculature. Nat. Methods 17, 442–449 (2020).
Article CAS PubMed PubMed Central Google Scholar
Bakker, R., Tiesinga, P. & Kotter, R. The scalable brain atlas: instant web-based access to public brain atlases and related content. Neuroinformatics 13, 353–366 (2015).
Article PubMed PubMed Central Google Scholar
Bria, A. & Iannello, G. TeraStitcher: a tool for fast automatic 3D-stitching of teravoxel-sized microscopy images. BMC Bioinform. 13, 316 (2012).
Article Google Scholar
Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).
Article CAS PubMed Google Scholar
Morigny, P. et al. Association of circulating PLA2G7 levels with cancer cachexia and assessment of darapladib as a therapy. J. Cachexia Sarcopenia Muscle 12, 1333–1351 (2021).
Article PubMed PubMed Central Google Scholar
DELiVR pipeline. (GitHub, 2024); https://zenodo.org/doi/10.5281/zenodo.10908720
Training code. (GitHub, 2024); https://zenodo.org/doi/10.5281/zenodo.10909998

Download references

Acknowledgements

We thank AIME (Berlin) for providing the processing time on their servers. We thank I. Horvath, H. Mai and L. Kümmerle from the Institute for Tissue Engineering and Regenerative Medicine (iTERM, Helmholtz Munich) for annotation tasks. We thank M. Ali and I. Horvath (iTERM) for helping with initial Docker container setup and slurm scripting. We thank L. Harrison from the Institute for Diabetes and Cancer (Helmholtz Munich) for editing the graphical summary. Figures 1a and 3a were generated with BioRender.com. We thank R. Zechner and M. Schweiger from the Institute of Molecular Biosciences (University of Graz) for kindly providing NC26 cancer cell lines. We thank M. Elsner and F. Hellal (iTERM) for proofreading and editing this manuscript. This work was supported by the Vascular Dementia Research Foundation, Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy within the framework of the Munich Cluster for Systems Neurology (EXC 2145 SyNergy, grant no. ID 390857198) and DFG (grant nos. SFB 1052, project A9; TR 296 project 03) as well as the German Federal Ministry of Education and Research (Bundesministerium für Bildung und Forschung) within the NATON collaboration (grant no. 01KX2121) and the HIVacToGC collaboration. This work was also supported by the European Research Council Consolidator grant (CALVARIA, grant no. GA 865323 to A.E.) and Nomis Heart Atlas Project Grant (Nomis Foundation). This work was supported by the European Research Council under the European Union’s Horizon 2020 research and innovation program (949017 to M.R.) and a grant from the Else-Kröner-Fresenius-Stiftung (2020 EKSE.23 to S.H.), as well as the Edith-Haberland-Wagner Stiftung. B.M., B.W. and F.K. are supported through the SFB 824, subproject B12, DFG through TUM International Graduate School of Science and Engineering, GSC 81. B.M. acknowledges support by the Helmut Horten Foundation.

Funding

Open access funding provided by Helmholtz Zentrum München - Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH).

Author information

These authors contributed equally: Doris Kaltenecker, Rami Al-Maskari, Moritz Negwer.

Authors and Affiliations

Institute for Diabetes and Cancer (IDC), Helmholtz Munich, Neuherberg, Germany
Doris Kaltenecker, Julia Geppert, Pauline Morigny, Maria Rohm, Stephan Herzig & Mauricio Berriel Diaz
Joint Heidelberg-IDC Translational Diabetes Program, Heidelberg University Hospital, Heidelberg, Germany
Doris Kaltenecker, Julia Geppert, Pauline Morigny, Maria Rohm, Stephan Herzig & Mauricio Berriel Diaz
German Center for Diabetes Research (DZD), Neuherberg, Germany
Doris Kaltenecker, Julia Geppert, Pauline Morigny, Maria Rohm, Stephan Herzig & Mauricio Berriel Diaz
Institute for Stroke and Dementia Research, Klinikum der Universität München, Ludwig-Maximilians-Universität LMU, Munich, Germany
Doris Kaltenecker, Rami Al-Maskari, Shan Zhao, Mihail Todorov, Zhouyi Rong & Ali Ertürk
Institute for Tissue Engineering and Regenerative Medicine, Helmholtz Munich, Neuherberg, Germany
Rami Al-Maskari, Moritz Negwer, Luciano Hoeher, Shan Zhao, Mihail Todorov, Zhouyi Rong, Johannes Christian Paetzold & Ali Ertürk
Department of Computer Science, TUM Computation, Information and Technology, Technical University of Munich (TUM), Munich, Germany
Rami Al-Maskari, Florian Kofler & Bjoern H. Menze
Center for Translational Cancer Research of the TUM (TranslaTUM), Munich, Germany
Rami Al-Maskari, Florian Kofler & Johannes Christian Paetzold
Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
Florian Kofler & Benedikt Wiestler
Helmholtz AI, Helmholtz Munich, Neuherberg, Germany
Florian Kofler & Marie Piraud
Department of Computing, Imperial College London, London, United Kingdom
Johannes Christian Paetzold & Daniel Rueckert
Department for Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
Bjoern H. Menze
Chair Molecular Metabolic Control, TU Munich, Munich, Germany
Stephan Herzig
School of Medicine, Koç University, İstanbul, Turkey
Ali Ertürk
Munich Cluster for Systems Neurology (SyNergy), Munich, Germany
Ali Ertürk
Deep Piction, Munich, Germany
Ali Ertürk

Authors

Doris Kaltenecker
View author publications
You can also search for this author in PubMed Google Scholar
Rami Al-Maskari
View author publications
You can also search for this author in PubMed Google Scholar
Moritz Negwer
View author publications
You can also search for this author in PubMed Google Scholar
Luciano Hoeher
View author publications
You can also search for this author in PubMed Google Scholar
Florian Kofler
View author publications
You can also search for this author in PubMed Google Scholar
Shan Zhao
View author publications
You can also search for this author in PubMed Google Scholar
Mihail Todorov
View author publications
You can also search for this author in PubMed Google Scholar
Zhouyi Rong
View author publications
You can also search for this author in PubMed Google Scholar
Johannes Christian Paetzold
View author publications
You can also search for this author in PubMed Google Scholar
Benedikt Wiestler
View author publications
You can also search for this author in PubMed Google Scholar
Marie Piraud
View author publications
You can also search for this author in PubMed Google Scholar
Daniel Rueckert
View author publications
You can also search for this author in PubMed Google Scholar
Julia Geppert
View author publications
You can also search for this author in PubMed Google Scholar
Pauline Morigny
View author publications
You can also search for this author in PubMed Google Scholar
Maria Rohm
View author publications
You can also search for this author in PubMed Google Scholar
Bjoern H. Menze
View author publications
You can also search for this author in PubMed Google Scholar
Stephan Herzig
View author publications
You can also search for this author in PubMed Google Scholar
Mauricio Berriel Diaz
View author publications
You can also search for this author in PubMed Google Scholar
Ali Ertürk
View author publications
You can also search for this author in PubMed Google Scholar

Contributions

D.K., R.A., S.H., M.B.D. and A.E. conceptualized the study. A.E. supervised and managed the entire project. D.K. performed in vivo work, antibody labeling, imaging and data processing. R.A. performed VR and deep-learning analysis. M.N. performed atlas registration, data analysis and interpreted the results. L.H. performed VR annotations. F.K. trained the deep-learning c-Fos models and wrote the initial inference pipeline. R.A., M.N. and F.K. improved the pipeline and ran data inference. M.N. and R.A. added visualization. R.A. and F.K. compared model performance metrics. R.A. wrote the Fiji plugin. M.N., R.A. and D.K. tested the Fiji plugin. M.N. and F.K. packaged the pipeline in a Docker container. D.K., R.A., M.N. and L.H. wrote the DELiVR handbook. S.Z. and M.T. supported imaging and antibody labeling. J.G. and P.M. supported in vivo experiments. Z.R. provided microglia data. R.A. trained the deep-learning microglia models. J.C.P. supported data interpretation. M.R. provided funding and useful discussions. A.E., M.R.B., S.H., M.R., B.W., D.R., B.M. and M.P. provided funding. S.H. supported supervision of the study. A.E. and M.B.D. supervised the study. D.K., R.A. und M.N. drafted the manuscript. A.E., D.K., M.N. and R.A. revised the manuscript. F.K. provided critical feedback on the revised manuscript.

Corresponding authors

Correspondence to Mauricio Berriel Diaz or Ali Ertürk.

Ethics declarations

Competing interests

A.E. is a co-founder of Deep Piction. The remaining authors declare no competing interests related to this work.

Peer review

Peer review information

Nature Methods thanks Hirofumi Kobayashi and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Nina Vogt, in collaboration with the Nature Methods team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 VR Segmentation in syGlass and 2D-slice-based segmentation using ITK-SNAP.

a, Volume of raw data (c-Fos labeled brain) that was generated by light-sheet microscopy and loaded into syGlass. Volume size represents 200³ voxel, rendered isotropically. b-d, Using VR, individual cells were segmented in syGlass by using three-dimensional euclidean shapes as ROI and adjusting a threshold until the segmentation was acceptable. Scale bar indicates 5 µm. e, ITK-SNAP view of a single plane of the image stack. Cells were labeled in 2D, slice by slice. Segmentations are color coded by cell ID.

Extended Data Fig. 2 DELiVR pre-processing automatically removes artefacts.

a-c, Horizontal view of an original image slice (a), the proposed mask (b) and the masked image slice generated (c). Scale bar = 1 mm. d, Architecture of the c-Fos deep-learning network; a MONAI 3D BasicUNet. e, Quantitative comparison (instance precision and sensitivity) of segmentation performance between deep-learning architectures and DELiVR’s 3D BasicUNet. f, Segmentation performance of non-deep-learning methods and DELiVR (Scores for DELiVR are the same as used in e).

Source data

Extended Data Fig. 3 Whole-brain segmentation output generated with DELiVR.

a, 3D visualization of a whole raw light-sheet image stack. b, 3D view of whole-brain segmentation output of detected cells by DELiVR. The area-wise color code from the Allen Brain Atlas was combined with the 3D segmentation. Thereby each cell is color coded according to the brain area it was detected in. The segmentation of cells is shown in the original image space. Scale bar = 500 µm. c, Visualization of the detected cells in CCF3 atlas space using BrainRender (same image as in Fig. 2e). Scale bar = 1 mm.

Extended Data Fig. 4 Tissue weights of mice with weight-stable cancer (NC26) and cancer-associated weight loss (C26).

a, Gastrocnemius (GC) muscle weight. n(PBS) = 12, n(NC26) = 8, n(C26) = 12, ****p < 0.0001,***p = 0.0004, One-way ANOVA with Sidak post hoc analysis. b, Epididymal white adipose tissue (eWAT) weight. n(PBS) = 12, n(NC26) = 8, n(C26) = 12, **p(PBS vs C26) = 0.0040,**p(NC26 vs C26) = 0.0015, Kruskal-Wallis test with Dunn´s multiple comparison test. c, Subcutaneous WAT (scWAT) weight. ***p = 0.0003, **p = 0.0019, Kruskal-Wallis test with Dunn´s multiple comparison test. d, Brain weight. n(PBS) = 12, n(NC26) = 7, n(C26) = 12, *p = 0.0479, One-way ANOVA with Sidak post hoc analysis. All data are presented as mean values +/− SEM.

Source data

Supplementary information

Supplementary Information

Supplementary Fig.1, Supplementary Table 1 and DELiVR handbook.

Reporting Summary

Supplementary Video 1

Annotation of cells in virtual reality using Arivis VisionVR.

Supplementary Video 2

Annotation of cells in VR using syGlass.

Supplementary Video 3

2D-slice-based annotation using ITK-snap.

Supplementary Video 4

Whole-brain with c-Fos⁺ cells detected by DELiVR depicted in the original image space. Cells are color coded by the area they belong to in the Allen Brain Atlas.

Supplementary Video 5

Example run of the ImageJ plugin on Ubuntu Linux.

Source data

Source Data Fig. 1

Numerical source data.

Source Data Fig. 2

Numerical source data.

Source Data Fig. 4

Numerical source data.

Source Data Fig. 5

Numerical source data.

Source Data Extended Data Fig. 2

Numerical source data.

Source Data Extended Data Fig. 4

Numerical source data.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Cite this article

Kaltenecker, D., Al-Maskari, R., Negwer, M. et al. Virtual reality-empowered deep-learning analysis of brain cells. Nat Methods (2024). https://doi.org/10.1038/s41592-024-02245-2

Download citation

Received: 03 June 2022
Accepted: 12 March 2024
Published: 22 April 2024
DOI: https://doi.org/10.1038/s41592-024-02245-2

Subjects

Abstract

Similar content being viewed by others

Main

Results

Reference annotation is faster in VR compared to 2D slices

DELiVR outperforms threshold-based c-Fos segmentation

DELiVR identifies activation patterns in tumor-bearing mice

Discussion

Methods

Whole-brain immunolabeling and clearing

Light-sheet imaging

ClearMap

Ventricle masking

Annotation

Deep learning

Evaluation of the segmentation model

Atlas registration and statistical analysis

Visualization

DELiVR Docker and Fiji plugin

Docker for training and Fiji plugin

Cell culture

Animal experimentation

Statistical analysis

Reporting summary

Data availability

Code availability

References

Acknowledgements

Funding

Author information

Authors and Affiliations

Contributions

Corresponding authors

Ethics declarations

Competing interests

Peer review

Peer review information

Additional information

Extended data

Supplementary information

Source data

Rights and permissions

About this article

Cite this article

Share this article

Search

Quick links