Introduction

Next-generation advanced nuclear reactors with improved safety and economics are the future of nuclear energy for the United States and worldwide1. The proposed fuel forms to be utilized in advanced reactors include U metal-based alloys, Tri-structural isotropic (TRISO) particles, Uranium dioxide fuel (UO2), MOX fuel, and molten salts2,3,4,5. Although fundamental understandings of most fuel forms are well known, developing predictive models and accelerating the qualification of new fuel forms are still challenging due to the lack of quantitative data and correlation between high-level burnup/temperature and microstructure evolution6,7,8,9. Especially, U metal-based alloys, as complex systems, may undergo microstructure transformation, phase redistribution, and thermal property degradation in the fuel phase, and embrittlement, hardening, and corrosion of the cladding and encapsulating materials during irradiation9,10,11. The abovementioned phenomena, including microstructure transformation, phase redistribution, and thermal property changing, are strongly interconnected and form a complex multi-factor problem. This problem, in turn, makes it difficult for conventional-empirical/physical-based fuel performance models to predict fuel behavior from estimated burnup and cladding temperature accurately12, 13.

U-10Zr-based metallic fuel is pursued as one of the leading candidate fuel forms for next-generation sodium-cooled fast reactors for the low fabrication cost and capability to achieve a higher burnup14, 16. Although thousands of U-10Zr fuel rods were irritated in test reactors, such as Experimental Breeder Reactor II (EBR-II) and Fast Flux Test Facility (FFTF) from the 1960s to the 1990s9, 14, a lack of fundamental understanding of the nuclear fuel microstructure properties and their evolution inside a reactor decelerate the qualification for commercial use of U-10Zr fuel. One of the critical factors affecting fuel performance is the Fuel Cladding Chemical Interaction (FCCI). The majority of FCCI in the cladding, i.e., wastage, is made up of lanthanides; therefore, it is important to understand how the lanthanides migrate and what compounds form the movement of lanthanides under a temperature gradient from the hot fuel center to the cool cladding surface. The lanthanide migration and its chemical interaction with cladding are critical aspects that may result in deterioration of the cladding mechanical properties, which could threaten fuel safety17. Moreover, PIE on advanced irradiated U-10Zr fuels in the Advanced Test Reactor (ATR) of Idaho National Laboratory (INL) discovered that the lanthanide particles/nodules located around the periphery of the pores18,19,20,21,22. Understanding the distribution changes of the pores in the cross-section of the advanced U-10Zr fuel will provide first-of-its-kind knowledge on the lanthanide transformation, which assists in revealing the mechanism of fuel cladding chemical interactions.

Accurate fission gas pore detection could provide trustable morphological distribution changes of pores along thermal gradient from hot fuel region to cold cladding rim and achieve a better understanding of the lanthanide movements. Cai et al. proposed a new framework to segment and classify fission gas pores in the (U, Zr) matrix regions of a U-10Zr annular fuel16. The authors applied image thresholding to segment fission gas bubbles and obtained good detection performance on a dataset of ~ 800 bubbles. As shown in Fig. 1, the bubble boundaries were manually labeled Fig. 1a by a material scientist, and the final annotated images were generated by filling the bubbles' contours (white image regions in Fig. 1c). A decision tree model was trained to classify the bubbles into different categories. Although the method achieved good bubble detection performance, the method cannot separate pores well, as shown in Fig. 1b, which causes incorrect calculation of the physical properties of pores, such as size, shape, and orientation. Although the performance of machine learning (ML) models largely depends on the quality of the training data, and their interpretability may be insufficient, they typically outperform traditional approaches in terms of reliability and accuracy. Many ML approaches can be found in literature, exploring complex and large datasets to gain insights and accelerate scientific discoveries, such as accelerating testing to develop new materials35, automating defect detection in electron microscopy23,24,25, and so on. Existing ML-based segmentation models achieved acceptable performance on natural images26, 27, biomedical images28, and material images29, but detecting fission gas bubbles on fuel cross-sections is more challenging since the bubbles' appearances, including gray level, size, and shape, vary greatly. Existing image processing techniques and pre-trained ML models cannot achieve good performance. Most ML models' performance heavily depends on the amount of annotated data used at the training stage. Advanced experimental characterization tools and modern imaging routinely provide high/ultrahigh-resolution images at an ever-increasing rate and volume. However, advanced experimental characterization tools lack sufficient and high-quality annotated training data.

Figure 1
figure 1

Sample results of fission gas bubble segmentation16.

Full size image

In this paper, a hybrid framework is proposed for more accurate and efficient fission gas bubble segmentation, and the contributions are summarized below.

  1. 1)

    The proposed hybrid segmentation only requires a small training set, and greatly reduces the time-consuming and expensive human efforts to manually annotate bubbles.

  2. 2)

    The proposed multitask instance segmentation network has a region segmentation branch and a boundary segmentation branch. It extracts and separates medium- and large-size bubbles accurately.

  3. 3)

    The proposed edge-based bubble segmentation approach generates accurate boundaries for small fission gas bubbles.

Proposed Method

Three significant challenges exist when extracting bubbles from PIE images using deep learning-based approaches. First, it requires enormous time and effort to manually label, namely marking the precise boundary for each bubble, a large dataset. In our PIE images, a significant number of bubbles are unlabeled. Second, many tiny bubbles widely exist in PIE images, and annotating the regions/boundaries of these bubbles is difficult. Third, existing instance segmentation approaches could be applied to extract and separate different bubbles. However, these segmentation approaches, e.g., Mask-RCNN30, are inefficient and inaccurate in segmenting many closely clustered objects. To address these challenges, the bubble segmentation tasks are decomposed into two independent processing steps. In the first step, we propose a multitask instance segmentation network that is trained using a small, annotated dataset to segment medium- and large-size bubbles. The second step is unsupervised and applies an edge detection approach to extract small bubbles.

Materials and data preparation

In recent years, INL has been the leading national laboratory for research and development (R&D) on metallic fuel. Thanks to the development of advanced characterization capabilities at the Material Fuel Complex (MFC), it is now possible to revisit the vast available PIE data accumulated in the past and the newly established PIE data ranging from sub-nanometer to micrometer to obtain new findings. In a recent study, Cai et al. proposed a segmentation framework on ~ 800 partially annotated bubbles on only three 500x-magnification image patches of a U-10Zr annular fuel named AF16. In this study, we collected 585 scanning electron microscope (SEM) image patches under 1000x-magnification of a partial cross-section of another advanced U-10Zr fuel named AF231. The patches were collected from the hot center to the cladding. To design a DL-based model, sufficient training data with image annotations and original images were needed. Moreover, the data of the two fuels under different magnifications will reveal the features of bubbles differently, for example, size, contour, and texture, even for the same bubble as shown in Fig. 3. Under this circumstance, we developed an interactive annotation tool to label the fission gas bubbles.

Multitask instance segmentation network (MTIS-Net) for extracting medium- and large-size bubbles

A novel instance segmentation network is proposed to extract and separate medium- and large-size bubbles from SEM images. It treats each bubble as an instance. As shown in Fig. 2, the proposed network consists of one encoder and two decoder subnetworks. The encoder uses convolutional and pooling layers to extract meaningful features from input images at different scales. A ResNet-50 network32 is applied as the backbone network in the encoder. The first decoder is developed to segment bubble regions, and the second is to detect bubble boundaries. The results from the two decoders are combined to achieve an instance segmentation. The two decoders share the same feature input from the encoder. For preserving details, the intermedium feature maps of the encoder are passed to the corresponding layers in both decoders by skipping connections. These two decoders use the standard U-Net33 decoder architecture.

Figure 2
figure 2

The proposed efficient instance segmentation framework.

Full size image

The Dice loss function34 is used in the bubble region segmentation branch. The Dice loss measures the quantitative difference between the region segmentation results (\({\widehat{p}}_{1}\)) and the ground truth (\({y}_{1}\)), and it is defined by

$${L}_{Dice}\left({y}_{1},{\widehat{p}}_{1}\right)=1-\frac{\sum 2{y}_{1}{\widehat{p}}_{1}+1}{\sum ({y}_{1}+{\widehat{p}}_{1})+1}$$
(1)

where \({y}_{1}\) is a 2D matrix that contains binary values in which value 1 denotes a bubble pixel, and value 0 represents a non-bubble pixel; and values in \({\widehat{p}}_{1}\) are the actual predictions produced by the bubble region segmentation network. The numerator and denominator are added to 1 as the smooth term to avoid division by zero.

The weighted binary cross-entropy loss function is used in the boundary detection branch, and it is given by.

$${L}_{WBCE}\left({y}_{2},{\widehat{p}}_{2}\right)=-\left[{w}_{0}\left(1-{y}_{2}\right)\mathrm{log}\left(1-{\widehat{p}}_{2}\right)+{w}_{1}{y}_{2}\mathrm{log}\left({\widehat{p}}_{2}\right)\right]$$
(2)

where \({y}_{2}\) is a binary map that uses 1 s to denote bubble boundary pixels; \({\widehat{p}}_{2}\) is the prediction of bubble boundaries; and \({w}_{0}\) and \({w}_{1}\) are the weights of boundary and non-boundary terms, respectively. In experiments, \({w}_{0}\) is set to 0.1, and \({w}_{1}\) is set to 0.9.

The segmentation results are produced by subtracting bubble boundaries from bubble regions. The boundaries can disconnect touching bubble regions. During the post-processing, the final bubble instances are generated by connecting bubble pixels using the 8-adjacency system35; and the morphological dilation operation is applied to compensate for the shrinking of bubble areas.

Small bubble segmentation using edge detection

As shown in Figs. 3f and 4b, small bubbles are usually presented as black or grey dots in SEM images. These dots have homogeneous interior intensities. The grey bubbles have similar intensities to the background areas; therefore, it is difficult to differentiate grey bubbles from the background by using intensity thresholds. However, the grey bubbles have dark boundaries that separate bubble regions and backgrounds, and it is more appropriate to use edge detection approaches to detect small bubbles.

Figure 3
figure 3

Image samples from two annular fuels under different magnifications.

Full size image
Figure 4
figure 4

Sample results of small bubble segmentation.

Full size image

In this work, we use the Canny edge detection36 approach which includes five steps, (1) applying a \(5\times 5\) Gaussian low-pass filter (\(\sigma =1\)) to remove the image noise; (2) calculating the intensity gradient of the image to identify pixels with sharp intensity changes (potential edge points); (3) applying non-maximum suppression to eliminate noise; (4) applying double thresholding to determine potential edges; and (5) suppressing weak edges. The Canny edge detection approach can capture high-quality edges and mitigate the impact of image noises.

The Canny edge detection is applied to extract small bubbles. Most medium- and large-size bubbles have long, irregular-shaped, and fuzzy boundaries; and edge detection approaches can only produce small, disconnected boundary pieces, but closed boundaries could be generated for small bubbles because they have more homogeneous boundary pixels. The final bubble regions are generated by applying the flood fill algorithm37 to fill the closed edges. Non-closed edges are removed from the final results. Figure 4 shows sample results of small bubble segmentation.

Experimental Results

Setup

Dataset

The images are cropped into 515 × 512 non-overlapped patches. The training set contains 18 image patches and 827 annotated bubbles. It only has precise contours labeled by experts, noted as ground truths, for medium- and large-size bubbles. Due to the significant resources required, the number of medium- and large-size bubbles in the training images is small. The test set contains 24 image patches and 685 bubbles.

Training

We adopted a ResNet-50 backbone as the encoder because it has been demonstrated strong performance on a variety of tasks in previous studies and strikes a balance between computational efficiency and accuracy. The used backbone was pretrained using ImageNet. In training, we use an Adam38 optimizer with a 0.001 learning rate and train the network for 100 epochs. An exponential learning rate scheduler with \(\gamma =0.97\) is used to decay the learning rate after every epoch. Training images are randomly augmented in every epoch with Gaussian blur, Gauss noise, brightness, horizontally/vertically flipping, contrast, scaling, and rotating. The batch size is set to 8.

Evaluation metrics

Intersection over union (IoU) is a popular metric for measuring overlap between multiple objects, especially for segmentation tasks. The higher value of IoU indicates the prediction/segmentation results align well with the actual results. In this study, both pixel-level and instance-level performances were investigated. The pixel-level evaluation assigned each pixel as pore or non-pore. Instance-level differentiates between individual pores with the pixel-level results. As the test image is partially labeled, both the instance-level recall ratio \({\mathrm{R}}_{\mathrm{iou}}^{\mathrm{I}}\) and the pixel-level ratio \({\mathrm{R}}_{\mathrm{iou}}^{\mathrm{P}}\) are used to evaluate the performance. The recall ratio is defined by Eq. (3).

$${\mathrm{R}}_{\mathrm{iou}}=\frac{\left|\mathrm{TP}\right|}{\left|\mathrm{P}\right|}$$
(3)

where \(\left|\mathrm{TP}\right|\) denotes the number of accurately segmented bubbles (\({\mathrm{R}}_{\mathrm{iou}}^{\mathrm{I}}\)) or pixels (\({\mathrm{R}}_{\mathrm{iou}}^{\mathrm{P}}\)) and \(\left|\mathrm{P}\right|\) represents the number of total bubbles or pixels in ground truths. For the instance-level recall ratio \({\mathrm{R}}_{\mathrm{iou}}^{\mathrm{I}}\), the total number of all labeled bubbles is treated as the \(\left|\mathrm{P}\right|\). Each ground truth bubble is paired with a predicted bubble with the largest IoU among all predictions. The \(\left|\mathrm{TP}\right|\) counts all paired predictions that have IoU values with a ground truth greater than a threshold. A set of values (\([0.5, 0.6, 0.7, 0.8, 0.9]\)) are used as thresholds in experiments. The pixel-level recall, \({\mathrm{R}}_{\mathrm{iou}}^{\mathrm{P}}\), simply calculates the number of true bubble pixels over all labeled bubble pixels.

MTIS-Net

Figure 5 shows an example of the outputs of the proposed MTIS-Net. The bubble region segmentation branch generates binary segmentation results for medium- and large-size bubbles. As shown in Fig. 5b, most bubbles are well-segmented, but some bubbles are connected. Figure 5c shows the segmentation results of bubble boundaries generated by the boundary segmentation branch of MTIS-Net. The final segmentation results are shown in Fig. 5d, and different bubbles are illustrated using different colors.

Figure 5
figure 5

Sample results of the MTIS-Net.

Full size image

Small bubble segmentation

Small bubbles are segmented using the Canny edge detection approach. As shown in Fig. 6, Fig. 6a is an original image patch from the studied fuel; Fig. 6b is the manually partial annotated ground truth (GT) image of Fig. 6a,c illustrates the bubble segmentation result from the multitask model discussed in Sect. "mtis-net"; Fig. 6d shows the results of the proposed small bubble segmentation approach; and illustrates that most small bubbles have regular shapes. The merged results of the MTIS-Net and edge-based approach are shown in Fig. 6e. Figure 6f uses blue contours to show medium- and large-size bubbles identified by MTIS-Net, and red contours to demonstrate small bubbles detected by the edge-based approach.

Figure 6
figure 6

Merged Results. (a) Original image; (b) partial annotated ground truth (GT); (c) the result of MTIS-Net; (d) the result of small bubble segmentation; (e) the result by merging (c) and (d); (f) The blue contours are predicted by multi-task U-Net, and red are from small detection.

Full size image

Overall performance

Table 1 demonstrates the instance-level recall ratios under different IoU thresholds (0.5, 0.6, 0.7, 0.8, and 0.9). In the "Bubble Numbers" table section, columns named " ≥ IoU threshold" show the number of bubbles that are successfully segmented based on specific IoU thresholds; and the "GT" column shows each image's total number of labeled objects. The "Instance-level Recall" section shows the corresponding recall values using different thresholds. In classic object localization and instant segmentation tasks, an IoU threshold of 0.5 is described as a loose criterion of the correct detection39, 40; and the value 0.75 is considered a “strict criterion”.

Table 1 Instance-level evaluation. Columns of ≥ 0.5 through to ≥ 0.9 represent the count of detected bubbles that match ground truths with an IoU equal to or larger than the threshold.
Full size table

Our method presents a recall ratio of 0.92 with IoU ≥ 0.5, and a recall ratio of 0.6 with IoU ≥ 0.9. With the increasing IoU thresholds, the instance level recall ratios drop slowly, which shows the precision and stability of the proposed method.

Even though GTs of bubble boundaries were created by experts, in specific situations, it is still challenging to appropriately define if a bubble should be separated into two. As shown in Fig. 7, the bubble inside the red box can be considered as one large bubble or two smaller bubbles. The expert labeled it a single bubble, but the model considered it two separate bubbles. Such a problem creates uncertainty in the instance-level evaluation. Hence, we also report the pixel-level recall ratio to provide a more comprehensive evaluation.

Figure 7
figure 7

Difficulties in separating bubbles.

Full size image

Table 2 shows the pixel-level recall ratios for each image and the entire test set. The “True Positive” column is the number of bubble pixels that are correctly classified. The "Total Positive" column represents each image’s labeled bubble pixels. The average recall ratio is 0.93. The highest and lowest recall ratio is 0.96 and 0.84, respectively.

Table 2 Pixel-level recall ratio.
Full size table

Comparison with existing work

In the previous study by Cai et al., the bubbles were segmented with a pure image processing process that utilized the thresholding method35. In this section, we compare the thresholding method and the proposed method on our dataset. As shown in Fig. 8, the thresholding method tends to over-segment the bubbles, and the proposed method can generate more accurate results. Table 3 shows the instance-level recall ratios of the thresholding method and proposed method. The recall ratio of the thresholding method reaches 0.54 with the loose criterion 0.5. Meanwhile, its recall ration is only 0.03 with the 0.9 IoU threshold, and the thresholding method has poor performance in segmenting objects precisely. Compared with the instance-level recalls in Table 1, the recall ratio of the proposed method is 0.92 at IoU ≥ 0.5. The improvement of the proposed method is over 70%.

Figure 8
figure 8

Comparison of the thresholding and proposed methods.

Full size image
Table 3 Instance-level evaluation of the thresholding method 16 and the proposed method.
Full size table

Discussions

The expensive cost of preparing ground truths is one of the major challenges in ML-based fission gas bubble segmentation, especially labeling tiny bubbles. The training images are not fully labeled. Part of medium-sized, large-sized, and all small bubble areas are marked as background during training. This circumstance hinders the model from fully learning the concept of the targeted object.

Due to the incomplete ground truth, the model's performance is evaluated by recall ratio. We cannot conduct a comprehensive evaluation using more conventional instances or semantic segmentation metrics, such as mean average precision (mAP) and IoU. The drawback of recall is that it only counts on provided ground truths but cannot fully reveal the performance with the occurrence of over-segmentation.

The completely labeled ground truth with any sized bubbles is a solution to overcome the existing defects of training and evaluation. However, creating a large number of precise labels for training and evaluation is challenging. A more feasible way is to develop a semi-supervised model that can learn from incompletely labeled images. Therefore, we could train a model on a partially labeled set and evaluate it on a smaller, fully labeled set.

Conclusion

In this study, we propose an instance-level PIE bubble segmentation approach. The proposed approach consists of a novel multitask instance segmentation network (MTIS-Net) and an image processing step for dealing with bubbles of different sizes. The proposed method obtains excellent performance with a small training set. Our model shows outstanding improvement by comparing the previously proposed thresholding method. The better performance provides more accurate quantitative results of fission gas bubbles, e.g., the distributions of different fission gas bubble classes, especially those with lanthanides. The model will be unitized on the other U-10Zr annular fuels and will contribute to building the relationship between thermal gradients and lanthanide movements. Moreover, the proposed method is promising to be applied to segmentation tasks of many materials.