Open Access
Add to BibSonomy
Abstract
Ultrasound (US)-guided diffuse optical tomography (DOT) is a portable and non-invasive imaging modality for breast cancer diagnosis and treatment response monitoring. However, DOT data pre-processing and imaging reconstruction often require labor intensive manual processing which hampers real-time diagnosis. In this study, we aim at providing an automated US-assisted DOT pre-processing, imaging and diagnosis pipeline to achieve near real-time diagnosis. We have developed an automated DOT pre-processing method including motion detection, mismatch classification using deep-learning approach, and outlier removal. US-lesion information needed for DOT reconstruction was extracted by a semi-automated lesion segmentation approach combined with a US reading algorithm. A deep learning model was used to evaluate the quality of the reconstructed DOT images and a two-step deep-learning model developed earlier is implemented to provide final diagnosis based on US imaging features and DOT measurements and imaging results. The presented US-assisted DOT pipeline accurately processed the DOT measurements and reconstruction and reduced the procedure time to 2 to 3 minutes while maintained a comparable classification result with manually processed dataset.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Multispectral diffuse optical tomography (DOT) utilizes near-infrared (NIR) light to reconstruct the distributions of optical absorption coefficients at selected wavelengths that are used to map the hemoglobin concentration of a disease site in biological tissue [1]. DOT has been explored extensively for breast cancer diagnosis and treatment response monitoring [2–5]. However, DOT reconstruction is an ill-posed and ill-conditioned problem, and thus other conventional image modalities, such as x-ray computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound (US), have been explored to provide a priori knowledge for DOT reconstruction [6–9]. X-ray mammography [6,10,11] and digital breast tomosynthesis (DBT) [6,10–14] have been extensively evaluated as a prior imaging guidance to DOT to improve DOT localization and reconstruction accuracy. The strength of mammography or DBT-guided DOT is its easy adaptation to the breast cancer screening population, however, the compression of mammography or DBT imposes a challenge to co-registered DOT imaging due to the hemoglobin distribution changes caused by the compression. MRI-guided DOT [7,15] has also been extensively investigated because MRI can provide detailed 3-D high-resolution imaging guidance to DOT. However, MRI is mainly used for high-risk breast patients, and its high cost restricts the use of MRI-guided DOT. On the other hand, US is a low cost imaging modality with an imaging resolution well suited to guiding DOT imaging reconstruction. Our group has pioneered US-guided DOT and translated it to breast cancer diagnosis and neoadjuvant treatment monitoring [9,16]. In breast DOT reconstruction, the prior information can either assist in reducing the number of unknown variables during spatial volume segmentation (referred as hard-priors) or penalize the changes within the specific regions (referred as soft-priors) [10]. We have developed a dual-mesh approach to use US-visible lesions to segment the breast volume into a lesion region and a background region, thus reducing the total number of unknown variables in the DOT reconstruction and improving the DOT reconstruction accuracy.
Our group developed a frequency domain US-guided DOT system for breast cancer diagnosis and cancer treatment monitoring [9,17]. We have investigated different data pre-processing algorithms for data cleaning and outlier removal, as well as different US-guided DOT reconstruction algorithms, including optimization algorithms [18,19], combined US and DOT imaging feature-based diagnostic algorithms [20], and deep-learning-based imaging algorithms [21–23].
In recent years, deep learning neural network has emerged as a powerful tool for improving DOT reconstruction. Feng et al. implemented Z-net for MRI-guided near-infrared spectral tomography to recover the concentrations of oxy-hemoglobin, deoxy-hemoglobin, and water [15,24]. Yedder et al. proposed a multitask deep learning model to improve the accuracy of DOT reconstruction and localization [25]. Additionally, Sabir et al. utilized deep learning models to estimate bulk optical properties like absorption and scattering coefficients [26].
However, implementing a sequence of data pre-processing procedures before imaging reconstruction is still done manually, which hinders real-time diagnosis. The ultimate clinical translation of the US-guided DOT technique requires a user-friendly and robust imaging pipeline. Recently, Sciacca et al. proposed a DOT pipeline study for breast cancer diagnosis [27]. They performed forward measurements, reconstruction, and classification on simulated breast data and achieved excellent classification results. However, simulation alone, without accurate modeling of the lesion’s heterogeneity, the chest wall depth/position, the probe-to-tissue coupling, and the system noise, is insufficient for translation to clinical applications. To accelerate the eventual clinical adoption of US-guided DOT, we need a robust automated pipeline that incorporates clinical data and requires minimal user interaction.
Here, we present an automated US-guided DOT pipeline that accelerates both data pre-processing and imaging reconstruction. In pre-processing, it automatically selects the DOT measurements on both the lesion and reference sides with acceptable motion profiles, then it identifies the mismatch between the lesion and the reference measurements and removes outliers to generate the perturbation. In imaging reconstruction, it measures the lesion size and depth from the co-registered US images and crops the US images. Then, it extracts the features from the cropped US images and the DOT perturbations to make a first stage near real-time diagnosis that singles out benign lesions that do not require DOT imaging reconstruction. The remaining lesions, including benign but suspicious lesions and malignant lesions, are then diagnosed in a second stage. The DOT images for this suspicious group are reconstructed with the target information. A deep learning model for DOT image confidence assessment is utilized to quantify the DOT image quality. The second stage diagnosis model utilizes the features from the DOT reconstruction, the DOT perturbation, and the US images to predict the probability of malignancy. To the best of our knowledge, this is the first near real-time imaging platform employing dual-modality US and DOT for breast cancer diagnosis.
2. Methods
2.1 US-guided DOT system and data calibration
A compact US-guided DOT frequency domain system previously developed by our group was used to collect phantom and clinical data [17,28]. The system and its handheld hybrid probe are shown in Fig. 1. Four laser diodes, with wavelengths 730, 785, 808, and 830 nm, are sequentially activated for optical measurements of tissue. Nine illumination source fibers deliver light that is collected by fourteen parallel photomultiplier (PMT) detectors on the opposite side, with source-detector separations ranging from 3.2 cm to 8.5 cm. A local 140.02 MHz oscillator modulates the laser signal, and the detected signal is demodulated to 20 kHz. The signal is further amplified and filtered, and then digitized and sampled by an analog-to-digital converter (ADC). The data acquisition time (DAQ) is 3 to 4 seconds for each data set. A total of 15 to 20 data sets were taken at the lesion side and normal contralateral reference side. The difference between lesion and reference side was used to compute perturbations for imaging reconstruction.
Since the powers of each laser diode and the gains of each particular PMT detector are different, the source powers and detector gains were calibrated by measuring a homogenous intralipid solution with known background optical absorption and scattering properties [17].
DOT measurements are influenced by the wavelength-dependent signal-to-noise ratio (SNR), skin-fiber coupling, motion during data acquisition, and the matching between the lesion-side breast and normal contralateral reference breast. These challenges, together with tissue heterogeneity, lead to potential errors and outliers in the perturbation measurements. To overcome such challenges, the proposed automated pipeline is aimed at optimizing the quality of the measurements and the robust imaging reconstruction. The following sections describe the two segments of the pipeline that perform data pre-processing and imaging reconstruction.
2.2 Overall structure of the automated data processing and imaging and diagnosis pipeline
The proposed automated US-guided DOT pipeline includes automated data pre-processing and US-guided DOT reconstruction. The final output is the diagnostic probabilities of malignant vs. benign lesions. As shown in Fig. 2, the data pre-processing pipeline includes motion detection, multilayer perception neural network based reference selection (which is developed in this manuscript), and outlier removal. The reconstruction pipeline starts from US images and DOT perturbations, which are DOT measurements of the lesion normalized by subtracting the contralateral normal reference breast. The CNN neural nets of US images and DOT histograms, developed earlier by our group [29], are used to extract features for first stage diagnosis of benign lesions. Lesions not diagnosed as benign in the first stage are labeled as suspicious and are passed on for US-guided DOT image reconstruction. The imaging evaluation neural net CNN developed in this manuscript is used to evaluate DOT image quality and decide if the artifact removal algorithm should be used to clean up the images. Then a CNN neural network, developed by our group earlier [30], extracts DOT image features and combines them with US features and DOT histogram features to conclude the second stage diagnosis for suspicious lesions.
Fig. 2. Overall structure of data pre-processing pipeline and US-guided DOT reconstruction pipeline.
2.3 DOT data pre-processing
The workflow of the pre-processing is shown in Fig. 3. In the pre-processing, motion during data acquisition is first detected and measurements with motion are excluded. Then, the normalized data from the measured differences between the lesion and reference sides are used to form the perturbation, and a multilayer perceptron (MLP) model is employed to detect the mismatch in the perturbation. After that, the best reference undergoes an outlier removal to generate a clean and matched perturbation for image reconstruction.
Fig. 3. (a) Data pre-processing workflow. Two sets of measurements, acquired at the lesion side and the reference side, are used to compute perturbation, which is assessed by matching between the two sides. (b) Architecture of MLP for mismatch detection.
2.3.1 Motion detection
Because the hybrid probe is hand-held and the DAQ time for each data set is several seconds, patient and operator motion can occur during data acquisition. Consequently, we developed an automated algorithm that evaluates and excludes motion data from both the reference and target measurements. Motion data is identified by repeating the data acquisition three times, and it is then removed if the amplitude profiles of any two consecutive data sets do not overlap to a sufficient degree. Specifically, an iterative, feature-based motion detection method is applied to all data sets to single out the data sets with motion. During data acquisition, the US-guided DOT system rapidly acquires three repeated measurements at each location. Point-wise differences between the first and second measurements (set 1 vs. set 2), and the second and third measurements (set 2 vs. set 3) are calculated. The mean and variance of the differences are calculated by
Figure 4 illustrates the procedure for motion detection. Prior to the calculation, a predetermined set of thresholds $[{{T_m},{T_v},{T_d}(i )} ]$ is established to identify the target and reference measurements that exhibit an acceptable motion profile. To determine whether a measurement indicates motion or not, we compare its mean and variance separately with their threshold values, ${T_m}$ and ${T_v}$. Additionally, we count the number of pointwise differences that exceed the difference threshold, ${T_d}(i )$. Measurements are then labeled accordingly. Where the number of acceptable measurements falls below 20% of the total measurement set, we relax the criteria and repeat the examination until enough measurements are available.
2.3.2 Mismatch classification
As mentioned above, DOT measurements are obtained from both the lesion and reference sides. The DOT perturbation, ${U_{sc}}$, is then calculated as
In a semi-infinite homogeneous medium, the logarithmic amplitude of a DOT measurement, which is the logarithm of the product of the amplitude and the square of the source-detector distance $r $ for each pair, should be negatively and linearly related to r, while the phase is proportional to r [31]. Equation (4) and (5) show the linear relationships:
where ${\mu _a}$ is the absorption coefficient and D is the photon diffusion coefficient of the homogeneous tissue. The values of $ {\mu _a}$ and D for the reference side are used to compute the weight matrix for imaging reconstruction.A clean and informative perturbation, an accurate absorption coefficient, and accurate photon diffusion coefficient estimations are all essential for a robust DOT reconstruction. However, perturbations are highly influenced by the chest wall’s angle and depth in the contralateral measurements, and the accuracy of optical coefficient estimations is affected by the presence of the chest wall. Thus, a matched reference measurement, including a matching depth and angle of the chest wall, helps to generate an informative perturbation and to estimate optical properties with good accuracy. With the hand-held US-guided probe used in this work, US localizes the lesion and visualizes the chest wall. Figure 5 shows matched and mismatched chest wall US images and their corresponding 2-D perturbation plots. The x-axis is the real-part of the perturbation and the y-axis is the imaginary part of the perturbation. The blue lines in the US images indicate the depths and tilt angles of the chest walls. In Fig. 5, the target and matched reference US images show the chest walls with similar depths and tilt angles, while the chest wall from the mismatched reference has a shallower depth, causing abnormality in the perturbation.
Fig. 5. Examples of matched and mismatched chest walls and their corresponding perturbations. The blue lines in the US images indicate the depth of the chest wall marked by an expert in US.
In principle, the amplitude of the lesion side ${A_l}(i )$ measurements is smaller than that of the reference side ${A_r}(i ),$ due to higher absorption by the target in general. From our previous Monte Carlo (MC) simulation study [32], we learned that extreme cases produce disproportionate measurements. For example, a 4 cm diameter lesion with 10 times higher optical contrast than the background tissue introduced a mere 22 degree phase delay, which means that the maximum phase shift for all source-detector pairs was no more than 45 degrees. For the matched case in Fig. 5, the real part of Eq. (1) is consequently distributed along the negative x-axis, albeit the imaginary part is located around zero. However, in the mismatched case, the data points are more spread out and tend to be located along the positive x-axis.
The MLP has proven quite effective in separating datasets which were not linearly separable, and it is easy and flexible to apply [33]. Therefore, an MLP was employed to classify the mismatch in the perturbation. The input, which consisted of the real and imaginary parts of the perturbation from 9 sources and 14 detectors, had a total of 252 features. Figure 3(b) showed the structure of the MLP, with four fully connected hidden layers. We trained the MLP with 7200 sets of simulated measurements generated by applying the finite element method (FEM) and Monte Carlo (MC) method, fine-tuned it with 1612 sets of clinical data, and tested it on the rest of the clinical data. The training epoch was 10, and the learning rate was set to be 0.0001, with a decay of 0.5.
The MLP model predicted the measurements and sorted the reference data based on its prediction scores. Subsequently, the best reference measurement was selected and examined using the local outlier factor (LOF) method [34]. By measuring the local deviation of a data point relative to its neighbors, this method effectively excludes outliers and generates a clean perturbation.
2.4 US-guided DOT imaging
After pre-processing, we integrated the reconstruction and classification steps. Initially, a semi-segmentation model was employed to crop the US image and provide such parameters as the target shape and depth. Subsequently, based on the perturbation, we generated a bivariate histogram with 32 × 32 bins. Then we used two neural networks to extract the features from the cropped US image and the histogram, and we averaged the first stage benign or suspicious predictions from the two modalities. For a suspicious lesion, we proceeded to the second stage, in which we reconstructed the DOT images and employed a deep learning model to assess the confidence level of the image quality, considering factors such as the target’s location and image artifacts. DOT image artifacts can occur when lesions are in shallow depth. Thus, we implemented a manual artifact removal algorithm which has been tested and can effectively remove artifacts [35]. Details about the DOT reconstruction can be found in Ref. [36]. In brief, we modeled photon migration using a diffusion equation of the photon-density wave and applied Born approximation to relate the scattered field, ${U_{sc}},$ to the absorption coefficients, $\delta {\mu _a},$ as:
where W is the weight matrix calculated from the diffusion equation for a semi-infinite medium, m and n represent the number of measurements and voxels. To solve $\delta {\mu _a}$, we formulated an inverse problem as:
In clinical trials, perturbation is a crucial prerequisite for the reconstruction process. Here, however, the low data quality in some cases caused the perturbation generated using all reference measurements to fail to identify mismatches. To address this problem, we implemented a deep learning-based method that leveraged a single measurement from the target side to generate a clean perturbation [21]. This approach effectively avoided mismatches in challenging situations, although it slightly decreased the contrast of the DOT image.
We then utilized another neural net to extract features from the DOT image [29]. We combined the features from the DOT images, US images, and the DOT histogram, and employed the second stage model to generate a final prediction of the probability of malignancy. The entire workflow of the imaging section in the pipeline is shown in Fig. 6.
Fig. 6. The workflow of the imaging section of the pipeline. The top row represents the procedures of the first stage classification, including segmentation with the DEXTR model and feature extractions from the US images and DOT histograms. To predict the outcome for the first stage, two fully connected layers are utilized, positioned to the right of the middle layer. The DOT procedure to the left of the middle row includes reconstruction, image evaluation, and feature extraction from DOT images. Three sets of features are fed into the second stage model to make a final prediction of malignancy. (a) Architecture of the DEXTR model. (b) Architecture of the CNN-based evaluation model. (c) Architectures of the first- and second-stage models.
2.4.1 US information extraction
Target information, including the center depth and radius, is critical in reconstructing the DOT image, and requires a US image focused on the lesion area. To extract the target information from the co-registered US image, we implemented a semi-automated segmentation method called deep extreme cut (DEXTR), shown in Fig. 6 (a) top panel [37]. Based on the ResNet-101 model, the DEXTR model replaced the fully connected layers with atrous convolutions to keep the receptive field and incorporated a pyramid scene parsing module [38] to aggregate the global context to improve the pixel-level prediction. After four corner points on the boundary of the lesion are manually selected, a convolutional neural network (CNN) architecture-based neural network then predicts the object mask. A cropped image of the lesion area is obtained, and the target information, including the center depth and radius, is automatically read based on the mask and depth labels in the US images.
2.4.2 Image quality evaluation
Using data from the same patient as in Fig. 5, Fig. 7 shows examples of a high-quality DOT reconstruction from a matched reference breast and a distorted DOT reconstruction resulting from a mismatched reference. The reconstruction was based on regularized conjugate gradient descent (CGD) reconstruction [39], which employed an iterative optimization algorithm with a regularization term as the penalty.
Fig. 7. DOT images of matched and mismatched references from the same patient as in Fig. 3. Each image is a 3-D map, with the first slice 0.5 cm below the skin surface and the last slice 3.5 cm below the skin surface. Each slice measures 9 cm by 9 cm.
Evaluating DOT image quality requires a new criterion. Traditional image quality assessment (IQA) methods, including full-reference (FR), reduced-reference (RR), and no-reference (NR) methods, have generally yielded satisfactory results in most scenarios. However, the low resolution and absence of a ground truth in DOT reconstruction have posed challenges to using traditional IQA methods to evaluate DOT images. To overcome these limitations, we leverage the power of CNNs for visual image analysis. By optimizing filter parameters, CNNs can effectively assess image quality. Utilizing a CNN-based model can reduce pre-processing requirements and yield relatively accurate image evaluations. The CNN model enables us to assess the confidence level of an image, where a higher confidence level indicates fewer artifacts, a more focused distribution of absorption coefficients, and a satisfactory depth profile.
As shown in Fig. 6(b), the CNN employed in this work has a $7 \times 33 \times 33$ input matrix, which is the size of the reconstructed image. The model has three convolutional blocks followed by max pooling layers, and three fully connected layers map features from the convolution layers to the final prediction. Based on the quality of the input reconstruction, we manually labeled 9216 sets of simulations from the MC method and 6480 sets from the FEM. Among the 15,696 sets of simulated data, 80 percent was used for training, 10 percent was used for validating the model, and the remaining 10 percent was used for testing. During training over 20 epochs, the learning rate was set to be 0.0001 and a weight decay of 0.1 was employed.
2.4.3 US-guided DOT diagnosis
Our group developed a fusion model that combined two neural networks and used it to extract features from the DOT reconstructed images and US images [30]. By combining the advantages of the DOT and US image modalities, the fusion model achieved a more accurate diagnostic result than a single-modality approach. To optimize the diagnostic process, we implemented a two-stage method [29]. The first stage utilized the DOT perturbation histogram and US images for classification, enabling near real-time diagnosis for 72% of benign lesions without the need for reconstruction. The rest of the subjects were passed to the second stage as suspicious and requiring further diagnosis. In the second stage, the fusion model integrated features from the US images, DOT reconstructions, and DOT perturbation histograms to diagnose suspicious cases, demonstrating high clinical accuracy. In this work, to accelerate the processing time and maximize the diagnostic accuracy, we implemented the two-stage strategy after the pre-processing.
2.5 Dataset
2.5.1 US-guided DOT diagnosis
The finite element method and Monte Carlo method were adopted to generate simulated measurements. We performed FEM with well-rounded lesions, and we assumed a homogenous target and background. At least 22,216 sets of measurements were used in different scenarios. Parameters for the target and background properties are provided in Table 1. For more details about the use of simulations, please refer to Refs. [23] and [35].
Table 1. Range of parameters used in FEM and MC simulations
2.5.2 DOT patient data
Our US-guided system has been applied in clinical studies whose protocols were approved by the relevant Institutional Review Boards and were HIPPA compliant [40,41]. All patients were fully informed and signed a consent form, and all data used in this work were deidentified.
3. Results
3.1 Clinical examples and time evaluation in pre-processing
Figure 8(a) depicts an example of automated pre-processing. One target measurement and two reference measurements are filtered out by the motion detection. The MLP model is then used to sort the reference measurements by the scores from the mismatch classification. The best reference measurement undergoes outlier removal, and a target measurement is randomly selected to generate a clean perturbation for the subsequent reconstruction process.
Fig. 8. (a) Data pre-processing pipeline, including motion data detection, mismatch data identification for selecting the best reference, and outlier removal of the selected reference, generating a perturbation for DOT image reconstruction. (b) Three examples (two benign cases and one malignant case) from the automated US-guided DOT pipeline. The first example is a benign lesion, and the first stage diagnosis yielded a benign result without the need for DOT image reconstruction. The second example is a benign lesion, and the first stage diagnosis categorized it as suspicious. The case was forwarded to second stage diagnosis with the DOT image reconstruction. The second stage diagnosis has a malignant score of 0.450, under the threshold of 0.5, and the diagnosis result is benign. The third example is a malignant lesion and was classified as suspicious in the first stage and malignant (score 0.659) in the second stage.
Figure 8(b) illustrates how the pipeline processed one malignant case and two benign cases. The left column displays the clean perturbations alongside their corresponding US images. The next column shows the target masks (highlighted in red) and cropped target US images (highlighted in blue) generated from four extreme points with the DEXTR model. Subsequently, a first-stage classification model filters out the benign cases, as indicated in the first row. The remaining suspicious cases undergo DOT reconstruction and are subjected to a second-stage classification. The benign lesion in the middle row appears relatively large and is initially classified as a suspicious case, but the second-stage classification accurately identifies the lesion as benign. Conversely, the malignant case in the last row consistently yields the same result (suspicious and malignant in two stages, respectively).
In general, manually processing each patient’s DOT data will require about 15 to 30 mins to identify and exclude motion data sets, to select the best reference from the 7 to 10 reference data sets based on the perturbation and chest wall matching, and to reconstruct DOT images based on the lesion depth and size measured from the co-registered US images. In comparison, the automated pipeline can perform pre-processing and image reconstruction for each patient in approximately 45 seconds for a single image. To pre-process and reconstruct the multiple DOT images needed for averaging will take 2 to 3 mins, which is sufficiently fast for near real-time diagnosis. Table 2 provides the detailed time frame for both manual and automated US-guided DOT data pre-processing and DOT reconstruction.
Table 2. Comparison of manual processing and the proposed automated pipeline
3.2 Evaluation of motion detection and mismatch identification
Using the US-guided DOT system, we collected DOT measurements for assessment from 80 patients. For each patient, measurements with motion were labeled during data acquisition, and an experienced operator chose the best reference as the “reference standard”. The proposed pipeline method agreed with the manual labeling of motion and was more strict. To select the best reference, the automated pipeline ranked the references: in 91% of the cases, the manually selected “reference standard” was among the top five references ranked by the pipeline. In the rest of the 9% cases, the manually selected “reference standard” was among the top 10 references ranked by the pipeline.
3.3 US information reading and image quality evaluation
The semi-automated segmentation method was implemented to extract target information and obtain cropped target US images. Original and cropped US examples are shown in the middle columns of Fig. 8(b). The scale of the US image was automatically read from labels on the right of the image. Then, based on the scale, the radius of the lesion and the center depth, which are respectively half of the height of the lesion and the distance from the center of the target to the skin surface in the US image, were calculated.
The reconstruction evaluation model (Fig. 6(b)) was trained and tested on the MC simulation data. We randomly split the data into training (80%), testing (10%), and validation (10%) sets, then ran the model 10 times and calculated the average result. The area under the curve (AUC) is the probability that the classifier can correctly classify the input, and it is widely used to illustrate the performance of a binary classifier. The evaluation model reached the mean testing AUC of 0.966 (95% CI: 0.964–0.968).
3.4 Classification performance comparison
We randomly selected 20 patients’ data sets, comprising 10 benign cases and 10 malignant cases, to evaluate the classification performance of the automated pipeline and compared with the manual processing (Table 3). We employed the two-stage classification approach to compare manual and automated processing [29]. In the first stage, the two models used ultrasound (US) images and perturbations as inputs. The averaged scores were used to exclude the truly benign cases, and the remaining suspicious cases underwent a reconstruction stage. In this second stage, a model combining US images, perturbations, and reconstructed DOT images was used to obtain the final classification.
Table 3. Evaluation patient characteristics
During the first stage, applying the same threshold as in Ref. [29], we identified 50.0% of the benign cases without going through image reconstruction. The receiver operating characteristic curve (ROC) efficiently displays the diagnostic ability of the classifier. Figure 9 illustrates the ROC result of the two-stage approach, as well as the model obtained from the proposed automated pipeline process. Remarkably, the final prediction in the two-stage process achieved an AUC of 0.922 (95% CI: 0.866–0.964), which was only marginally less than the AUC of 0.943 (95% CI: 0.894– 0.982) obtained from the manual process [29].
4. Discussion and summary
This paper presents an automated patient study pipeline encompassing pre-processing, reconstruction, and breast lesion classification. A motion detection method was applied to obtain target and reference measurements with acceptable motion profile. An MLP, initially trained with simulation measurements and subsequently fine-tuned with patient data, utilized perturbation data to identify the mismatch dataset. The pre-processing pipeline was validated with a clinical dataset, significantly reducing operation time from 15-30 mins to less than 2 to 3 minutes. After the pre-processing, a semi-automated segmentation algorithm was introduced to extract the target information and obtain cropped lesion images from original US images. In an assessment of reconstruction using simulated measurements, the CNN-based evaluation model achieved acceptable results, indicating its suitability for application in patient studies. An evaluation study from 20 patients demonstrated comparable classification performance: the automated pipeline had an AUC of 0.922, while the manually processed datasets had an AUC of 0.946. The slightly lower AUC of the automated pipeline was due to the larger spreads of the DOT histograms of some benign cases, which resulted in fewer benign cases being categorized as benign in the first stage diagnosis (50%) than were thus classified by manual processing (63%). The pipeline software will be fine-tuned by incorporating more patient data to improve the outlier removal portion of the software. Nevertheless, with only a slight decrease in AUC, the presented automated pipeline accurately processed DOT data and significantly accelerated the entire reconstruction operation.
Our pipeline still has limitations. The spherical target and optically homogeneous background of the simulated measurements oversimplify the actual clinical situation, which could be more accurately represented by using targets with different shapes and adding more complex tissues and chest wall backgrounds. Furthermore, the limited availability of clinical data hampers the robustness of the proposed algorithms, particularly in the context of deep learning approaches. To address this issue, we are actively collecting more patient data through an ongoing clinical trial, aiming to improve the performance of our algorithms.
Finally, for US-guided DOT, the contralateral reference breast is critical to obtaining a matched reference for DOT perturbation calculation and subsequent DOT imaging reconstruction. However, certain rare conditions can prevent us from obtaining reference data, for example, patients may have only one breast due to prior surgery of the other breast or, if they have two breasts, the breast contralateral to the index breast lesion may have a disease. Even though such cases are rare, we have excluded these patients during the patient screening process. Moreover, it is difficult for phantoms to match the optical properties of each individual breast tissue region well. Additionally, the chest wall position of a phantom does not closely match that of the lesion breast. We have developed a deep-learning approach which leveraged a single measurement from the target side to generate a clean perturbation [21]. This approach effectively avoided mismatches, however, the contrast of DOT images had reduced. Further studies will improve this method.
Funding
National Cancer Institute (R01CA228047).
Acknowledgments
The authors acknowledge the funding support from U.S. National Cancer Institute (R01CA228047). We thank Professor James Ballard for reviewing and editing the manuscript.
Disclosures
The authors declare that there are no conflicts of interest related to this article.
Data availability
Associated code is uploaded to GitHub [42]. Data is available from the corresponding author upon reasonable request.
References
1. D. R. Leff, O. J. Warren, L. C. Enfield, et al., “Diffuse optical imaging of the healthy and diseased breast: A systematic review,” Breast Cancer Res. Treat 108(1), 9–22 (2008). [CrossRef]
2. M. L. Flexman, H. K. Kim, J. E. Gunther, et al., “Optical biomarkers for breast cancer derived from dynamic diffuse optical tomography,” J. Biomed. Opt. 18(9), 096012 (2013). [CrossRef]
3. J. M. Cochran, D. R. Busch, L. Lin, et al., “Hybrid time-domain and continuous-wave diffuse optical tomography instrument with concurrent, clinical magnetic resonance imaging for breast cancer imaging,” J. Biomed. Opt. 24(05), 1 (2019). [CrossRef]
4. G. di Sciacca, L. di Sieno, A. Farina, et al., “Enhanced diffuse optical tomographic reconstruction using concurrent ultrasound information,” Phil. Trans. R. Soc. A. 379(2204), 20200195 (2021). [CrossRef]
5. B. J. Tromberg, B. W. Pogue, K. D. Paulsen, et al., “Assessing the future of diffuse optical imaging technologies for breast cancer management,” Med. Phys. 35(6Part1), 2443–2451 (2008). [CrossRef]
6. Q. Fang, J. Selb, S. A. Carp, et al., “Combined Optical and X-ray Tomosynthesis Breast Imaging,” Radiology 258(1), 89–97 (2011). [CrossRef]
7. V. Ntziachristos, A. G. Yodh, M. Schnall, et al., “Concurrent MRI and diffuse optical tomography of breast after indocyanine green enhancement,” Proc. Natl. Acad. Sci. 97(6), 2767–2772 (2000). [CrossRef]
8. Q. Zhu, “Combined ultrasound and near infrared diffused light imaging system,” U.S. patent 6,264,610 (24 July 2001).
9. Q. Zhu, P. U. Hegde, A. Ricci, et al., “Early-Stage Invasive Breast Cancers: Potential Role of Optical Tomography with US Localization in Assisting Diagnosis,” Radiology 256(2), 367–378 (2010). [CrossRef]
10. B. Deng, D. H. Brooks, D. A. Boas, et al., “Characterization of structural-prior guided optical tomography using realistic breast models derived from dual-energy x-ray mammography,” Biomed. Opt. Express 6(7), 2366 (2015). [CrossRef]
11. H. Ding and S. Molloi, “Quantification of breast density with spectral mammography based on a scanned multi-slit photon-counting detector: a feasibility study,” Phys. Med. Biol. 57(15), 4719–4738 (2012). [CrossRef]
12. B. B. Zimmermann, B. Deng, B. Singh, et al., “Multimodal breast cancer imaging using coregistered dynamic diffuse optical tomography and digital breast tomosynthesis,” J. Biomed. Opt. 22(4), 046008 (2017). [CrossRef]
13. E. Y. Chae, H. H. Kim, S. Sabir, et al., “Development of digital breast tomosynthesis and diffuse optical tomography fusion imaging for breast cancer detection,” Sci. Rep. 10(1), 13127 (2020). [CrossRef]
14. S. Yun, Y. Kim, H. Kim, et al., “Three-compartment-breast (3CB) prior-guided diffuse optical tomography based on dual-energy digital breast tomosynthesis (DBT),” Biomed. Opt. Express 12(8), 4837 (2021). [CrossRef]
15. J. Feng, W. Zhang, Z. Li, et al., “Deep-learning based image reconstruction for MRI-guided near-infrared spectral tomography,” Optica 9(3), 264 (2022). [CrossRef]
16. Q. Zhu, P. A. DeFusco, A. Ricci, et al., “Breast Cancer: Assessing Response to Neoadjuvant Chemotherapy by Using US-guided Near-Infrared Tomography,” Radiology 266(2), 433–442 (2013). [CrossRef]
17. H. Vavadi, A. Mostafa, F. Zhou, et al., “Compact ultrasound-guided diffuse optical tomography system for breast cancer imaging,” J. Biomed. Opt. 24(02), 1 (2018). [CrossRef]
18. S. Xu, K. M. Shihab Uddin, and Q. Zhu, “Improving DOT reconstruction with a Born iterative method and US-guided sparse regularization,” Biomed. Opt. Express 10(5), 2528 (2019). [CrossRef]
19. M. Zhang, K. M. S. Uddin, S. Li, et al., “Target depth-regularized reconstruction in diffuse optical tomography using ultrasound segmentation as prior information,” Biomed. Opt. Express 11(6), 3331 (2020). [CrossRef]
20. K. M. S. Uddin, M. Zhang, M. Anastasio, et al., “Optimal breast cancer diagnostic strategy using combined ultrasound and diffuse optical tomography,” Biomed. Opt. Express 11(5), 2722 (2020). [CrossRef]
21. S. Li, M. Zhang, M. Xue, et al., “Difference imaging from single measurements in diffuse optical tomography: a deep learning approach,” J. Biomed. Opt. 27(08), 086003 (2022). [CrossRef]
22. M. Zhang, S. Li, Y. Zou, et al., “Deep learning-based method to accurately estimate breast tissue optical properties in the presence of the chest wall,” J. Biomed. Opt. 26(10), 106004 (2021). [CrossRef]
23. Y. Zou, Y. Zeng, S. Li, et al., “Machine learning model with physical constraints for diffuse optical tomography,” Biomed. Opt. Express 12(9), 5720 (2021). [CrossRef]
24. J. Feng, Q. Sun, Z. Li, et al., “Back-propagation neural network-based reconstruction algorithm for diffuse optical tomography,” J. Biomed. Opt. 24(05), 1 (2018). [CrossRef]
25. G. Hamarneh, B. Cardoen, M. Shokoufi, et al., “Multitask deep learning reconstruction and localization of lesions in limited angle diffuse optical tomography,” IEEE Trans. Med. Imaging 41(3), 515–530 (2022). [CrossRef]
26. S. Sabir, S. Cho, Y. Kim, et al., “Convolutional neural network-based approach to estimate bulk optical properties in diffuse optical tomography,” Appl. Opt. 59(5), 1461 (2020). [CrossRef]
27. G. di Sciacca, G. Maffeis, A. Farina, et al., “Evaluation of a pipeline for simulation, reconstruction, and classification in ultrasound-aided diffuse optical tomography of breast tumors,” J. Biomed. Opt. 27(03), 036003 (2022). [CrossRef]
28. Q. Zhu, C. Xu, P. Guo, et al., “Optimal Probing of Optical Contrast of Breast Lesions of Different Size Located at Different Depths by US Localization,” Technol. Cancer Res. Treat 5(4), 365–380 (2006). [CrossRef]
29. M. Zhang, S. Li, M. Xue, et al., “Two-stage classification strategy for breast cancer diagnosis using ultrasound-guided diffuse optical tomography and deep learning,” J. Biomed. Opt. 28(08), 086002 (2023). [CrossRef]
30. M. Zhang, M. Xue, S. Li, et al., “A Fusion Deep Learning Approach Combining Diffuse Optical Tomography and Ultrasound for Improving Breast Cancer Classification,” Biomed. Opt. Express 14(4), 1636–1646 (2023). [CrossRef]
31. T. Durduran, R. Choe, W. B. Baker, et al., “Diffuse optics for tissue monitoring and tomography,” Rep. Prog. Phys. 73(7), 076701 (2010). [CrossRef]
32. H. Vavadi and Q. Zhu, “Automated data selection method to improve robustness of diffuse optical tomography for breast cancer imaging,” Biomed. Opt. Express 7(10), 4007 (2016). [CrossRef]
33. G. Cybenkot, Mathematics of Control, Signals, and Systems Approximation by Superpositions of a Sigmoidal Function* (1989), 2.
34. M. M. Breunig, H.-P. Kriegel, R. T. Ng, et al., LOF: Identifying Density-Based Local Outliers (n.d.).
35. S. Li, M. Zhang, and Q. Zhu, “Ultrasound segmentation-guided edge artifact reduction in diffuse optical tomography using connected component analysis,” Biomed. Opt. Express 12(8), 5320 (2021). [CrossRef]
36. K. M. S. Uddin, A. Mostafa, M. Anastasio, et al., “Two step imaging reconstruction using truncated pseudoinverse as a preliminary estimate in ultrasound guided diffuse optical tomography,” Biomed. Opt. Express 8(12), 5437 (2017). [CrossRef]
37. K.-K. Maninis, S. Caelles, J. Pont-Tuset, et al., Deep Extreme Cut: From Extreme Points to Object Segmentation (n.d.).
38. H. Zhao, J. Shi, X. Qi, et al., “Pyramid Scene Parsing Network,” (2016).
39. B. Tavakoli and Q. Zhu, “Two-step reconstruction method using global optimization and conjugate gradient for ultrasound-guided diffuse optical tomography,” J. Biomed. Opt. 18(1), 016006 (2013). [CrossRef]
40. Q. Zhu, A. Ricci, P. Hegde, et al., “Assessment of Functional Differences in Malignant and Benign Breast Lesions and Improvement of Diagnostic Accuracy by Using US-guided Diffuse Optical Tomography in Conjunction with Conventional US,” Radiology 280(2), 387–397 (2016). [CrossRef]
41. Q. Zhu, S. Tannenbaum, S. H. Kurtzman, et al., “Identifying an early treatment window for predicting breast cancer response to neoadjuvant chemotherapy using immunohistopathology and hemoglobin parameters,” Breast Cancer Res. 20(1), 56 (2018). [CrossRef]
42. M. Zhu, M. Zhang, S. Li, et al., “Automated pipeline for breast cancer: code,” Github, 2018, https://github.com/OpticalUltrasoundImaging/DOT_GUI
