1. Introduction

As a synonym of leukocytes, white blood cells (WBCs) make up 1% of the total blood volume in a healthy body and play an important role for defending the human body against infectious agents such as virus, bacteria, foreign invaders and infected or cancerous cells [1,2]. Therefore, observing the changes of WBCs in the human blood volume is a valid indicator for disease diagnosis. With the rapid development of microscope imaging techniques [3–6], there are many WBC images captured from peripheral blood smear being obtained [2,7–11], which can be directly used for WBC detection and classification. Since all WBCs are over 5 micrometers in diameter, of which the size is large enough to be seen using a typical optical microscope (compound microscope). Stained with Leishman’s stain, different types of leukocytes can be easily identified, and it is also easy to count them. There are various cellular entities in human blood, which are responsible for different functions of body such as carrying oxygen, regeneration, clotting and immunity. Blood mainly consists of four components, including plasma, red blood cells (RBC), platelets, and WBC. Of the four components, WBCs play important role in protecting the body from infection through fighting with foreign pathogens [1,12–14]. Therefore, WBC classification is important and beneficial for clinical medical disease detection. WBCs are composed of granulocytes (neutrophils, eosinophils, and basophils) and non-granulocytes (lymphocytes and monocytes), which can be categorized into five types, i.e., Monocyte, Eosinophil, Basophil, Lymphocytes and Neutrophil [7,15] (As shown in Fig. 1). All of these types of WBC have a certain function for human body in the fight against kinds of infections such as virus, bacteria and fungus. In a normal human body, the ratio of RBC and WBC is 600:1, which makes the identification of WBC type in a blood smear sample a difficult task. When some diseases occur, it will cause changes in the number and shape of WBCs. For example, when the human body is invaded by influenza virus, in order to resist the damage of the virus to the human body, the number of white blood cells will increase at this time. In addition, if the human body develops malignant tumors and other diseases, not only the number but also the proportion of different types of WBCs will change obviously. Furthermore, WBCs will also deform significantly. Therefore, when doctors need to check whether a patient suffers from a certain disease, the most intuitive way is to detect whether WBCs in the human body have undergone quantitative changes or lesions [16,17]. In clinical practice, WBCs are generally identified by manual microscopy [18–20], which is time-consuming and labour-intensive. In addition, human errors are inevitable during the identifying process due to some human subjective factors. As a result, it is necessary to develop intelligent WBC classification methods based on advanced artificial intelligence techniques. During the past decades, machine learning-based methods are widely used to reduce the workload of human beings for WBC classification and have earned great success. In this paper, we also focus on designing an effective method for automatically classifying different types of WBCs based on machine learning techniques.

 

Fig. 1. Five different types of white blood Cells [21].

Download Full Size | PDF

As to machine learning-based WBC classification methods, it is a requisite to collect an appropriate dataset by taking quality, variety, and size into account. In the early days, the lack of high quality dataset is the major challenge and hinders the classification accuracies of different methods [22]. With sufficient datasets, diverse machine learning techniques can be utilized for WBC classification. In general, previous machine learning based methods can be roughly classified into two categories, i.e., traditional hand-crafted feature extraction based methods and deep learning driven frameworks. In traditional methods, appropriate hand-crafted features are firstly extracted from cell images, and then, different WBC types are classified by certain one or an ensemble of several classifiers. Therefore, it is very critical to extract discriminative features for final classification, and various features have been proposed, such as hough transform [23], local binary pattern [24] and shape [25]. Different to traditional methods, deep learning based framework automatically extract features from original cell images by means of deep neural networks [26–28], which are powerful for feature representation learning. The most commonly used networks for image classification are convolutional neural networks (CNNs). In order to obtain good classification results, deep CNNs with numerous parameters are often required and plenty of data are needed to train such deep neural networks. However, unlike natural images, it is often not easy to obtain enough medical data for deep neural network training. Therefore, pre-trained networks from natural images are usually used for feature extraction or fine-tuning on small-scale medical dataset.

Although great success has been achieved in previous WBC classification methods, some practical issues such as uneven staining, boundary blur and nuclear intra-class variability are still hindering the final classification accuracy. In this work, we propose a deep neural network for WBC classification via discriminative region detection assisted feature aggregation, referred as DRFA-Net, which can accurately locate the WBC area from cell images to boost final classification performance. In detail, DRFA-Net uses an adaptive feature enhancement module to refine multi-level deep features in a bilateral manner for efficiently capturing both high-level semantic information and low-level details of cell images. In order to suppress the interference caused by background areas, we design a network branch to detect the WBC area supervised by segmented ground truth. The bilaterally refined features are finally aggregated for final classification, and the detected WBC area is utilized to highlight the features of discriminative regions by an attention mechanism. In a nutshell, we summarize the main contributions of this paper as follows:

2. Related Work

In the past few years, many intelligent approaches have been proposed for WBC classification. In order to improve the classification accuracy, digital image technology has become one of the main directions to combine WBC detection and classification into a unified task. By taking a blood smear image of cells under a microscope, cell images can be easily obtained. Then, various techniques in digital image processing community can be used to classify the WBCs. By observing the classified WBC images, professional doctors can easily find the lesions of WBCs. In such a manner, the error caused by personnel operation can be effectively reduced, and the efficiency of disease diagnosis can be well improved. Generally, previous computer aided intelligent methods can be roughly classified into two categories, i.e., traditional hand-crafted feature extraction based methods and deep learning based frameworks.

As to hand-crafted feature extraction based WBC classification methods, how to extract effective features is critical. In previous researches, many kinds of traditional methods are put-forward. Since the white blood cell density is very high in the bone area, Nipon et al. [29] proposed morphological granulometric features of nucleus for bone marrow WBC classification. Mishra et al. [30] proposed a gray level co-occurrence matrix based feature extraction method and a probabilistic PCA based feature section model, which are followed by random forest based classification techniques for acute lymphoblastic leukemia detection and classification. By extracting hybrid features such as texture, shape, color, and performing data normalization, Ahmed et al. [31] successfully used traditional classifiers including Naive Bayes, Decision Tree, support vector machine (SVM) and $k$-nearest neighbor (KNN) to classify different acute lymphoblastic leukemia from WBC images. In [32], Das et al. first extracted gray-level-run-length matrix and gray-level co-occurrence matrix-based texture features together with color and shape, then PCA and SVM are utilized for efficient feature selection and cell classification, respectively. Mishra et al. [33] performed linear discriminant analysis (LDA) on the features extracted by a newly introduced discrete orthogonal S-transform to learn more discriminative low-dimensional features and then employed the Adaboost algorithm as well as random forest for leukemia classification. Considering that the shape changes of WBCs are very complicated, Lamberti [25] proposed a kind of interpretable metrics based shape feature for WBCs classification. Although above mentioned methods can work efficiently on small-scale dataset without needing of time-consuming model training, they need to accurately segment the area of WBCs from blood smear images, of which itself is also a challenging task.

Due to the powerful ability for learning to extract hierarchical features [27,34,35], deep convolution neural networks (CNNs) have been widely used for image processing and refreshed the records of many visual tasks [27,36–42], including WBC classification [17,43–46]. Since training a deep neural network from scratch needs large-scale datasets, while it is not easy to capture medical datasets with plentiful enough training samples. To this end, pre-trained models are often used. Given a certain pre-trained model, one way is to use it for extracting features as the input of traditional classifiers such as SVM, KNN, etc. Another way is to transfer the pre-trained model to small dataset by fine-tuning. For feature extraction with pre-trained models, many previous works have also been proposed for WBC classification [47–50]. In [49], three different CNN architectures including AlexNet, GoogleNet and ResNet-50 are used for feature extraction, then the features are merged and discriminative features are selected as the input of quadratic discriminant analysis model for final WBC classification. With features extracted from pre-trained CNN model, Sahlol et al. [50] employed a statistically enhanced salp swarm algorithm and SVM for feature selection and classification, respectively. In addition, there are also many researches that focus on directly training deep neural network to classify WBCs [51–56]. Without using pre-trained model, even better classification results can be reached [52]. In [57], Jung et al. introduced a new CNN architecture (referred as W-Net) to classify WBCs. Other advanced network architectures are also used for this task, such as capsule networks [55] and long short term memory network [51]. Compared with traditional hand-crafted feature based methods, deep learning based methods often obtain better results due to the strong representation capability of deep features. However, their performance is still degenerated by some issues such as noisy background, uneven staining, boundary blur and nuclear intra-class variability.

3. Proposed DRFA-Net

In this section, we give the details of our proposed DRFA-Net, which takes a WBC image as input and outputs its corresponding type. Overall, DRFA-Net consists of three major parts: discriminative region detection module (DRDM), attention-wise feature aggregation module (AFAM) and classification module (CM). In detail, given an input WBC image, we first use previous pre-trained deep neural network model to extract hierarchical deep features for locating the WBC area. Then the detected WBC area is embedded to highlight discriminative features from deep feature maps with an attention mechanism, which can effectively suppress noisy information for boosting final classification performance. Figure 2 gives a simple architecture of the proposed DRFA-Net.

 

Fig. 2. The pipeline of our DRFA-Net.

Download Full Size | PDF

It should be noted that there are many network structures can be used for our task, such as Densenet, VGG16, etc. However, the performance of VGG16 usually cannot match Resnet structure, while training a network with Densenet structure needs more computational cost than Resnet structure. As to Resnet structure, it can be easily trained as well as obtaining good performance. Therefore, in order to take a compromise between computational cost and efficacy, we choose the ResNet structure [58,59] as the backbone for feature extraction and deploy the pre-trained ResNeXt model [60] for network initialization. As a result, there are five basic feature extraction layers: conv1, conv2_x, conv3_x, conv4_x, and conv5_x. Firstly, we use the backbone network to extract a series of hierarchical deep feature maps which encode the low-level details and high-level semantic information at different scales of an blood smear image. On the one hand, the fine details of original input image are inevitably gradually lost due to a series of spatial pooling and convolution operations, which is harmful for distinguishing the WBC boundary from background. On the other hand, the features extracted by deep layers are with sufficient high-level semantic information, which can help accurately locate WBC areas. Therefore, it is necessary to exploit the complementary information extracted from both shallow and deep layers for the detection of WBC regions.

3.1 DRDM

During the blood smear imaging process, many interference factors including dyeing impurities and illumination change could induce various challenges such as cytoplasm with low image contrast, variable appearances under different staining conditions, which makes it difficult to separate WBCs from blood smear images. Therefore, in our proposed DRFA-Net, we first design a discriminative region detection module to highlight the WBC area for assisting following classification. For WBC detection, we should consider two aspects. On one hand, high-level semantic information encoded in deep layers should be used to locate the WBC area. One the other hand, low-level details encoded in shallow layers should be used to distinguish the boundary between WBC area and background. In order to capture both deep and shallow features, we design a discriminative region detection module to detect WBC by adaptively refining multi-level deep features in a bilateral manner. Since there is little semantic information in deep features extracted from shallow layers, we design a branch for feature enhancement from deep to shallow layers by embedding multiple deep to shallow modules (DTSMs). Similarly, in order to remedy the fine details missed in the features extracted from deep layers, we design another branch for feature enhancement from shallow to deep layers by embedding a series of shallow to deep modules (STDMs).

With the pre-trained ResNeXt model, we denote the extracted feature maps as $\textbf {F}_1, \textbf {F}_2, \textbf {F}_3, \textbf {F}_4, \textbf {F}_5$ from shallow layers to the deep layers. In order to ensure that the final WBC detection map is the same as original input, we first upsample the five layers of feature maps to the size of input image. For the shallow to deep pathway, we embed a series of STDMs to enhance the details in deep layers by refining features from the shallow layers to deep ones in a hierarchical manner. For each STDM (as shown in the top part of Fig. 3), the input consists of two parts, including the output of last STDM and the features extracted from the corresponding layer. For the sake of simple presentation, we denote $\textbf {F}_L$ and $\textbf {F}_H$ as the output of last STDM and the corresponding input deep features, respectively. Then the output of STDM can be formulated as:

 

Fig. 3. The brief structure of our designed STDM (top) and DTSM (bottom).

Download Full Size | PDF

Similarly, For the deep to shallow pathway, we embed a series of DTSMs to enhance the details in deep layers by refining features from the shallow layers to deep ones. For each DTSMs (as shown in the bottom part of Fig. 3), the input also consists of two parts, including the output of last DTSM and the features extracted from the corresponding layer. We denote $\textbf {F}_H$ and $\textbf {F}_L$ as the output of last DTSM and the corresponding input deep features, respectively. Then the output of DTSM can be formulated as:

After we get the outputs from the two pathways (denoted as $\textbf {F}^{STDM}$ and $\textbf {F}^{DTSM}$, respectively), we concat them to learn a WBC score map $\textbf {S}$ which highlights the area of WBC from background as well as suppress background as much as possible. The learning process can be mathematically defined as follows:

3.2 AFAM

With the WBC score map and a series of hierarchical deep feature maps, we design a attention-wise feature aggregation module to learn discriminative features for final classification. During the feature learning process, we should consider two points. Firstly, the global context information of original blood smear image should be aggregated to capture global features. Secondly, features of the most important WBC area should be emphasised. To this end, original deep features from different layers are weighted by the WBC sore map obtained from previous DRDM, then the weighted feature maps are concatenated and aggregated by our designed AFAM. Specifically, the learned discriminative features can be obtained as follows:

Finally, the learned discriminative features $\textbf {F}_D$ are used to classify the types of original WBC images by the classification module.

3.3 Model training and testing

In this work, we use the ResNet architecture [58] as the backbone for feature extraction and implement the proposed DRFA-Net by using the PyTorch framework. The well-trained ResNeXt network on ImageNet [60] consists of 101 basic convolution layers, we use it to initialize the parameters of the feature extraction network. As a result, we have five feature extraction layers as mentioned in previous sections [58,59]. Training: Different to many previous works that only use the classification loss to train networks, we add a WBC detection loss in the proposed DRFA-Net since the manually annotated WBC regions can be used to train the network for WBC detection. Therefore, the total loss of DRFA-Net consists of two parts, i.e., classification loss $L_C$ and detection loss $L_D$.

For the output of WBC detection block, the cross-entropy loss is used for training the network to detect WBC regions. Specifically, the pixel-wise cross-entropy loss between $\textbf {S}$ and the ground-truth WBC mask $\textbf {G}$ is calculated as:

As to the classification loss, since the task is a multi-class classification problem, we use the multi-class cross-entropy to train the network for learning discriminative features, which can be formulated as follows:

Finally, the total loss of our DRFA-Net can be written as follows:

Our DRFA-Net is initialized by the well-trained ResNeXt network on ImageNet [60]. In order to learn more discriminative features for WBC classification, we fine-tune the pre-trained model on blood smear image dataset. Specificity, we train DRFA-Net on BCISC dataset [2], which consists of five types of WBCs, i.e., Monocyte, Eosinophil, Basophil, Lymphocytes and Neutrophil. For each type, there are 50+ samples with manually annotated WBC mask labelled by pathologists. The statistic details of this training dataset are summarized in Table 1. The total number of samples is 268. In addition, there are also 268 sub-images that cut from original images and these sub-images focus more on the local information of WBC. Since the number of training samples in BCISC is not enough for training a deep neural network, we perform data augmentation by randomly rotating, resizing and horizontally flipping all of the images and their corresponding ground masks, and finally, the training set is enlarged to 8,576 images. We fine-tune the newly designed network by using this enlarged dataset. The proposed DRFA-Net is trained on a machine equipped with an Intel 3.4 GHz CPU with 32 GB memory and a Nvidia Titan V GPU. The stochastic gradient descent (SGD) algorithm with a momentum of 0.9 and a weight decay of 0.0005 is used to optimize the whole network. The learning rate is adjusted by the “poly" policy with a power of 0.9. The training batch size is set to 4, and we stop the whole learning process after 5,000 iterations. The training process is completed after approximately 0.5 hours. Testing: For each input image in the testing phase, we feed it into the proposed network and obtain its corresponding predicted types. Only approximately 0.037s is needed for type prediction for an input image of size $320 \times 320$ pixels by using a single Nvidia Titan Xp GPU, which is very efficient.

Table 1. Statistic details of the training dataset BCISC

View Table | View all tables in this article

4. Experiments

4.1 Datasets

In our experiments, three different datasets are used for classification test, including Raabin-WBC [10], LISC [7], and BCCD [8].

Raabin-WBC is a large-scale dataset which was released in 2021. There are three sets of cropped sub-images for classifcation, i.e., Train, Test-A, and Test-B. As to all WBCs in Train and Test-A sets, two experts have separately labeled the types of each image. Yet, images in Test-B have not been labeled thoroughly. As a result, we only used Train and Test-A sets in this study. The two sets were collected from 56 normal peripheral blood smears (for lymphocyte, monocyte, neutrophil, and eosinophil) and one chronic myeloid leukemia (CML) case (for basophil) and there are totally 14,514 WBC images. Giemsa technique was used to capture all the films, and the normal peripheral blood smears have been taken using the camera phone of Samsung Galaxy S5 and the microscope of Olympus CX18. As to the CML slide, it has been imaged utilizing an LG G3 camera phone along with a microscope of Zeiss brand. In addition, all of the images were taken with a magnifcation of 100.

LISC contains 257 WBCs from peripheral blood, which have been labeled by only one expert. The images in this dataset were acquired from peripheral blood smear and stained through Gismo-right technique, and they were taken at a magnification of 100 by using a light microscope (Microscope-Axioskope 40) and a digital camera (Sony Model No. SSCDC50AP).

BCCD was taken from the peripheral blood and there are totally 349 WBCs labeled by one expert. The Gismo-right technique has been employed for staining the blood smears. The images in this dataset were also captured at a magnification of 100 by using a regular light microscope together with a CCD color camera. In addition, similar to [61], we correct the label of one image which has been incorrectly labeled.

Since the WBC detection module of the proposed DRFA-Net needs the segmented WBC mask for training, while there is no manually annotated WBC mask in Raabin-WBC and BCCD datasets, we fix the WBC detection module and just fine-tune the classification parameters for the three testing datasets. Then, the fine-tuned model will be used for final classification on the test subsets of different datasets.

4.2 Evaluation metrics

In order to evaluate the classification performance of our proposed network, we use four metrics including Precision, Sensitivity, F1-score (F1), and Accuracy (Acc) in our experiments. With following definition:

4.3 Experimental results

4.3.1 Comparison with other state-of-the-art methods

In order to verify the superiority of our proposed network, we compare it with some previous WBC classification methods, which include both deep learning based methods and traditional hand-crafted feature based methods. The details of the compared methods are as follows:

Table 2. Classification performance comparison of our proposed method with other competitors in terms of four metrics

View Table | View all tables in this article

4.3.2 Ablation analysis

As shown the Fig. 2, there are three major elements designed in our proposed DRFA-Net, i.e., WBC detection module, DTSM and STDM. In order to validate the efficacy of different modules, we remove each module individually from the whole network and obtain final classification performance. Firstly, we remove the WBC detection module and append the enhanced features from the two feature enhancing pathways, i.e., $\textbf {F}^{STDM}$ and $\textbf {F}^{DTSM}$, to the concatenated up-sampled deep features extracted from original five feature extraction layers. Therefore, we have another version of DRFA-Net without the attention mechanism, referred as DRFA-Net_noAtt. Secondly, we remove DTSM and STDM from DRFA-Net, respectively and obtain another two versions of DRFA-Net, i.e., DRFA-Net_noDTSM and DRFA-Net_noSTDM. In Table 3, we report the classification results on different datasets obtained by different versions of DRFA-Net. As can be seen, our designed different modules are important and can effectively improve the final classification accuracy. In addition, in order to give an intuitive validation of the WBC detection module, we show some WBC detection results in Fig. 4. As can be seen, the proposed WBC detection module can accurately locate the WBC areas, which is very important for final classification. Since this work does not focus on accurate WBC segmentation, we just use the WBC detection module to locate the discriminative region for final classification. Therefore, we do not use any other post-processing technique to refine the detection results.

 

Fig. 4. An intuitive showing of the WBC region detection results of the proposed method. The top row represents original WBC images, the bottom row denotes the WBC area detection results of the proposed network.

Download Full Size | PDF

 

Fig. 5. The training loss and testing accuracies of different datasets with varying iteration times.

Download Full Size | PDF

Table 3. WBC classification results of the proposed methods on different datasets

View Table | View all tables in this article

4.3.3 Convergence property analysis

In order to intuitively show the convergence property of the proposed network, we plot the training loss and ACC values ($\times 100$) obtained with varying iteration times in Fig. 5. As can be seen, the optimization process is with fast convergence property and the testing accuracies of different datasets increase fast and keep stable when the whole training process achieves convergence.

5. Conclusion

In this paper, we introduce a deep neural network for white blood cell classification via discriminative region detection assisted feature aggregation. The proposed network can accurately locate the WBC area for boosting final classification performance. An adaptive feature enhancement module is designed to refine multi-level deep features in a bilateral manner for efficiently capturing both high-level semantic information and low-level details of WBC images. Since background areas could inevitably produce interference in the WBC images, we design a network branch to detect the WBC area with the supervision of segmented ground truth, then the detected WBC area is utilized to highlight the features of discriminative regions by an attention mechanism. Experiments on three public datasets are conducted to validate the efficacy of the proposed method in terms of different classification metrics.