1. Introduction

Color reconstruction of multi- and hyperspectral images (MSI/HSI) is, without a doubt, a mature field of research. As could be expected, given the vast amount of information contained in HSI data, translating this inherently physical content into color pictures has always been a fundamental objective. There are several ways to achieve this, but two are particularly frequent, namely (1) generating a false-color image from an a priori calculated magnitude and (2) using Color Matching Functions (CMFs) –as defined by the International Commision on Illumination (CIE)– on carefully modified spectra so that only some pixels with particular spectral properties turn to a different color. The origins of both techniques date back to the 1970s [1], and were typically employed in the analysis of multispectral remote sensing measurements [2–9]. In these methods, color reconstructions are obtained by integrating spectra with the CMFs, with varying degrees of success in improving contrast in such representations.

Highlighting abnormal spectral variations within an image has been of particular interest in the past decade in Biomedical Optics research, and thus much work has been dedicated to translating these ideas to biomedical images. The main focus of these methods is to make spectral anomalies significantly more visible in color reconstructions of MSI/HSI images, obtaining an intuitive, direct visual outlier and anomaly inspection and detection framework. Examples include changing RGB channels from entropy measures of spectral information to enhance contrast in gastrointestinal HSI images [10], or extracting sublingual veins from hyperspectral images [11], as well as vein identification via blending NIR and VisNIR data [12]. However, the most significant references are the results and findings of M. Mitsui et al. [13] and N. Hashimoto et al. [14], in which abnormal variations across different wavelengths can be enhanced or turned into a different color. This procedure –highlighting or enhancing multispectral image abnormalities on different wavelengths– provides stunning visual results that implicitly relate to the physical properties of the biological material being sampled. As a consequence, these methods have been tested in ex vivo samples of breast tissue tumorectomies [15], palm vein visualization [16], multispectral images of Hematoxylin-and-Eosin (H&E) stained histological sections [17,18], as well as in vivo fluorescence-guided surgery in animal models [19]. Other methods, less related to colorimetry or biomedical optics, use the same methodology as a core component in anomaly detection for HSI remote sensing images [20], detecting bruises in apples in the Vis-NIR range [21], or for background segmentation for human detection algorithms using HSI images of the ocean [22].

In these applications, spectra are arranged in a matrix as column vectors, and Principal Component Analysis (PCA) is used to find a small set of basis vectors that can explain most of their variance, under the assumption that high-dimensional spectra actually lay in a low-dimensional manifold that can be inferred linearly. This effectively serves as a lossy compression algorithm, in which any spectrum can be specified with a few coordinates. If there are spectral features that differentiate a particular spectrum from more typical signatures, such features are exaggerated in the spectrum, producing significant contrast once the image is converted to RGB.

Unfortunately, this procedure is deeply context-dependent, as PCA exploits one well-known dimensionality reduction method from Linear Algebra, the Singular Value Decomposition (SVD) of a matrix. The SVD can be understood as an operator with provable uniqueness and, therefore, in the context of multi- and hyperspectral imaging of malignant tissue in surgical applications, its exploitation could be problematic. If the SVD provides a different result when provided different spectra in different amounts, the output singular vectors will not be the same on different images, and thus the enhancement results cannot be systematically repeatable. Such notions are mentioned in previous work [9,23], but are not further studied or analyzed. Similar problems could be faced by other methods that use context information to obtain results, such as ICA [24]. There is, thus, a requirement for a colorimetric enhancement method that only highlights desired information in a controllable and constrained fashion.

This work attempts to understand the aforementioned issues, and provide an improvement upon current techniques. First, we must describe the theoretical context and framework for hyperspectral enhancement as a spectral anomaly enhancement method, and provide proofs and examples that explain the fundamental issues shown in our experiments. Second, we propose a solution which uses a much simpler color enhancement framework, supported by a set of affinity or similarity functions that will provide the same results independently from the context of the image. We then define tunable contrast, gain and threshold parameters for color enhancement of any particular affinity function, having in mind its use in a fast-paced, timely surgical environment. Many state-of-the-art spectral optical quantification and classification algorithms could exploit the proposed method and enhance images in real time, so two applications (vein visualization and chromophore quantification) are presented. These novel solutions cannot be approached by the previously described enhancement methods, ensuring the practical applicability of the proposed methodology in a clinical setting.

2. Materials and methods

2.1 Imaging equipment, preprocessing and reflectance calculations

The imaging system used for hyperspectral image capture is a custom-built hyperspectral scanning system, with an embedded computer for storage and control [25]. A general description is left in Table 1. Its fundamental optical components are an ImSpector V10E spectrograph (Specim, Spectral Imaging Ltd., Finland) paired with a generic C-mount objective lens and CMOS (Mako G-223B NIR, Allied Vision Technologies GmbH., Germany). Hot pixels due to long exposure times were corrected via dark image substraction as well as a spatial median filter of size 2. In order to obtain reflectance images, each measurement was calibrated with a white reference image at the same distance and lighting conditions. Reflectance $R(\lambda )$, therefore, was calculated with the well-known equation

Table 1. Specifications of the custom-built imaging instrument.

View Table

2.2 Color reconstruction

Spectra will be transformed into RGB images by using well-known methods in colorimetry. More specifically, we will use CIE 1931 Standard Observer Color Matching Functions (CMFs), which describe retinal perception of color as the sum of three stimuli. Given a sampled spectrum $r\in \mathbb {R}^l$, with $l$ being the number of sampled channels and/or wavelengths, the CIE 1931 tristimulus values $(X, Y, Z)$ can be approximated by Riemannian sums as follows [27]:

2.3 Existing enhancement methods

A brief description of the algorithms that exploit PCA/SVD should be included. In particular, we will address the methods in [13] and [14,28] for multispectral enhancement. Both algorithms treat spectra in the same fashion. For the following discussion, let us arrange our $n$ spectra as column vectors in a matrix $R\in \mathbb {R}^{l\times n}$, i.e. $R = (r_1,\dots , r_n)^T$, where $r_i$ corresponds to a single spectrum, with $l$ channels (or sample wavelengths) in total. For a single spectrum $r_i$, its enhanced spectrum is calculated via

The methods differ, therefore, by choosing $W$ wisely, resulting in different enhancement images. Mitsui et al.’s method [13] has the objective of highlighting uncommon variations of spectra with respect to the average spectrum and to its $m$-rank approximation, and then exaggerating those differences selectively, depending on the wavelength we wish to enhance. In particular, $W$ is a diagonal matrix with all of its entries set to zero except for a single channel: $W=\textrm {diag}(w)$, $\{w_i\} = 0, \forall i\neq q$, and $w_q = k\in \mathbb {R^+}$. The term $k$ usually receives the name of enhancement gain. Hashimoto et al.’s method [14], on the other hand, adds color to the spectrum as a function of this same residual variation, and therefore chooses to create $W$ as a set of column vectors, $W=(w_1,\dots ,w_l)$, where every column is a vector of zeroes $\{w_i\}=0$, except for the $q$th channel, the channel whose variation we wish to highlight: $w_q = k\cdot c$, and here $c\in \mathbb {R}^l$ is a particular desired color, synthesized in the wavelengths described by the $l$ channels. Variable $k$ is still the enhancement gain, a positive real number.

2.4 Singular value decomposition of a matrix

The Singular Value Decomposition of a matrix $A\in \mathbb {R}^{m\times n}$ is the factorization

Another way of understanding the SVD of a matrix is by describing it as a quadratic optimization problem. The following definitions will be helpful in the coming sections:

Definition 1 (First singular value of a matrix as an optimization problem) The first left singular vector of matrix $A\in \mathbb {R}^{m\times n}$, $u_1$, is the solution of the following optimization problem:

This first definition simply states that the first left singular vector is the only vector with unit norm that is most ’similar’ (in terms of dot product magnitude) to all the column vectors in the matrix. In statistical terms, this is equivalent to finding the first vector that maximizes the explained variance with respect to all the columns in $A$ (or, alternatively, the vector that explains most columns of covariance matrix $A^T A$). In other words, $u_1$ is the vector that best explains the columns in $A$ with the smallest possible error. The error in this approximation is $\left \Vert A-\tilde {A}_1 \right \Vert _F = \left \Vert A-u_1\sigma _1 v_1^T\right \Vert _F = \sigma _2$, i.e. the next singular value.

Definition 2 (Singular values of a matrix as an optimization problem) The $i$-th left singular vector of matrix $A\in \mathbb {R}^{m\times n}$, $u_i, i=2,\dots ,rank(A)$, can be obtained recursively:

The proof of these two statements are left in [30]. Once described in this manner, the SVD of $A$ can be calculated. The solution, $A=U\Sigma V^T$, has a set of properties that are certainly appealing, due to the fact that the most frequent and relevant features can be seen in the first singular vectors. Unfortunately, these definitions are rather obscure which, combined with the visible features in the output of the numerical methods, may suggest assuming that a specific singular vector is associated with a specific feature, or that for every $A$ the number of singular vectors needed to explain all the variance is always the same. Both of these assumptions are easily refuted by the following theorem:

Theorem 1 (Uniqueness of the SVD of a matrix) Every matrix $A\in \mathbb {C}^{m\times n}$ has a singular value decomposition. Its singular values $\{\sigma _j\}$ are uniquely determined and, if $A$ is square and the $\sigma _j$ are distinct, the left and right singular vectors $\{u_j\}$ and $\{v_j\}$ are uniquely determined up to complex signs.

The proof of this theorem is left in the Refs. [29–31], is sufficient to reject methods that use the SVD in a case-by-case basis, and is well beyond the scope of this article. Despite this fact, it can be insightful to find some example problems and study their properties, in order to improve our understanding of the effects that the magnitude and frequency of appearance of measured spectra produce on the output of the SVD.

2.5 Understanding why the SVD is context-dependent

The following exercise shows how PCA/SVD provides different results depending on how well-balanced the input dataset is. We will work with a matrix $A$ with specific properties in their columns, and will show what is returned by the SVD by obtaining its singular vectors and values by hand. Measured spectra will be represented by column vectors in a matrix. This example will consist of a matrix with three different types of spectra repeated over the columns.

Proposition 1 Let $\alpha , \beta ,\gamma \in \mathbb {N}-\{0\}$, and let $a_1, a_2, a_3 \in \mathbb {R}^n$ be three column vectors such that $\left \Vert a_i \times a_j \right \Vert \neq 0$, $\langle a_i, a_j\rangle >0 \hspace {0.2em}, \forall i\neq j$. Let matrix $A\in \mathbb {R}^{m\times n}$ be a matrix with the following structure:

In summary, when applying the SVD to a set of spectra belonging to three similar (yet different) categories, the first singular vector (the vector that explains most spectra) will be near identical to the most frequent spectrum. The second singular vector will be similar to the most typical column in $A$, after subtracting what was explained by the first singular vector. This goes on recursively until there is no information to explain. Additionally, if vectors are proportional to one another, we will find similar undesired relationships.

A practical example of Proposition 1 is shown in Fig. 1. Each row represents a different selection of column vectors for $A$. The leftmost subfigure in each row shows the column vectors that compose a given matrix. The amount of column vectors of each type in the matrix are described in the legend. The other three plots show the first, second, and third left singular vectors, respectively, of a matrix with such composition. The blue and green plots in each subfigure represent the results obtained numerically via numpy.linalg.svd(), i.e. $\pm u_i$, due to the fact that the numerical method outputs each basis vector with sign indeterminacy. The black, dotted plot that is plotted over either $+u_i$ or $-u_i$ is the theoretical solution given by Proposition 1. For this example, we have used $a = 0.5(1+\cos (2\pi (0.1)x))$, $b = 0.5\cos (2\pi (0.35)x)$, and $c=\frac {1}{\sigma \sqrt {2\pi }} e^{-\frac {1}{2}\left (\frac {x-\mu }{\sigma }\right )^2}$ with $\mu =-4$ and $\sigma =4$ as a small Gaussian peak anomaly in the ’spectra’. For the first hypothesis (first row of the figure), $\alpha =\beta =\gamma =50$ will be used, with $a_1 = a$, $a_2 = a+b$, and $a_3 = a + b + c$. Since all three vectors cannot be expressed as a sum of the other two, $\textrm {rank}(A)=3$. As expected, $u_1$ is more similar to $a$, then $u_2$ is more similar to $b$, and finally $u_3$ is more similar to $c$. The second row shows a different hypothesis, where $a_1=a_2=a$ and $a_3 =a + c$. Here, $\textrm {rank}(A)=2$, and $u_1$ is mostly similar (dot product-wise) to $a$, while $u_2$ tends to look like $c$. The third left singular vector, both in the theoretical and numerical case, just represents quantization noise due to using double floating point precision, not representing anything significant (as expected).

 

Fig. 1. Testing the proposition empirically with some sample spectra. Top row: case with $\alpha =\beta =\gamma =50$, $a_1 = a$, $a_2 = a+b$ and $a_3 = a+b+c$; from left to right: example ’spectra’ used to generate the matrix and number of column vectors per class, and first, second and third singular vectors, respectively, obtained numerically (in color) and theoretically (in black dotted line). Bottom row: case with $\alpha =\beta =\gamma =50$, $a_1=a_2=a$, $a_3=a+c$. Curves $a, b$, and $c$ are defined in Section 2.5.

Download Full Size | PDF

In the context of image enhancement, we must assume some value of $m$ in Eq. (6). Information given by the first $m$ singular values will be discarded, and the variations among spectra beyond those vectors will be amplified. The values showcased by the first $m$ singular values will be representative of the most frequent, most common spectra, and this result will certainly vary across hyperspectral images. The differences we will enhance, therefore, may be obscured partially or completely, depending on the image we are working with:

2.6 Affinity-based color enhancement (ACE)

If we rearrange a hyperspectral image as a matrix $A$ with its columns corresponding to hyperspectral pixels, the result obtained through the SVD is certainly dependent of the types of spectra that are present in the image. When working in a surgical environment, with constant artifacts and objects appearing and disappearing from the field of view, any enhancement method that exploits the SVD of data has the risk of either compromising image contrast, or producing unwanted enhancement results, depending on the types of spectra present in the image.

To overcome this problem, we must modify the response of the CIE CMF functions in a consistent and systematic way. The following method can produce the same results shown in state-of-the-art work, by separating diagnosis-related calculations from color enhancement. The only restriction is that the analytical calculation must highlight the same type of spectral responses, regardless of what is present in the image.

The proposed novel framework is summarized in Fig. 2. From now on, we will refer to references as vectors that resemble a specific spectral signature of interest, $w\in \mathbb {R}^{l}$. From a practical standpoint, for example, we could study oxy- and deoxyhaemoglobin, fat, or water extinction coefficients, sampled at $l$ different wavelengths. Secondly, we will use the name spectral affinity or spectral similarity for any mapping that compares two vectors of the same type ($a: \mathbb {R}^l \times \mathbb {R}^l \rightarrow \mathbb {R}$), or any mapping that provides some type of measure as a function of the type of spectrum and some internal parameters ($a: (\mathbb {R}^l ,\theta ) \rightarrow \mathbb {R}$). When an affinity function outputs high values when comparing two spectra, we will consider those spectra as similar.

 

Fig. 2. Proposal for the separation of affinity from color enhancement.

Download Full Size | PDF

A method that exploits this separation between affinity functions and spectral enhancement will be in effect a type of affinity-based color enhancement algorithm (ACE). An ACE method combines a case-specific affinity metric, with a fully-customizable color enhancement method. This adds flexibility as well as reliability to the procedure, since there may be a multitude of affinity metrics that are not context dependent, which can now be used. For the following examples, we will consider that there exists an affinity function that generates an affinity value $a_i = a(r_i)$ for every spectrum, thus providing an affinity map $\{a_i\}$. For this manuscript, we propose the following spectral enhancement function:

2.7 Affinity metrics

The final enhancement operation should be performed with an appropriate affinity map containing the diagnostic and/or quantification information we wish to highlight. The following methods provide a bounded scalar result obtained from different types of input vector data. These methods are well known and already used in various specific classification tasks, and could be used to provide a useful affinity map for the ACE method.

Spectral Angle Mapper (SAM) Given a reference spectrum $w\in \mathbb {R}^n$, the affinity of any spectrum $r_i$ with $w$ can be calculated by the dot product $a_i = w r_i^T$. Since $w r_i^T$ is the cosine of the angle between these two vectors, this function is bounded ($a_i \in [0, 1]$) [32].

Nonparametric kernel density estimation (KDE). Kernel density estimation attempts to infer the probability density function (PDF) of an arbitrary multivariate distribution from a series of sample vectors $w_1, \dots , w_N \in \mathbb {R}^n$, which act as a Ref. [33].

Neural networks This will be our preferred method, since it is rather flexible when compared to other procedures. A typical neural network with a sigmoid output layer is typically the best for extracting a smooth probability map. Instead of working with binary classification, the response of each output neuron can be used to encode color at various wavelengths and intensities.

3. Results and discussion

The following experiments have been carried out to show the sensitivity of the SVD to changes in the input image, and its effect on typical color enhancement, as well as to showcase the enhancement capabilities of the ACE method. Once we show the context-dependent variability of using PCA/SVD for analysis, we provide some example uses of ACE that, unfortunately, cannot be implemented with merely enhancing the differences that remain past the first singular vectors.

3.1 Susceptibility of current methods to changes in the environment

To compare the performance of current methodology against ACE, we have chosen to replicate the results in vein enhancement that exploited PCA [14,16,28], and then force the method to fail by changing the background. Given the considerations in Section 2.5, we can hypothesize that, by changing the background color of an HSI image of a hand, spectral enhancement will be worsened when the background is most similar to the diffuse spectral response of veins, and amplified when it is as different to vein diffuse spectra as possible.

Such susceptibility is exactly what was found in the experiments. Figure 3 shows the behavior of current enhancement methods for vein visualization and enhancement as a function of $m$, i.e. the number of left singular vectors that are ignored in the enhancement. This result was obtained with $K=30$ and enhancing in the $600-620$ nanometer wavelength range. As expected, the value of $m$ that enhances the veins varies depending on the background color. Given a black background (first row), the first enhanced features are the shadows within the fingers; then, veins are enhanced as blue. Deep veins will appear blue due to the strong scattering properties of skin, and thus this feature is enhanced, as it is the least frequent of the four present (skin, background, shadows, and veins). With a blue background, blue spectra is not considered as rare, and is explained by the first left singular vectors, hindering vein contrast as $m$ increases. Furthermore, areas that typically present more oxygenated hemoglobin content, such as the fingertips, are enhanced instead. Finally, given a red background, bluish spectra is the least frequent type of signature, and thus anything with properties akin to blue spectra will be enhanced, also losing contrast.

 

Fig. 3. Hyperspectral enhancement methods that use PCA/SVD can be affected by changes in background spectra. Each row represents an increase in $m$, the amount of left singular vectors ignored for enhancement, for a hyperspectral image of a hand with three different colored backgrounds. In the case of a blue background, deep veins (which have a blue scattering signature) cannot be enhanced adequately.

Download Full Size | PDF

While this sort of behavior is not necessarily fundamentally flawed (this is the expected behavior of the algorithm), for in vivo imaging this implies that contrast will fluctuate as different sorts of spectra are presented to the camera in various proportions. For example, in the case of margin delineation, tumors may be accentuated if surrounded by adipose tissue, or darkened if surrounded by normal or connective tissue (adipose and normal tissue can present lower and higher scattering coefficients and average reflectance than malignant tissue, respectively) [15,34]. These inconsistencies may be problematic in a clinical setting, explaining the motivation behind this desired improvement, and providing an insight on how these techniques cannot be exploited for specific applications unless the dependency with the SVD is avoided.

3.2 Vein visualization and enhancement

Once we have observed how traditional methods are susceptible to changes unrelated to the task at hand, we must repeat the experiment with the proposed methodology. Figure 4 shows how ACE can be applied to vein visualization, showing its resilience against changes in context, background, and illumination. Note that this example is only shown for illustrative purposes but, to avoid overfitting, dropout with $p=0.1$ probability was also introduced. In Fig. 4.(a), the points from the first image used to train the model are shown. The affinity function that produces each of the presented maps is $\hat {y}_2-\hat {y}_1$, that is, subtracting the output unit value for skin from the output unit for vein spectra, for each of the spectra presented to the network. This forces the affinity map range to take values within $[-1, 1]$, which is then normalized to $[0, 1]$.

 

Fig. 4. An example of context-resilient vein enhancement using ACE. Using the 850 nm response (a) of the hand as reference, a series of background, skin, and vein spectra are collected (b). A network trained with these samples provides an affinity map (c) that can be used to enhance the original RGB reconstruction (d) and provide a vein map via ACE (e–g).

Download Full Size | PDF

The network never sees the other two images (blue and red background) during training, yet is able to highlight the same veins that could be seen at 850 nm in the image with a black background. Although the background is slightly modified due to the enhancement process, the technique does not show variability on skin and/or vein regions. For the cases shown in Fig. 4(e) through (g), the parameter values were $K=-5.0$, $k=0.75$, $b=-0.75$. Setting $K=-5$ implies reducing brightness to the veins, $k=0.75$ was chosen to avoid a very abrupt change in color, and $b=-0.75$ signifies that approximately the upper quartile of the affinity map will be enhanced. These parameters were manually tuned to get the desired result. The selected enhancement color is a positive constant for the first 100 wavelengths (equal to one), thus producing a general darkening of the vein regions, replicating the images in the near infrared while still including color information.

In order to test the flexibility of the presented method, various values of $K$, $k$, and $b$ are also shown (Fig. 5), as well as the enhancement wavelength. Being able to change not only gain $K$, but also contrast gain $k$ and thresholding value $b$, as well as the output enhancement wavelength, provides a rather simple 4-parameter method that can be modified in real time during surgery. Compared to other similar methods [11,12,16], our technique allows for vein visualization regardless of the background data, and with no motion artifacts due to capturing Vis and NIR images separately. The method can be deployed to any HSI system, no matter the configuration. Furthermore, the neural network that performs the vein detection method could be better trained to be resilient to noise, absorption and scattering-based artifacts, or skin pigmentation, without requiring any further changes in the algorithm or the imaging system.

 

Fig. 5. Testing the four ACE parameters: enhancement gain $K$ (first row), enhancement contrast $k$ (second row), enhancement threshold $b$ (third row) and enhancement wavelength color $c_e$ (bottom row). Affinity-based enhancement provides a framework that can replicate the results of state-of-the-art methods, but without depending on the SVD.

Download Full Size | PDF

3.3 Chromophore concentration estimation

Now that we have a method that is resilient to changes in the background, we can test out other applications that cannot be solved with unsupervised color enhancement. This example shows an application of ACE in the area of optical property and chromophore concentration estimation. The experiment shown in Figs. 6 and 7 consists of a series of liquid phantoms prepared with mixtures of 10% Intralipid and red ink. More specifically, 0%, 10%, 20%, 30%, 40%, 50%, 60%, and 70% ink-to-Intralipid concentration ratios were obtained. Two phantoms per concentration ratio were prepared for training the network (lowest two rows of phantoms), and a validation set of 8 phantoms was obtained as well (top row, one phantom per concentration).

 

Fig. 6. Generating an affinity metric for precise chromophore quantification. A series of liquid phantoms with known concentrations are prepared (a); the dataset is divided into training (marked with a T) and validation (marked with a V) sets (b). The network outputs a complete concentration estimation map (c) which can be quantitatively evaluated (d). Once the concentration estimates are deemed accurate enough, the metrics devised in Section 2. allow for full control of the distance to a specific concentration (in this case, 40%) by varying the affinity bandwidth (e). This metric can now be used for ACE for any specific chromophore concentration.

Download Full Size | PDF

 

Fig. 7. Hyperspectral enhancement for materials presenting a 40% red ink concentration. Affinity bandwidth is set to $h=30$. Left and center columns show the result of varying enhancement gain and enhancement contrast, respectively. The right column provides a selection of enhancement colors. For the two left columns, the baseline was $K=5.0$, $k=0.2$, $b=-0.4$, and $c_e=1$ for the first 50 of the 218 channels (380 to 450 nm), and zero for all the other channels. For the right column, $K=20.0$, and 30 channels (90 nm) are set to 1 in $c_e$; the center enhancement wavelength is specified in each image.

Download Full Size | PDF

Subplots in Fig. 6(a) and (b) show a color reconstruction of the phantoms, and the manually selected circular regions of interest (ROIs), respectively. In Fig. 6(c), the concentration estimates given by the trained neural network are presented. Validation of the estimator is shown in 6(d). Finally, (e) shows how affinity contrast can be controlled via Eq. (20). In this example, the tolerance of what can be considered a reflectance similar to 40% concentration is tuned, by varying $h$ (the affinity bandwidth). The higher $h$ is, the more stringent the affinity function becomes.

Once a functional affinity function is found, then hyperspectral enhancement can take place. Figure 7 shows the result of enhancing the 40% concentration estimates given by the affinity function. The first column shows a variation in enhancement gain $K$, and the second presents the response of the CMFs to enhancement contrast $k$. The third column shows how the output enhancement color can be selected at every specific value of the visible electromagnetic spectrum. If more than one fluorophore is utilized in guided surgery, for example, this tool may allow for the selection of the desired enhancement wavelengths and output colors, perhaps improving the existing contrast already provided by the fluorescent substances.

Diagnostic-based contrast control shows potential applicability with other optical methods, such as fluorescence-guided surgery, SFDI/SSOP or qF-SSOP, as long as the methods can provide an affinity metric that is specific to the problem at hand (e.g., tissue oxygenation, burn severity, vasculature functionality, optical properties, fluorophore concentration, and fluorophore bleaching, among others) [35,36]. This type of control can be adjusted as shown, depending on the type of surgery being performed. Ideally, with an adequate optical property estimation system, the color reconstruction could be enhanced in real time by ACE, depending on the exact information that we wished to observe.

4. Conclusion

In this work, the main concerns regarding the sensitivity to changes of current hyperspectral enhancement methods have been addressed, and a solution inspired in their original formulation is proposed. With a series of adequate affinity functions, the current state-of-the-art methods can be improved in terms of consistency and repeatability.

The method, herein termed affinity-based color enhancement, or ACE for short, separates the similarity metric from the enhancement process. Results shown in this manuscript only apply the method two particular examples, namely chromophore quantification or detection and vein enhancement, but the possible uses of ACE are only bounded by the affinity metrics that can be found for a specific problem, and by whether or not a color reconstruction is desired.

Separating analysis from representation achieves three different objectives. First, the affinity function can be chosen from a series of possible candidates, and the most optimal for the specific task can be used with no loss of applicability. Second, it enables the usage of hardware-accelerated software, providing a promising potential for real-time applications of hyperspectral enhancement techniques. Third, and final, it provides a framework that may allow clinicians and/or surgeons to control how visualization is carried out, and maybe help them improve their performance in the operating room.