1. INTRODUCTION

Imaging allows us to understand the world in which we live. Digital imaging has traditionally been based on a detector array that can detect variations in light intensities with spatial resolution. Recently, however, there have been rapid developments of a technology capable of imaging without a detector array, using only a single pixel. This so-called “single-pixel imaging” [1] allows imaging in which a reduction in camera complexity is gained at the cost of increased computational time. While it is true that in the visible region of the spectrum the development of detector arrays has reached the stage where this trade-off is not beneficial, it is in other regions of the electromagnetic spectrum where this technology can provide huge benefits.

In this work we present an application of single-pixel imaging to the field of microscopy [2]. Microscopy is typically performed in the visible region of the spectrum where camera technology is extremely well developed. Microscopy in other regions of the spectrum, such as infrared or ultraviolet, traditionally requires expensive cameras. Single-pixel imaging is, therefore, an attractive prospect as it allows the development of imaging systems that can be significantly cheaper than current camera technology. This may allow imaging at previously undeveloped wavelengths, increasing the range of spectral imaging [3].

Single-pixel imaging has previsouly been applied to a microscope, e.g., Wu et al. [4] and Studer et al. [5]. In [5] they image hyperspectrally across the visible spectrum. Here we present a prototype single-pixel microscope that images not only in the traditional visible spectrum but also simultaneously in the short-wave infrared (SWIR). Previous work in single-pixel imaging has utilized compressive sensing [6–9], which minimizes the number of measurements required to produce an image. However, these compressive approaches result in long image reconstruction times of minutes to hours and, hence, low frame rates. To better suit the operational needs of a microscope, here we concentrate on methods to produce images at video rates.

As an alternative to a detector array, single-pixel imaging shifts the spatial information away from the detector and onto a set of masks. If the image is masked such that some of the light is blocked and some passes to create a signal on a single detector, then this signal contains information about how similar the mask is to the image. If many different masks are used, their shapes and corresponding signals can be combined to infer a reconstruction of the image. An intuitive set of masks would scan a single bright pixel to build up the image as in confocal microscopy [10]. However, for systems in which the source is not also scanned, these masks waste most of the light, resulting in very poor signal-to-noise ratios (SNRs). An ideal set of masks would therefore have an equal amount of bright and dark pixels.

A commonly used set of such binary masks is the Hadamard set [11]. Each mask in this set is orthogonal to every other mask, which, in the absence of noise, allows perfect reconstruction of the image when the number of masks equals the number of pixels. This number is large even for images with a moderate resolution (4096 measurements required for a resolution of $64\times 64$ pixels), which has lead to the development of compressive techniques.

Compressive imaging methods allow images to be reconstructed using a number of masks that is only a fraction of the number of pixels. These methods are usually reliant on iterative algorithms to solve optimization problems that are computationally difficult. This can result in long reconstruction times measured in minutes or hours. Recent developments make progress toward compressive video reconstruction [12–18], with emphasis on developing methods to reduce computational time [19,20]; however, in this paper we concentrate on compressive techniques that do not require computationally expensive reconstruction and are therefore able to provide near video frame rates.

Our microscope design is shown in Fig. 1(b) and is based on a modified Zeiss Axiovert 200. The halogen light source has been removed and the illumination is provided by a white-light LED in addition to a SWIR source comprising a superluminescent diode emitting at 1550 nm (Thorlabs S5FC1005S). The objective lens is a reflecting objective with a broadband aluminium coating, providing an image plane with $36\times$ magnification at 0.52 NA (Newport 50102-01) . Two parabolic mirrors then re-image this plane onto a digital mirror device (DMD). This optical system provides a field of view of 381 μm, giving physical resolution of 12, 6, and 3 μm for mask sizes of $32\times 32$, $64\times 64$, and $128\times 128$, respectively.

 

Fig. 1. Experimental setup and dual band images. (a) Sketch of the DMD operation. The input beam can be diverted, pixel by pixel, into one of two output arms. (b) Experimental setup. The diffuser (aperture) in the SWIR (visible) illumination arm can be removed (closed) to switch to a dark field configuration. (c) Reconstructed images from the visible and SWIR detectors of a silicon CMOS chip edge. The resolution is $128\times 128$ and the images are the result of five averages and have been upscaled to $256\times 256$ using linear interpolation.

Download Full Size | PDF

The DMD is a Vialux ALP 4.2 model with $1024\times 768$ pixels, a binary pattern switching rate of 22 kHz, and onboard storage for up to 45,000 patterns. As illustrated in Fig. 1(a), the micromirrors can be orientated either $+24$ or $-24\mathrm{deg}$ with respect to the normal incidence input beam. Two pick-off mirrors are positioned to steer these beams through a collection lens and into a single-pixel detector. One arm has an amplified photodiode sensitive in the visible (Thorlabs PDA100A) and the other has a large area Ge detector (Thorlabs FDG1010) amplified by a Thorlabs PDA 200C.

The whole optical system is composed of a mixture of lenses and curved mirrors. To provide maximum imaging performance over as large a wavelength range as possible, all imaging optics prior to the DMD are reflecting, specifically to avoid any chromatic aberrations. Imaging is not required after the DMD and, therefore, lenses are used to collect light into the single pixel detectors. In addition to this collection lens, a field lens is placed as close to the DMD plane as possible in order to reduce the size of the beam in the collection arms.

2. RESULTS

A. Optical Modes

To illustrate the dual, visible/SWIR, waveband capability provided by the microscope, we image a silicon CMOS chip. The visible and infrared images are shown in Fig. 1(c). Since silicon is opaque to visible light, the edge of the chip appears as a sharp boundary in the visible image. The infrared light, on the other hand, is partially transmitted by silicon and this image shows additional features. The example in Fig. 1(c) shows that there is a band of silicon near the edge of the chip, with metal on either side that appears opaque (black). We note that the single-pixel approach has the advantage of perfect spatial registration between the two images since the spatial information is dictated by the DMD.

The flexibility of the microscope is further extended with the addition of a “dark-field” imaging configuration for both SWIR and visible. Examples of the images obtained for bright- and dark-field modes can be seen in Fig. 2. The sample comprises an amalgamation of 200 nm sized gold beads and, by comparing the bright-field images, it is apparent that the beads are almost transparent at the SWIR wavelength. If we switch to dark-field imaging, however, the results in the visible and IR are similar. The bright field measures the absorption of the sample, while the dark field measures scattering.

 

Fig. 2. Optical modes. Comparison of bright- and dark-field operating modes. The visible and SWIR images were acquired simultaneously, but the sets of bright- and dark-field images were from separate acquisitions. The images are the result of five averages and have been upscaled to $256\times 256$ using linear interpolation.

Download Full Size | PDF

B. Image Reconstruction Modes

The nature of single-pixel imaging enforces a reciprocal relationship between frame rate and image resolution, and we have therefore designed several different modes of operation to give the widest possible range of utility.

The most basic operating mode allows the user to adjust the resolution and corresponding frame rate to suit their needs. The reconstructions are formed from a full set of masks in order to give the best possible reconstruction. The quality of the reconstruction is dependent on the SNR of the system. This can be maximized by taking a differential measurement between each mask and its inverse, effectively throwing away noise below the pattern display rate. The resulting SNR of the reconstructed images is of order 115 and 95 (taken from images in the right hand column of Fig. 5) for the visible and SWIR arms, respectively.

For a given resolution, the frame rate is simply a function of the number of masks and the mask display rate. The resulting frame rate is then $R_{\mathrm{Patt}}/2N$ where $R_{\mathrm{Patt}}$ is the pattern display rate and $N$ is the number of a pixels in the image. For our particular DMD, the pattern display rate of 22 kHz results in a frame rate of 10 Hz at $32\times 32$ and 0.6 Hz at $128\times 128$.

We expand upon this “standard” operational mode to provide a user-friendly “adaptive” mode that provides the benefits of both high frame rate and high resolution. Typical use of a microscope involves two distinct stages, an initial stage in which the user searches around the sample for a particular feature of interest, and a subsequent stage in which they wish to image the feature with as high a resolution as possible.

This adaptive mode constantly compares the current signal from the single-pixel with the equivalent signal from the previous frame in order to determine if the sample is moving. If the difference between current and previous signals exceeds some threshold, then the sample is judged to be moving and the frame rate remains high. Otherwise, after a fixed number of frames the program increases the resolution from the base $32\times 32$ to $64\times 64$ and then to $128\times 128$, so long as the sample is still not judged to be moving. Any movement in the sample resets this process and returns to the high-frame-rate mode.

An example image sequence for the adaptive mode is shown in Fig 3. It shows an image of a patterned contact on a CMOS chip that, in the first image, is moving. The program compares the signals from one frame to the next, detects movement, and, as a result, maintains a high frame rate with low resolution. After a few seconds, the contact is stationary and having detected no movement, the program doubles the resolution. After 16 s there is still no movement so the resolution is doubled again to the maximum value of $128\times 128$ pixels. The image is still stationary, so the program now begins to average the data, improving the signal to noise of the image further. After 44 s, the sample begins to move again; the program detects this and switches back into high-frame-rate mode.

 

Fig. 3. Transmission images of a silicon CMOS chip captured in the SWIR band. The picture series shows a typical timeline when using sharpen mode. The resolution and averaging is controlled automatically based on the residuals. The line showing residuals is only for illustration and is not quantitatively accurate. The arrows indicate that the object in the scene is moving. The sample is entirely absorbing in the visible and thus no visible image is shown. All images have been upscaled to $256\times 256$ using linear interpolation.

Download Full Size | PDF

The standard and adaptive modes have both used a complete set of Hadamard masks for image reconstruction. A further approach, which we call “evolutionary mode”, can reconstruct an image from a subset of masks and is therefore compressive. However, unlike most forms of compressive imaging it is not computationally intensive and so is compatible with video frame rates.

The fundamental idea behind our evolutionary mode is outlined in Fig. 4. For the image in Fig. 4(a) a numerical prediction is made of the anticipated signal corresponding to each mask in a complete Hadamard set. The signals are then ordered in a sequence with the largest first, and the result is shown in Fig. 4(b). If we wish to re-image the same object using less than a full set of masks we should start with the masks producing the largest signal, but how many masks do we need? A good estimate can be obtained by summing the signals within a subset and comparing that with the total of all signals. As can be seen, the largest 25% of the signals accounts for 70% of the total signal. The reconstructions corresponding to various subsets are shown in Fig. 4(c) and reveal that as few as 12.5% of the masks produce a good reconstruction with only 28% relative error.

 

Fig. 4. Hadamard spectrum analysis and subset reconstruction. (a) Original image taken with a standard optical microscope. (b) Signal strength for each Hadamard pattern, ordered with largest signal first and plotted in log space for clarity. The lines are drawn at specific numbers of patterns and the number above is the percentage of total signal contained within that subset. (c) Numerical simulation reconstructions formed from subsets of patterns and signals. The subsets start with the largest signal and include ever smaller signals up to the number of patterns listed. The relative error is obtained from the absolute difference to a 4096 pattern reconstruction.

Download Full Size | PDF

However, this reduction in itself is not directly useful for imaging since one would have to know the object a priori. The evolutionary mode takes this insight and expands it. Let us consider an image with a resolution of $64\times 64$ pixels; to be compressive, we wish to use only 1024 patterns, giving a compressive ratio of 25%. To optimize the choice of masks we keep a fraction of the 1024 masks with the highest signals from the previous frame, replacing the rest with randomly chosen masks from the full set. This continued optimization allows good masks to be filtered into the set of 1024 after several frames.

The experimental realization of this method is shown in Fig. 5. The images show the visible and SWIR images of two shapes, a metal tip in the lower left and a silicon edge in the upper right. The transmission of silicon in the infrared leads to it appearing almost invisible in the SWIR channel. We see that the image quality remains high even for a low number of patterns and that this low number of patterns also results in a significant boost to the frame rate.

 

Fig. 5. Experimental results of the evolutionary mode. The images are taken simultaneously from the visible and SWIR channels, and the images are reconstructed from the number of patterns shown underneath. The images are the result of five averages and have been upscaled to $256\times 256$ using linear interpolation.

Download Full Size | PDF

The fraction of patterns to retain and the number to randomize from one frame to the next can be controlled, allowing control of the final image quality and speed of adaptability. In the example shown in Fig. 5 we use a 25% compression (1024 masks) wherein we retain 70% and randomize 30%, a ratio that we find is a good balance between image quality and speed of response. This results in the image quality shown in Fig. 5, and, if the objects in the scene move, the image returns to this quality within 2 or 3 s.

Evolutionary mode has the further advantage that it provides an increased robustness to noise. If the noise level is fixed, for instance, detector noise due to dark counts, then the SNR scales directly with signal strength. Evolutionary mode uses only the largest signals with the highest SNR and is therefore robust to this type of noise. Extreme cases with significant noise can result in the evolutionary mode reconstruction having even lower absolute error than a reconstruction from a full set.

3. CONCLUSIONS

We have demonstrated a prototype microscope system based on a single-pixel approach to image simultaneously in the visible and SWIR at frame rates approaching video rates. We demonstrate imaging in visible and SWIR with perfect pixel registration, both in standard bright-field and dark-field configurations. We have furthermore demonstrated that the instrument has a high degree of flexibility, switching from a high frame rate to high resolution automatically. We have also introduced a compressive technique that allows high resolution in tandem with high frame rate by basing our reconstruction on only the important patterns. All of these factors result in a microscope system that can produce images in multiple spectral bands simultaneously and gives great flexibility to tune its operation to suit a variety of applications. In comparison with commercially available systems based on standard microscopes utilizing IR cameras, this prototype system offers hyperspectrality and significant cost savings, at the expense of some performance in terms of frame rate and resolution. Furthermore, the current implementation extends the detection bandwidth into the SWIR, but this technique can be applied to a wide variety of wavelengths.