Single-pixel cameras provide a means to perform imaging at wavelengths where pixelated detector arrays are expensive or limited. The image is reconstructed from measurements of the correlation between the scene and a series of masks. Although there has been much research in the field in recent years, the fact that the signal-to-noise ratio (SNR) scales poorly with increasing resolution has been one of the main limitations prohibiting the uptake of such systems. Microscanning is a technique that provides a final higher resolution image by combining multiple images of a lower resolution. Each of these low resolution images is subject to a sub-pixel sized lateral displacement. In this work we apply a digital microscanning approach to an infrared single-pixel camera. Our approach requires no additional hardware, but is achieved simply by using a modified set of masks. Compared to the conventional Hadamard based single-pixel imaging scheme, our proposed framework improves the SNR of reconstructed images by ∼ 50 % for the same acquisition time. In addition, this strategy also provides access to a stream of low-resolution ‘preview’ images throughout each high-resolution acquisition.
© 2016 Optical Society of America
Conventional digital cameras use a lens system to form an image of a scene onto a detector array. The spatial resolution of the recorded image can be limited either by the point spread function of the optical system, or by the pitch of the pixels in the detector array. If the resolution is limited by the pixel pitch, the most common method to improve this is to increase the number of pixels per unit area by reducing their physical size. However, apart from the technological challenges associated with this approach, smaller pixels detect less light, which degrades image quality . An alternative approach to increase the pixel resolution is microscanning [2–6]. In this approach, multiple images of the same scene are recorded, and the pixelated detector is displaced by sub-pixel sized translations between each images. Data from these images is then combined to reconstruct a composite image with a spatial resolution exceeding that of the detector array.
While there has been a global drive to increase the number of pixels in camera sensors, there has also been significant developments in camera technology that records images using just a single-pixel detector [7–9]. These techniques require a means for sampling the spatial distribution of scene intensity, which can be achieved using a series of masks. The single-pixel detector is then used to measure the correlation between the scene and each mask. Knowledge of the transmitted intensities and the corresponding masks enables reconstruction of the image. Two of the earliest examples of single-pixel imaging were the flying-spot camera patented by Paul Nipkow in 1884 and the ‘Televisor’ (television) pioneered by John Logie Baird in 1926, both of which provided a method for encoding and transmitting image information using a rotating Nipkow disk.
At visible wavelengths, the advent of pixelated detector arrays such as charge-coupled device (CCD), which offered better signal-to-noise performance, superseded single-pixel techniques. However, at spectral regions where detector arrays are prohibitively expensive or unavailable, single-pixel imaging still has applications. In particular, the advent of MicroElectroMechanical Systems (MEMS) devices, which have been made affordable due to huge consumer demand, provides a digital means for spatial light modulation and perhaps most importantly, doing so at high-speed. In addition to reconstructing individual image frames [7, 8], single pixel cameras have also been demonstrated at near video-rates by employing efficient image reconstruction algorithms [10–12].
Within the single-pixel camera approach, it is the spatial resolution of the masking patterns that is equivalent to the pixel resolution. The number of masks required to reconstruct a fully sampled image increases with the square of the resolution (i.e. in proportion to the total number of pixels in the reconstructed image). However, in single-pixel imaging systems there is also a further restriction: the signal-to-noise ratio (SNR) of the reconstructed image decreases as the resolution is increased. There are two main sources of noise: fluctuations in the ambient illumination level between different mask pattern displays (illumination noise) and noise associated with the photodiode measurement itself (detector noise). In addition there are also sources of noise correlated with some aspects of the system, such as stray light rejected by the digital micromirror device (DMD) bouncing around the camera housing back into the photodiode. The ratio of illumination to detector noise is dependent upon the light level. At low light levels the detector noise tends to dominate, whilst at higher light levels, the illumination noise tends to dominate. However, both illumination and detector noise scale in the same way with increased resolution, and so the SNR is inversely proportional to the square root of the number of image pixels, regardless of illumination level. This trade-off between the image resolution and its SNR is a key drawback limiting applications of single-pixel camera technology. To improve SNR, schemes such as differential ghost imaging [13–18], have been developed to reduce the noise without jeopardizing resolution.
In this work, we show that it is possible to further enhance the pixel resolution of a single-pixel camera, while maintaining its SNR, by adopting a digital microscanning approach. We obtain multiple low-resolution images, each laterally shifted by sub-pixel steps. These images are then co-registered on a higher resolution grid to give a single high-resolution image. We demonstrate that by sampling with the same number of masks, and therefore in the same acquisition time, this methodology results in a reduced noise level in the final high-resolution reconstructed image for only a slight reduction in resolution. In addition, our method simultaneously delivers a sequence of low-resolution ‘preview’ images during the high-resolution image acquisition. We note that the concept of digital microscanning in the context of single-pixel imaging was recently considered theoretically in . Our digital microscanning technique is applicable to all single-pixel imaging systems, both those based on projected light fields and those based on image masking, the latter of which is demonstrated here. More importantly, digital microscanning in the single-pixel camera context requires no extra hardware and only a trivial increment in algorithm complexity. Therefore, our method can be deployed as a complement to existing single-pixel camera schemes [8, 9, 20, 21], including those utilising compressive sensing [8,22,23].
2. Experimental setup
Figure 1 illustrates our infrared single-pixel camera set-up, which is a modified and compacted version of the system in our previous work . We test our system by imaging a grey-scale picture, which is located ∼ 0.5 m from the single-pixel camera and illuminated with a heat reflector lamp (Philips PAR38 IR 175C, wavelength: 400nm−1200nm). The camera lens (Nikon AF Nikkor, f/1.8D) images the scene onto a high-speed DMD (Texas Instruments Discovery 4100, 1024 × 768, wavelength: 350nm − 2500nm, operating at 22kHz) which is used to sequentially mask the image of the scene with a preloaded sequence of binary masks. The total intensity of light transmitted through each mask is detected by a InGaAs detector (Thorlabs PDA20CS InGaAs, 800 nm to 1800 nm, 0 dB–70 dB gain). A high dynamic range analogue-to-digital converter (ADC, National Instruments DAQ USB-6221 BNC, sampling at 250kSs−1), synchronised with the DMD, acquires and transfers the intensity data to a computer for image reconstruction. A photograph of our integrated single-pixel camera prototype is shown in Fig. 1.
As in our previous work, we make use of Hadamard matrices to form our DMD masking patterns [24,25]. This is a convenient basis, as the rows (or columns) of the Hadamard matrices form a complete orthogonal set, enabling efficient sampling of the image at a well defined resolution using a given set of patterns [10, 12]. We note that in all images shown in this work, the end-to-end framework is limited by Gaussian sensor noise, and thus Hadamard mask sets were the optimal basis (chosen over pixel scanning, for example) . The elements of the Hadamard matrices take values of ‘1’ or ‘-1’, and each row is reformatted into a 2D grid and displayed as a 2D binary mask on the DMD, where ‘1’ and ‘-1’ denote micromirrors states ‘On’ and ‘Off’ respectively. Therefore, the intensity signal Sp associated with a particular masking pattern Mp is given by:
In order to reduce sources of noise such as fluctuations in ambient light levels, we obtain differential signals by displaying each Hadamard mask immediately followed by its inverse (where the micromirror status ‘On’ and ‘Off’ are reversed), and taking the difference in the measured intensities [13, 14, 16]. This scheme also ensures that each difference measurement is linearly independent from the others, and so the image is critically sampled (the same number of measurements as pixels), and Eq. (2) is equivalent to a linear sum of orthogonal measurements.
In our experiments, we utilise the square central region of our DMD, which consists of 768 × 768 micromirrors. The DMD micromirrors are grouped together to display the reformatted Hadamard masks. For example, if displaying an 8 × 8 pixel Hadamard mask, each Hadamard pixel comprises of 96 × 96 adjacent micromirrors. In order to apply digital microscanning to single-pixel imaging, it is the position of the Hadamard masks on the DMD that must be laterally displaced by sub-Hadamard-pixel sized translations. In our results, we compare two different microscanning approaches, based on the percentage of active micromirrors used within each pixel, which are detailed below. We contrast both of these digital microscanning methods with the standard technique of increasing the image resolution by simply increasing the resolution of the Hadamard mask set, which we refer to here as ‘normal’ high-resolution (NHR) imaging.
The complete microscanning (CM) method uses lower resolution masks which utilise all of micromirrors within each Hadamard pixel. Figure 2 illustrates the reconstruction of a 16 × 16 pixel image from four 8 × 8 pixel images of the equivalent field-of-view. A set of 8 × 8 pixel Hadamard masks (each pixel comprising of 96 × 96 DMD micromirrors, except along the edges of the shifted images as shown in Fig. 2(b). In these cases the edge pixels are 96 × 48 micromirrors for x shifted patterns, 48 × 96 micromirrors for y shifted patterns, and 48 × 48 micromirrors for the bottom right pixel of the patterns shifted in both x and y) is displayed four times, with a lateral shift in x and/or y of half a pixel width (i.e. 48 DMD micromirrors) applied between each set (Fig. 2(b)). This operation yields four 8 × 8 pixel images (Fig. 2(c)), using the same number of masks as required to obtain a 16 × 16 pixel NHR image. Each of these 8 × 8 pixel images contains different spatial information. A 16 × 16 pixel image is then reconstructed by co-registering each 8 × 8 pixel image in its laterally shifted location on a 16 × 16 grid, and averaging the overlapping pixel values for each high-resolution pixel (Fig. 2(d)). Therefore, as shown in Fig. 3(a), the value of the hatched central pixel on the high-resolution grid is computed as:
In the absence of noise, the reconstructed result using CM is mathematically equivalent to the convolution of the NHR image (obtained in the standard way using a sequence of 16 × 16 Hadamard masks in this example) with a smoothing kernel:
This convolution causes a modest reduction in the contrast of the high spatial frequencies in the reconstructed CM image. However, CM does offer a significantly improved SNR compared to the NHR method, because the SNR of a single-pixel camera image decreases roughly in proportion to the square root of the number of mask pixels. Therefore a high-resolution image reconstructed from four lower-resolution images inherits the higher SNR of the lower-resolution images.
We note that it is possible to use a matrix inversion to recover the NHR image from the four laterally shifted low resolution images. The matrix inversion method is equivalent to solving a set of simultaneous equations describing the intensity of each high-resolution pixel in terms of the low resolution measurements. However, matrix inversion requires the use of appropriate boundary conditions at the edges of the image, and we found this method to be highly unstable with respect to small levels of noise in the measurements. Deconvolution algorithms such as Weiner deconvolution [27,28] could also be used to restore the high resolution image from the CM image.
The quarter microscanning (QM) method again allows reconstruction of a higher resolution image from four half-pixel laterally shifted images. However, in this case each lower resolution image is recorded with only one quarter of the micromirrors active within each pixel, as shown in Fig. 3(b). The four quadrants of the pixel are sequentially utilised, and each quadrant corresponds to a half-pixel shift in x and/or y. This ensures that each low-resolution image samples a non-overlapping region of the scene. Therefore, the four low-resolution images obtained using QM can simply be co-registered accordingly on a high-resolution image grid, with no need for averaging. Consequently, the resolution of the QM image is identical to that of the NHR image in the absence of noise. As with CM, QM also delivers low-resolution images within each high-resolution image acquisition. However, as only a quarter of the micromirrors within each pixel are active during the acquisition, QM suffers from a reduced SNR compared to CM and NHR.
Figure 4 shows a comparison of images recorded using the different approaches. To test the performance of each method, we image a modified US Air Force (USAF) resolution test chart. Figures 4(a)–4(d) show 128 × 128 pixel images reconstructed using (a) CM, (b) QM, (c) NHR and (d) NHR convolved (NHRC) with a kernel (Eq. (4)), the data for Figs. 4(a)–4(d) are obtained in less than 2seconds using 32,768 patterns (2 × 1282. The factor of 2 accounts for the pattern and its inverse). Figure 4(e) shows an example of a 64 × 64 pixel image, the data for (e) are obtained in 0.5seconds using 8,192 patterns (2 × 642) and linearly interpolated into 128 × 128 resolution for comparison. For each image we calculate the SNR using [29,30]:Fig. 4(e)), 〈Ib〉 is the average intensity of the background (here calculated from the data highlighted by the dashed red square in Fig. 4(e)), and σf and σb are the standard deviations of the intensities in the feature and the background respectively. The calculated results show that the SNR of the image reconstructed using CM is significantly higher than that of the image obtained using NHR and NHRC with the smoothing kernel. Magnified views of the high frequency regions (highlighted by the blue dashed box in Fig. 4(e)) are shown at bottom-right of each sub-figure.
Figures 4(f) and 4(g) show intensity distributions through two vertical lines of the USAF test chart containing different spatial frequencies. In Fig. 4(f), the low frequency features (highlighted by the blue solid line in Fig. 4(e)) are well resolved using all five imaging methods. In Fig. 4(g), the intensity distributions (highlighted by the blue dashed line in Fig. 4(e)) of the high frequency features are given. In this case, the features are now only resolved in the images obtained using CM, QM, NHR and NHRC. The effect of the convolution is evident in both CM and NHRC, resulting in a modest reduction in contrast compared to NHR and QM. Figure 5 demonstrates that the trend in image SNR discussed above is also reproducible when imaging another object.
In low light levels, the advantage of improved SNR using CM becomes more significant. For example, in Figs. 6(a)–6(b) the noise is so severe when using the NHR method that parts of the image are unidentifiable. However, under identical illumination conditions, CM enables a dramatic improvement in image quality, as shown in Figs. 6(e)–6(f). As discussed above, in the absence of noise CM is mathematically equivalent to NHR convolved with the smoothing kernel . However even when the NHR images are themselves smoothed with kernel , the SNRs of the resulting images (Figs. 6(c)–6(d)) are still far lower than the images obtained using CM.
While CM offers advantages at low light levels, the QM may find more use operating in high light conditions. For example, in our experiments the photodiode gain is set in order to maximise the dynamic range of the ADC. As the QM method reduces the amount of light arriving at the photodiode, the gain can be increased to improve the QM image quality. This is demonstrated in Fig. 7 where Fig. 7(a) shows images obtained using QM with a 40 dB gain (equivalent to the gains used in all images shown in Figs. 4 and 5), while Figs. 7(b) shows the improvement in SNR when using a gain of 50 dB with the QM method. However, the same improvement cannot be achieved in NHR and CM, because increasing the detector gain saturates the photodiode outputs, resulting in inaccurate meansurements and distorted image reconstructions.
We have demonstrated two image reconstruction strategies based on a digital microscanning approach applied to an infrared single-pixel camera. Both of these strategies deliver a sequence of low-resolution ‘preview’ images throughout the high-resolution image acquisition. For example we obtain 128 × 128 pixel images at a rate of ∼0.5 Hz. During the high-resolution acquisition our method delivers 64 × 64 pixel ‘preview’ images at a rate of ∼2 Hz. Compared to conventional sampling, the complete microscanning strategy improves the SNR of images, at the expense of a modest reduction in the contrast of high spatial frequencies. The quarter microscanning strategy suffers from reduced SNR, which under certain situations can be improved by optimising the photodiode gain. These techniques are applicable to all single-pixel imaging systems, both those based on structured illumination and those based on masked detection. Importantly, our approach requires no additional hardware and can be utilised as a complement to existing single-pixel imaging schemes, such as compressive sensing. Our work demonstrates an additional degree of flexibility in the trade-off between SNR and spatial resolution in single-pixel imaging techniques, which can be optimized as the case demands.
MS and DBP thank Dr. Jonathan Taylor for helpful discussion. MS acknowledges the support from National Natural Science Foundation of China (Grant No. 61307021) and China Scholarship Council (Grant No. 201306025016). DBP is grateful for support from the Royal Academy of Engineering. MJP acknowledges financial support from the Wolfson foundation, EPSRC QuantIC (EP/M01326X/1) and the Royal Society. MS and MJP conceived the concept of the experiment. MS, MPE and GMG conducted the experiments. MS, DBP and MJP designed the reconstruction algorithm and analysed the results. All authors contributed to writing the manuscript. The authors declare no competing financial interests.
References and links
1. K. Nasrollahi and T. B. Moeslund, “Super-resolution: a comprehensive survey,” Mach. Vision Appl. 25, 1423–1468 (2014). [CrossRef]
2. P. Milanfar, Super-Resolution Imaging (CRC, 2010), Ch. 1, pp. 2–4.
3. R. C. Hardie, K. J. Barnard, and R. Ordonez, “Fast super-resolution with affine motion using an adaptive wiener filter and its application to airborne imaging,” Opt. Express 19, 26208–26231 (2011). [CrossRef]
5. M. Sun and K. Yu, “A sur-pixel scan method for super-resolution reconstruction,” Optik 124, 6905–6909 (2013). [CrossRef]
7. P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. Lensch, “Dual photography,” ACM Transactions on Graphics (TOG) , 24, 745–755 (2005). [CrossRef]
8. M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. Kelly, and B. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25, 83–91 (2008). [CrossRef]
9. J. H. Shapiro, “Computational ghost imaging,” Phys. Rev. A 78, 061802 (2008). [CrossRef]
10. A. C. Sankaranarayanan, C. Studer, and R. G. Baraniuk, “Cs-muvi: Video compressive sensing for spatial-multiplexing cameras,” in Proceedings of IEEE International Conference on Computational Photography (IEEE2012), pp. 1–10.
11. N. Radwell, K. J. Mitchell, G. M. Gibson, M. P. Edgar, R. W. Bowman, and M. J. Padgett, “Single-pixel infrared and visible microscope,” Optica 1, 285–289 (2014). [CrossRef]
12. M. P. Edgar, G. M. Gibson, R. W. Bowman, B. Sun, N. Radwell, K. J. Mitchell, S. S. Welsh, and M. J. Padgett, “Simultaneous real-time visible and infrared video with single-pixel detectors,” Sci. Rep. 5, 10669 (2015). [CrossRef] [PubMed]
13. F. Ferri, D. Magatii, L. A. Lugiato, and A. Gatti, “Differential ghost imaging,” Phys. Rev. A 104, 253603 (2010).
14. B. Sun, S. S. Welsh, M. P. Edgar, J. H. Shapiro, and M. J. Padgett, “Normalized ghost imaging,” Opt. Express 20, 16892–16901 (2012). [CrossRef]
15. K.-H. Luo, B.-Q. Huang, W.-M. Zheng, and L.-A. Wu, “Nonlocal imaging by conditional averaging of random reference measurements,” Chin. Phys. Lett. 29, 074216 (2012). [CrossRef]
16. B. Sun, M. P. Edgar, R. W. Bowman, L. E. Vittert, S. S. Welsh, A. Bowman, and M. J. Padgett, “Differential computational ghost imaging,” in Computational Optical Sensing and Imaging Conference (Optical Society of America, 2013), paper CTu1C–4.
18. S.-C. Song, M.-J. Sun, and L.-A. Wu, “Improving the signal-to-noise ratio of thermal ghost imaging based on positive-negative intensity correlation,” Opt. Commun. 366, 8–12 (2016). [CrossRef]
19. D. Shi, J. Huang, F. Wang, K. Cao, K. Yuan, S. Hu, and Y. Wang, “Enhancing resolution of single-pixel imaging system,” Opt. Rev. 22, 802–808 (2015). [CrossRef]
20. B. Sun, M. P. Edgar, R. W. Bowman, L. E. Vittert, S. S. Welsh, A. Bowman, and M. J. Padgett, “3D computational imaging with single-pixel detectors,” Science 340, 844–847 (2013). [CrossRef] [PubMed]
21. Z. Zhang, X. Ma, and J. Zhong, “Single-pixel imaging by means of fourier spectrum acquisition,” Nat. Comm. 6, 6225(2015). [CrossRef]
22. E. J. Candès, “Compressive sampling,” in Proceedings of the International Congress of Mathematicians (International Mathematical Union, 2006) 3, pp. 1433–1452.
23. D. L. Donoho, “Compressed sensing,” IEEE T. Inform. Theory 52, 1289–1306 (2006). [CrossRef]
24. W. K. Pratt, J. Kane, and H. C. Andrews, “Hadamard transform image coding,” in Proceedings of the IEEE (IEEE, 1969) 57, pp. 58–68.
27. W. Wallace, L. H. Schaefer, and J. R. Swedlow, “A working person’s guide to deconvolution in light microscopy,” Biotechniques 31, 1076–1097 (2001).
28. N. Dey, L. Blanc-Feraud, C. Zimmer, P. Roux, Z. Kam, J. C. Olivo-Marin, and J. Zerubia, “Richardson–lucy algorithm with total variation regularization for 3d confocal microscope deconvolution,” Microsc. Res. Tech. 69, 260–266 (2006). [CrossRef] [PubMed]
29. R. N. Bryan, Introduction to the Science of Medical Imaging (Cambridge University, 2009), Chap. 3, pp. 85–88.