We propose a method to perform color imaging with a single photodiode by using light structured illumination generated with a low-cost color LED array. The LED array is used to generate a sequence of color Hadamard patterns which are projected onto the object by a simple optical system while the photodiode records the light intensity. A field programmable gate array (FPGA) controls the LED panel allowing us to obtain high refresh rates up to 10 kHz. The system is extended to 3D imaging by simply adding a low number of photodiodes at different locations. The 3D shape of the object is obtained by using a non-calibrated photometric stereo technique. Experimental results are provided for an LED array with 32 × 32 elements.
© 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
Digital cameras based on CCDs or CMOS sensors are key devices in most of the current imaging techniques. However, light sensors with a pixelated structure are not always necessary. In fact, imaging methods requiring raster scanning techniques, such as confocal or two-photon microscopy, use in general a single high sensitivity photosensor. Some other imaging techniques can benefit from light detectors with no spatial structure. This is the case of imaging with low light levels or with electromagnetic radiation in spectral ranges out of the visible region, particularly in the near IR and terahertz spectral bands, where it is more difficult to make a pixelated sensor [1–3]. Imaging with single-pixel detectors can be useful also when the purpose is to measure the spatial distribution of several optical parameters of the light beam simultaneously, not only the intensity but also the polarization state , the phase [5,6], the color [7–10], or the spectral content [11–16]. In particular, single-pixel recording strategies are well adapted to provide depth information and have been applied with success in many different 3D imaging techniques. For example, in time-of flight techniques [17,18], photometric stereo methods , binocular stereo approaches [8,20], and virtual fringe projection techniques [16,21]. Single-pixel recording techniques also provide extra flexible trade-off among SNR, frame-rate and resolution of the imaging system [22,23]. Some diffuse optical imaging (DOI) techniques  benefit also from single-pixel recording.
Single-pixel imaging techniques [25,26] are based on illuminating the scene with a sequence of light structured patterns while the light reflected or transmitted by the objects is recorded by a single photosensor such as a photodiode or photomultiplier tube (PMT). Alternatively to this active imaging system, where a spatial light modulator (SLM) modifies the input light beam, in the passive configuration the light coming from the object is sampled sequentially by a set of masks codified in the SLM. The image is then computed numerically by using different mathematical algorithms such as a simple linear superposition, a change of basis transformation, or a correlation operation. Aside from the ordinary raster scanning techniques, a common approach is to use light patterns codifying functions of a basis, such as Hadamard or Fourier components [26,27]. In this case, the image is obtained computationally by just an inverse transformation of the corresponding basis. The technique is very well adapted to apply compressive sensing algorithms by using different basis of functions, highly incoherent between them, to sample the object and to reconstruct the image .
Ghost imaging methods are similar to active single-pixel techniques, particularly in the case of computational ghost imaging. In these approaches the object is scanned with random light patterns and the image is obtained by correlation operations [29,30]. Active single-pixel imaging techniques are also related to dual photography, where the image is obtained by exploiting the Helmholtz reciprocity principle . Temporal modulation imaging is based on similar principles. In this case, images are obtained by using a single-pixel detector and scanning the object with temporally multiplexing spatial frequency functions from orthogonal, time varying spatial line modulation gratings .
As we have mentioned above, the capabilities of single-pixel imaging systems have been verified in many different applications [1–24] since the first introduction of the technique by Decker  and the significant improvement made by Duarte with programmable SLMs . However, two main limitations constrain further development. One is the time required to scan light patterns onto the object. The second is the cost of the spatial light modulators required to project light patterns with high flexibility. The first problem is being tackled by reducing the number of light patterns, for example using CS or adapting sensing techniques [26,28,33]. Hardware improvements have also allowed reducing the acquisition time by means of balanced detection  or by multimode detection [35,36]. Recently, a way to reduce cost has been proposed based on the use of LED panels to generate light patterns. LED arrays have been widely used in different imaging applications such as 4D light field capture , phase contrast imaging , and Fourier ptychography [39,40]. Also, it has been proposed to use LED arrays for single-pixel imaging in fluorescence application, achieving a very high pattern modulation frequency (250 kHz) although at a low resolution of 8 × 8 pixels . A remarkable single-pixel imaging system to obtain 2D monochromatic images, working at a very high frequency with a resolution of 32 × 32 pixels, was developed very recently .
In this paper we propose a method to perform 2D color imaging with a single photodiode by using light structured illumination generated with a low-cost LED array. Moreover, the system is extended to 3D imaging by simply adding a low number of photodiodes in different locations. The LED array is used to generate a sequence of Hadamard patterns which are projected onto the object by a simple optical system. A field programmable gate array (FPGA) controls the LCD panel allowing us to obtain high refresh rates (up to 10 kHz). Color is obtained by generating a Hadamard pattern for each RGB chromatic component by controlling the color of the LEDs, which allows us to use a single monochromatic photodiode. Experimental results are provided for an LED frequency operation of 10 kHz.
Information of the 3D structure of the object is obtained by combining a low number of images generated from photodiodes located at different positions. We take advantage from the fact that shifting a photodiode in single-pixel imaging is equivalent to shifting the light source in conventional imaging [19,31]. We use a photometric stereo technique based on shape-from-shading , which obtains information about the 3D surface from a series of images of the object acquired while varying the illumination direction in the scene. There are two different methodologies to tackle the problem, called uncalibrated and calibrated photometric stereo. In this paper we use an uncalibrated method , which does not require to know the positions of the light sources in advance, allowing us to develop a very flexible 3D imaging system.
2. Single-pixel 3D imaging by using a color LED display
The layout of our single-pixel imaging system based on a color LED display is shown in Fig. 1. An LED panel controlled by an FPGA is used to generate a sequence of light patterns codifying the Hadamard functions. The FPGA also stores the binary patterns to be sequentially sent. The light patterns are projected onto the object by using a conventional optical system. The light reflected by the object is collected by a single photodiode and digitized by a DAQ system attached to a computer. To obtain a color image, a sequence of Hadamard patterns is projected for each RGB chromatic component. The image is generated in the computer by a simple Hadamard inverse transformation for each chromatic component.
Let us consider an LED panel containing a total of N light emitters. In mathematical terms, the image of the object provided by photodiode k is considered as a column vector, Ok, containing the N components of the object, in lexicographical order, which will be sampled by the light patterns. The light intensity measured by the photodiode can be written as:Equation (1) is, in fact, the Hadamard transformation of the object onto the Hadamard frequency space. The object is then recovered by the inverse transformation .
For the case of 3D imaging, a low number K of photodiodes is located at different positions around the object, as is shown in Fig. 1. Alternatively, a single photodiode is shifted to different positions and the sequence of patterns is sent again. As is well known, shifting the photodiode in single-pixel imaging techniques is equivalent to illuminate the object from a different direction in conventional imaging. This property has been used in dual photography  and in single-pixel 3D imaging by applying shape from shading algorithms . In our system, with this approach we are able to obtain a set of different images to apply photometric stereo techniques without preliminary calibration.
Photometric stereo aims at inferring the normal vectors that define the surface of the object starting from K input images obtained with a conventional sensor in a specific position and illuminated by a light point source located at a distant position characterized by the direction vector . In our case, the LED projector is at a fixed position and the photodiodes are moved to K different positions to obtain the different images. If we assume that surfaces in the scene have a Lambertian behavior, the photodiode has a linear response, and assuming an orthographic projection, the object provided by our single-pixel camera in Fig. 1 from photodiode k, can be written as . Therefore, the image obtained for all the photodiodes is given as the result of the following matrix operation:
In principle, to solve the linear system in Eq. (2) for S, we need to know the photodiode directions. However, this is not always possible or convenient. Instead, we apply an uncalibrated method, which does not require to know the positions of the photodiodes in advance . Firstly, an initial pair of surface normals, , and photodiode directions, , are estimated by minimizing a least square cost function and using a singular value decomposition (SVD) method . The SVD method allows vectors S and L to be obtained from the initial estimation up to a linear transform ambiguity characterized by a matrix G, where G∈GL(3), in the following way 47], using an integrability constraint, which enforces that a consistent surface can be reconstructed from the estimated normal field. In the uncalibrated method used here, the GBR ambiguity is eliminated by exploiting points where the Lambertian reflection is maximal, called Lambertian diffuse reflectance maxima (LDRM) points . These points are located approximately at the same pixel position of the image intensity maxima. It has been show that this method is tolerant to the presence of ambient illumination and shadows.
3. Experimental results
Figure 2 shows a zenithal picture of the LED-based single-pixel camera configured by following the layout in Fig. 1. We use a 32 × 32 RGB LED matrix panel, with 4mm grid spacing (Adafruit). A simple lens, with a focal length f = 100 mm, located at a distance 35 cm far from the matrix, is used to project the patterns onto the object. The sensor is an amplified photodiode (Thorlabs PDA 100A-EC) and the FPGA module is an FPGA development KIT (Terasic, DE1-SoC board) with the Quartus Prime software used to codify the light patterns.
The resolution of the Hadamard patterns is limited by that of the LED array. Therefore, the images reconstructed by our system have a resolution limited to 32 × 32 pixels. For the optical configuration in Fig. 2, the size of the scene illuminated by the light patterns is 4 × 4 cm. The pattern modulation frequency is limited by the maximum clock frequency of the LED array, which is 25MHz. Our 32 × 32 LED matrix is electronically composed by two matrices with 16 × 32 LEDs that can operate simultaneously. Then the maximum modulation frequency for patterns with 32 × 32 pixels is around 40 kHz if we take into account the multiplexing time and the control signals. However, the quality of the images in terms of the signal to noise ratio (SNR) was not satisfactory for frequencies larger than 10 kHz. Without applying CS, or similar optimization algorithms, we need 32 × 32 Hadamard patterns to obtain an image. Thus, at a pattern projection rate of 10 kHz, the image frame rate is almost 10 Hz.
We have performed two imaging experiments at a sampling rate of 10 kHz with the setup in Fig. 2. In the first one, we just record color images of the input scene. To this end, we send the full set of 1024 Walsh-Hadamard patterns associated to a space with 32 × 32 pixels, for each primary color, red, green and blue. The color image is reconstructed from the coefficients provided by the photodiode intensity signal by using Eq. (2). The result of this experiment is shown in Fig. 3. To quantify the quality of the image we measured the SNR for each chromatic channel RGB. The result of the averaged SNR for the three measurements is 53 dB. We measured also the SNR of the 2D monochromatic images used in the 3D imaging experiment shown in Fig. 4. As an example, the SNR of the central image in Fig. 4(a) is 62 dB. The SNR increases by using all LED colors at the same time to codify the light patterns.
We can compare the resolution, modulation frequency, and cost of our single-pixel camera based on LEDs with those of light projection systems based on spatial light modulators. The resolution of our system is much lower than that provided by off-the-shelf video projectors based on digital micromirror device (DMD) or liquid crystal on silicon (LCOS) spatial light modulators, which is typically 1280 × 1024 pixels. However, the modulation frequency of these video projectors is limited to 60 Hz, which is two orders of magnitude lower than the frequency of the LED system with a similar cost. Some commercial video projectors allow to operate at higher frequencies by modulating binary patterns instead of three grey-level chromatic channels. In this case the modulation frequency can arrive to 1200 Hz but at a higher cost. Finally, scientific spatial light modulators based on DMDs, when controlled with high quality drivers, can arrive to frequencies as high as 22 kHz with high resolution and flexibility. However, with a cost higher in one order of magnitude.
In our second experiment we obtain a 3D reconstruction of the surface of the object. As explained in the previous section, firstly we record a set of images of the object with our single-pixel camera in Fig. 2, by shifting the photodiode to different positions. In particular, we record 5 images by moving the photodiode in the same horizontal plane (see Fig. 1). This ensures us that we had enough LDRM points to obtain a robust GBR parameter estimation. The images are shown in Fig. 4(a). It is possible to see how the object appears illuminated from different directions. These images are used as inputs of the uncalibrated photometric stereo algorithm in . The software code to apply this method is available in . The resulting 3D shape is shown in Fig. 4(b) and 4(c). The axis labels in Fig. 4(b) show the x, y, and z coordinates of the object surface in mm. Please note that the surrounding of the figure looks uniform blue because the shape-from-shading algorithm do not provide information for low values of the intensity. Figure 4(c) shows just a 3D representation of the surface.
To estimate the quality of our result, we have measured relative depths at different locations in the object with a Vernier caliper appreciating 0.05 mm. The object has a size of 32.5 × 37.2 × 27.1 mm. The relative error of our surface, averaged from measurements in three locations of the object, is 3.1 ± 0.2%. Also, we have estimated the error of an uncalibrated shape-from-shading method in the bibliography performed with a conventional camera. In the experiment by Hayakawa , the average error is estimated to be 2.2%, which is of the same order of magnitude of our result. Finally, we have compared the result of our single-pixel uncalibrated 3D imaging method with the single-pixel calibrated one described in . The root mean square error (RMSE) of our measurements compared to those performed with the caliper is 0,17 mm for an object with a total depth of 27.1 mm, while the RMSE of the measurements performed with the system in  compared with those provided by a stereo photogrammetric camera system is 4 mm for an object with a total depth of 250 mm. The RMSE error is lower in our case, but it should be taken into account that we measured the error only from a few measurements, while in  the comparison was performed for all the surface. Besides, in our case the total depth is much lower. The relative error in terms of the total depth is of the same order of magnitude in both cases.
We have proposed a method to perform 2D color imaging and 3D imaging with a single photodiode by using light structured illumination generated with an LED array. Our system allows us to record low-resolution images of color objects with a resolution limited to 32 × 32 pixels, the number of LEDs of the array. Despite the maximum modulation frequency of the LED panel is 40 kHz, we need to operate at a frequency of 10 kHz to obtain a satisfactory SNR. Therefore, the image frame rate is almost 10 Hz. With this frequency we obtained 2D images with SNR equal to 53 dB for color images and 62 dB for monochromatic ones.
The imaging system has a cost similar to off-the shelf video projectors based on DMD or LCOS spatial light modulators, which have a higher resolution but are limited to operate at lower frequencies in single-pixel imaging applications. Spatial light modulators based on DMDs and controlled with high quality drivers provide both larger resolutions and modulation frequencies but at a higher cost. The resolution of our system is low because of the limited size and pitch of the LED array. It could be possible to improve the resolution by combining several LED panels operating in parallel at the same frequency. In our case, however, the characteristics of our FPGA board would limit the total number of panels to three.
For 3D imaging we have used a non-calibrated photometric stereo technique avoiding a precise calibration of the position of the photodiodes. The accuracy of the depth profile was determined by direct comparison with measurements using a Vernier caliper. We estimated a depth RMSE of 0,17 mm and an averaged relative error of 3.1%.
We have used the Walsh-Hadamard basis for the sampling operation, but other different basis such as Fourier or Morlet wavelet could also be used. In this prove of concept experiment we did not performed CS but we plan to do it in the next applications of the camera. By using CS and optimizing the system to obtain higher frequency rates, we expect to get color 2D images at a frame rate of 40 fps.
Spanish Ministerio de Economía y Competitividad (project FIS2016-75618-R and FIS2015-72872-EXP), Generalitat Valenciana (project PROMETEO 2016-079) and Universitat Jaume I (project P1·1B2015-35).
References and links
1. C. M. Watts, D. Shrekenhamer, J. Montoya, G. Lipworth, J. Hunt, T. Sleasman, S. Krishna, D. R. Smith, and W. J. Padilla, “Terahertz compressive imaging with metamaterial spatial light modulators,” Nat. Photonics 8(8), 605–609 (2014). [CrossRef]
2. H. Chen, N. Xi, B. Song, and K. Lai, “Single pixel infrared camera using a carbon nanotube photodetector,” in Proc. IEEE Sens. (IEEE, 2011), 1362–1366.
3. M. P. Edgar, G. M. Gibson, R. W. Bowman, B. Sun, N. Radwell, K. J. Mitchell, S. S. Welsh, and M. J. Padgett, “Simultaneous real-time visible and infrared video with single-pixel detectors,” Sci. Rep. 5(1), 10669 (2015). [CrossRef] [PubMed]
6. F. Soldevila, V. Durán, P. Clemente, J. Lancis, and E. Tajahuerce, “Phase imaging by spatial wavefront sampling,” Optica 5(2), 164–174 (2018). [CrossRef]
7. S. S. Welsh, M. P. Edgar, R. Bowman, P. Jonathan, B. Sun, and M. J. Padgett, “Fast full-color computational imaging with single-pixel detectors,” Opt. Express 21(20), 23068–23074 (2013). [CrossRef] [PubMed]
8. E. Salvador-Balaguer, P. Clemente, E. Tajahuerce, F. Pla, and J. Lancis, “Full-color stereoscopic imaging with a single-pixel photodetector,” J. Disp. Technol. 12, 417–422 (2016).
10. B. L. Liu, Z. H. Yang, X. Liu, and L. A. Wu, “Coloured computational imaging with single-pixel detectors based on a 2D discrete cosine transform,” J. Mod. Opt. 64(3), 259–264 (2017). [CrossRef]
11. F. Magalhães, M. Abolbashari, F. M. Araùjo, M. V. Correia, and F. Farahi, “High-resolution hyperspectral single-pixel imaging system based on compressive sensing,” Opt. Eng. 51(7), 071406 (2012). [CrossRef]
12. F. Soldevila, E. Irles, V. Durán, P. Clemente, M. Fernández-Alonso, E. Tajahuerce, and J. Lancis, “Single-pixel polarimetric imaging spectrometer by compressive sensing,” Appl. Phys. B 113(4), 551–559 (2013). [CrossRef]
14. J. Huang and D. F. Shi, “Multispectral computational ghost imaging with multiplexed illumination,” J. Opt. 19(7), 075701 (2017). [CrossRef]
15. K. Shibuya, T. Minamikawa, Y. Mizutani, H. Yamamoto, K. Minoshima, T. Yasui, and T. Iwata, “Scan-less hyperspectral dual-comb single-pixel-imaging in both amplitude and phase,” Opt. Express 25(18), 21947–21957 (2017). [CrossRef] [PubMed]
16. Z. Zhang, S. Liu, J. Peng, M. Yao, G. Zheng, and J. Zhong, “Simultaneous spatial, spectral, and 3D compressive imaging via efficient Fourier single-pixel measurements,” Optica 5(3), 315–319 (2018). [CrossRef]
18. M. J. Sun, M. P. Edgar, G. M. Gibson, B. Sun, N. Radwell, R. Lamb, and M. J. Padgett, “Single-pixel three-dimensional imaging with time-based depth resolution,” Nat. Commun. 7, 12010 (2016). [CrossRef] [PubMed]
19. B. Sun, M. P. Edgar, R. Bowman, L. E. Vittert, S. Welsh, A. Bowman, and M. J. Padgett, “3D computational imaging with single-pixel detectors,” Science 340(6134), 844–847 (2013). [CrossRef] [PubMed]
20. W. K. Yu, X. R. Yao, X. F. Liu, L. Z. Li, and G. J. Zhai, “Three-dimensional single-pixel compressive reflectivity imaging based on complementary modulation,” Appl. Opt. 54(3), 363–367 (2015). [CrossRef]
22. D. B. Phillips, M.-J. Sun, J. M. Taylor, M. P. Edgar, S. M. Barnett, G. M. Gibson, and M. J. Padgett, “Adaptive foveated single-pixel imaging with dynamic supersampling,” Sci. Adv. 3(4), e1601782 (2017). [CrossRef] [PubMed]
23. M. J. Sun, M. P. Edgar, D. B. Phillips, G. M. Gibson, and M. J. Padgett, “Improving the signal-to-noise ratio of single-pixel imaging using digital microscanning,” Opt. Express 24(10), 10476–10485 (2016). [CrossRef] [PubMed]
24. A. Farina, M. Betcke, L. di Sieno, A. Bassi, N. Ducros, A. Pifferi, G. Valentini, S. Arridge, and C. D’Andrea, “Multiple-view diffuse optical tomography system based on time-domain compressive measurements,” Opt. Lett. 42(14), 2822–2825 (2017). [CrossRef] [PubMed]
26. M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008). [CrossRef]
28. E. J. Candès and M. B. Walkin, “An introduction to compressive sampling,” IEEE Signal Process. Mag. 25(2), 21–30 (2008). [CrossRef]
29. J. H. Shapiro, “Computational ghost imaging,” Phys. Rev. A 78(6), 061802 (2008). [CrossRef]
30. F. Devaux, P. A. Moreau, S. Denis, and E. Lantz, “Computational temporal ghost imaging,” Optica 3(7), 698 (2016). [CrossRef]
31. P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. P. A. Lensch, “Dual photography,” ACM Trans. Graph. 24(3), 745–755 (2005). [CrossRef]
32. P. Schlup, G. Futia, and R. A. Bartels, “Lateral tomographic spatial frequency modulated imaging,” Appl. Phys. Lett. 98(21), 211115 (2011). [CrossRef]
35. M. Herman, J. Tidman, D. Hewitt, T. Weston, and L. McMackin, “A higher-speed compressive sensing camera through multi-diode design,” Proc. SPIE 8717, 871706 (2013). [CrossRef]
36. M.-J. Sun, W. Chen, T.-F. Liu, and L.-J. Li, “Image retrieval in spatial and temporal domains with a quadrant detector,” IEEE Photonics J. 9(5), 3901206 (2017). [CrossRef]
38. L. Tian, J. Wang, and L. Waller, “3D differential phase-contrast microscopy with computational illumination using an LED array,” Opt. Lett. 39, 1326-1329 (2014).
40. L. Tian and L. Waller, “3D intensity and phase imaging from light field measurements in an LED array microscope,” Optica 2(2), 104–111 (2015). [CrossRef]
42. Z.-H. Xu, W. Chen, J. Penuelas, M. Padgett, and M.-J. Sun, “1000 fps computational ghost imaging using LED-based structured illumination,” Opt. Express 26(3), 2427–2434 (2018). [CrossRef] [PubMed]
43. R. J. Woodham, “Photometric method for determining Surface orientation from multiple images,” Opt. Eng. 19(1), 191139 (1980). [CrossRef]
44. P. Favaro and T. Papadhimitri, “A closed-form solution to uncalibrated photometric stereo via diffuse maxima”, IEEE Conference on Computer Vision and Pattern Recognition, 821–828 (2012). [CrossRef]
45. A. Yuille and D. Snow, “Shape and albedo from multiple images using integrability”, IEEE Conference on Computer Vision and Pattern Recognition, 158–164 (1997). [CrossRef]
46. H. Hayakawa, “Photometric stereo under a light source with arbitrary motion,” J. Opt. Soc. Am. A 11(11), 3079–3089 (1994). [CrossRef]
47. P. N. Belhumeur, D. J. Kriegman, and A. L. Yuille, “The bas-relief ambiguity,” Int. J. Comput. Vis. 35(1), 33–44 (1999). [CrossRef]