Open Access
Add to BibSonomyAbstract
A very thin image capturing system called TOMBO (thin observation module by bound optics) is developed with compound-eye imaging and digital post-processing. As an application of TOMBO, a multispectral imaging system is proposed. With a specific arrangement of the optical system, spatial points can be observed by multiple photodetectors simultaneously. A filter array inserted in front of the image sensor enables observation of the spectrum of the target. The captured image is reconstructed by a modified pixel rearranging method extended to treat multi-channel spectral data, in which pixels in the captured image are geometrically rearranged onto a multi-channel virtual image plane. Experimental results of the image reconstruction show the effectiveness of the proposed system.
©2004 Optical Society of America
1. Introduction
Recently, image capturing and modification have become extremely easy as a result of the popularization of digital cameras and the improved performance of personal computers. The rapid development of the Internet has resulted in the availability of various network-based services such as electronic commerce [1], telemedicine [2], and electronic art museums, which are all applications requiring high-resolution imaging. In such applications, accurate reproduction of the color, luster, and texture of real-world objects is demanded. For example, accurate color reproduction is important for correct diagnosis in telemedicine and for in-home medical care.
Multispectral cameras and multispectral displays have been studied to meet such demands [3]. However, current multispectral cameras are larger than normal cameras, because they must be equipped with a mechanism for changing the spectral transmittance, such as a filter wheel or a liquid crystal tunable filter. Therefore, in order to provide improved usability, there are demands for more compact multispectral imaging systems.
We have previously presented a very thin image capturing system called TOMBO (Thin Observation Module by Bound Optics) [4]. The TOMBO system is constructed of a compact compound-eye imaging system whose images are subjected to digital post-processing. The structure of the TOMBO imaging system is shown in Fig. 1. It consists of a microlens array, a signal separator, and an image sensor.
The imaging system of the TOMBO is considered as being composed of elemental parts called units. A small image captured by one unit is called a unit image, and the image captured by all the units is called a compound image. The compound image is a set of unit images which contain different information due to the difference in position of the individual units. Image reconstruction is achieved by digital processing using these differences. As an effective reconstruction method, a pixel rearranging method has been proposed [5], which we use in the present study; in this method, the captured image is reconstructed by projecting the pixels in the captured unit images onto a virtual image plane. In this process, software correction of the lens misalignment improves the projection accuracy. The reconstructed image produced by the pixel rearranging method, however, is generally not sharp due to partial overlapping of neighboring observation areas of the photodetectors. Therefore, to improve the image quality, we also apply digital filtering.
The TOMBO system is considered as a compound imaging system which is composed of multiple imaging units. This indicates that the different information of the target can be captured by the multiple imaging units concurrently. The different information of the target can be obtained by changing the characteristics of the individual imaging units. The color TOMBO [6] which captures the color information of red, green and blue channels independently is an application of the feature. As an extension in the number of the wavelength channels, multispectral information can be captured by the same architecture. However, modification of the system configuration and the reconstruction algorism are required for the multispectral imaging.
In this paper, as an application of TOMBO, a compact multispectral imaging system capable of observing the same point by multiple photodetectors is proposed. In Section 2, a TOMBO system including the reconstruction method is explained. In Section 3, an implementation method for the multispectral TOMBO system is explained. In Section 4, several experimental results obtained by a prototype multispectral TOMBO system are presented. In Section 5, the performance of the proposed method is verified.
2. TOMBO
As shown in Fig. 1, the TOMBO system is a compact imaging system composed of a microlens array, a signal separator, and an image sensor. Multiple images are captured by the individual imaging units, which are used for reconstruction of the target image. For the image reconstruction, a pixel rearranging method is utilized [5].
The image reconstruction with the pixel rearranging method is achieved by the following procedure, referring to the side view of the optical system shown in Fig. 2. The pixel values in each unit image are mapped on a virtual image plane using the information of the system setup. As a notable point, the pixels are projected without scaling. Namely, the information of a pixel in the captured image is corresponded to that of a point of the virtual image plane. This process approximates the information of a point of the virtual image plane by that of a blurred pixel in the unit images.
In the original algorism, the optical setup is assumed to be described geometrically for simplicity. Figure 2 suggests that the pixel coordinate on the virtual image plane can be described by the following parameters: the pixel coordinate on the unit image, the lens pitch of the lens array, the distance between the image sensor and the lens array, and the distance between the lens array and the virtual image plane. These parameters except the distance between the lens array and the virtual image plane can be predetermined as the hardware specification of the TOMBO system. Therefore, the pixel mapping is achieved by providing the distance between the lens array and the virtual image plane.
To reconstruct the correct information of the object, the distance between the lens array and the virtual image plane must be set equal to that between the lens array and the object. Note that the distance between the lens array and the object can be obtained by the parallax included in the individual unit images. The several methods are considered for this processing. For example, the shift value between two unit images is calculated by the cross-correlation, then the distance between the lens array and the virtual image plane can be determined.
On the virtual image plane whose location is determined the above process, all the pixels in all the unit images are projected. In this process, the projected pixels are nonuniformly distributed on the virtual image plane. For the pixel-dense region, the averaged value of the overlapped pixels is used to represent the point. For the pixel-sparse region, the point without a projected value is calculated by interpolation using the neighboring non-blank pixels. By these processes, the image of the object can be retrieved on the virtual image plane.
In the practical system, rotation and shift of the microlens array cause by the system misalignment must be considered. Thus a system model including these alignment errors is constructed and utilized for precise pixel mapping. In addition, calibration process using reference images for adjusting the system model to the actual system enables us to improve the processing accuracy.
3. Multispectral TOMBO
3.1. System setup
Figure 3 shows a schematic diagram of the multispectral TOMBO system. The multispectral TOMBO system consists of an interference filter array, a microlens array, a signal separator, and an image sensor. Using interference filters, we can select a narrower spectral range of transmittance, so that more spectral channels of the image can be separated. As a note, red, green, and blue color filters are usually used in a color image sensor because broad spectral range is required. For the interference filter array, two options can be considered as the case of the color TOMBO [6] One is a method wherein different spectral transmittances are assigned unit by unit, and the other is a method wherein different spectral transmittances are assigned pixel by pixel. Unit by unit separation of spectral channels is suitable for the multispectral TOMBO due to the ease of fabrication of such an interference filter array. The filter array can be placed in front of the microlens array or the image sensor.
3.2. Principle
Multiplexed observation of the same point on the target by multiple photodetectors is an essential condition to acquire multispectral information. To explain the multiplexed observation, the distribution of the points observed by the photodetectors with respect to the object distance is considered. The optical setup of the proposed system is shown in Fig. 4. The figure is shown in two dimensions for simplicity.
A uniformly sampled object distance, d unif, is defined as the distance at which the target is observed with an identical interval in an interleave manner by all photodetectors. The target is observed over uniformly, when the following equation is satisfied.
where M, p, l, and N are the magnification, the pixel pitch, the lens pitch, and the number of units used in image reconstruction, respectively. In this condition, the target is observed by the adjacent units with a shift of Mp/N in the object plane for the case of N×N units. The uniformly sampled object distance d unif is given by
where f is the focal length.
In the multispectral TOMBO system, the object distance is set at 1/2, 1/3, 1/4,… of the uniformly sampled object distance d unif. Planes A, B, and D in Fig. 4 indicate the observation planes at the uniformly sampled object distance d unif and 1/2, 1/4 thereof, respectively. Plane C in Fig. 4 represents the case of a distance other than the ones specified above. Figure 5 shows a simulation of the distributions of the observed points at the observation planes in Fig. 4.
The sizes of the dots indicate the multiplexed number of observations by the different photodetectors, where the sizes of the dots correspond to 1, 4, 9, and 16 multiplexed observations in increasing order of size. The target is observed uniformly at the object distance A, whereas at B and D, several points of the target are observed simultaneously by multiple photodetectors. Therefore, if the multiplexed photodetectors are set to capture different wavelength channels using filters, spectral images can be captured.
Note that, at an observation plane other than B and D such as C, the observation points are not overlapped at all. This means each observation point is observed by just one detector. As a result, photo detectors assigned to different spectral channels capture different spatial positions and true multispectral information of a point can not be observed.
3.3. Reconstruction
Image reconstruction is achieved by an extended version of the pixel rearranging method to treat multi-channel data. The processing flow is shown in Fig. 6. Processing for the reconstruction consists of pre-processing, pixel-remapping, and post-processing.
In the pre-processing, the captured compound image is corrected to eliminate shading error and fixed pattern noise using previously captured black and white reference images. The shading error is non-uniformity caused by the shadow of the separation wall and the non-uniform sensitivity of the image sensor. The pre-processing is achieved by
where c′(x,y), c(x,y), w(x,y), and b(x,y), are the corrected image, the captured image, black reference image and white reference image, respectively. x and y are the pixel coordinate.
In the pixel-remapping processing, a virtual plane composed of multiple channels is set at the object distance. In this step, the pixel structure is not assumed on a virtual plane. The projected positions of the pixels of the captured images on the virtual plane are then calculated by the same process as the original pixel rearranging method. Since each pixel must be projected to either of wavelength channels, the channel is assigned referring to the pattern of the color filter array. The pixel values of the captured image are geometrically projected to the virtual plane. The pixels are projected without scaling. Then the virtual plane is spatially sampled with an appropriate interval to generate the remapped image. With a finer interval, more accurate projecting positions can be obtained. It makes possible to reconstruct independent of the alignment errors and the object distance. To compensate spatial distribution of the projected pixels in the individual spectral channels, either averaging or interpolation is applied as the original method.
As the post-processing, the remapped image is sharpened by digital filtering. For instance, the unsharp mask, one of the digital filtering, or an iterative backprojection method [8], one of the super resolution method could be applied.
4. Experiment
4.1. System setup
To study the performance of the proposed method, a color image is captured using a prototype multispectral TOMBO system shown in Figure 7.
The specifications of the microlens array (Advanced Microoptic Systems, GmbH, APO-Q-P500-AF1.3) are as follows: lens pitch, 500 µm; focal length, 1.3 mm; and lens diameter, 500 µm.
As the image sensor, a custom-designed CMOS photodetector array (Austria Micro Systems, 0.35-µm design rule) was used. The number of pixels was 1040×960 and the pixel pitch was 6.25 µm. Therefore, the number of pixels per unit was 80×80, as determined by the lens pitch and the pixel pitch. Up to 12×12 units could be used for the reconstruction processing. The 80 marginal pixels at each of the left and right sides were utilized for lens alignment.
The signal separator was made by thermocompression of 21 plates of stainless steel (50 µm thickness) and holes corresponding to the square apertures in Figs. 1 to 3 were fabricated by etching. The inner surface of each etched hole was finished with an anti-reflection coating. The wall thickness and the height of the each signal separator were approximately 50 µm and 1050 µm, respectively. The hole pitch was 500 µm, which was equal to the lens pitch.
The microlens array was held by a six-axis stage and placed in front of the signal separator and the image sensor. The uniformly sampled object distance in this experiment was 1248 mm, as given by Eq. (2).
Experiments were carried out using this prototype system equipped with seven narrow-band interference filters (CVI Laser Corp., product code: FS40-VIS-2.00) instead of a specially constructed array of interference filters. The spectral transmittances of the filters are shown in Fig. 8. The central wavelengths of the filters range from 400 to 700 nm in steps of 50 nm. A multi-channel compound image equivalent to the image captured with the filter array is generated from the seven compound images captured with these filters.
4.2. Experimental results
Images of some fruits and vegetables were captured by the TOMBO prototype. The object distance was set to 416 mm, which is 1/3 of the uniformly sampled object distance. The captured image size was 1040×960 pixels and the bit-depth was 12 bits. Only the 960×960 pixels at the center of the captured image were clipped out and utilized for reconstruction. The captured image was then corrected for non-uniform shading and fixed pattern noise using reference images captured previously.
The filter array pattern shown in Fig. 9(a) was used as the filter arrangement. A multispectral compound image was generated from seven single-channel compound images, as shown in Fig. 9(b). The experimental results reconstructed from the multispectral compound image are shown in Figs. 9(c)-(i) for different wavelengths. The reconstructed image size was 480×480 pixels and the bit-depth was 12 bits. Thus, reconstructed images corresponding to the interference filters were obtained from the multispectral compound image.
An RGB image, which is obtained by applying spectral-RGB conversion to the seven spectral images, is shown in Fig. 10. The size of the reconstructed image was 480×480 pixels and the bit-depth in each channel was 12 bits.
5. Discussion
To evaluate the spatial displacement of the observation area, a simple target having broad spectral components was observed. As the target, an opaque sheet with a 50×50 mm square aperture, shown in Fig. 11(a), was illuminated by white light from the back side. The images reconstructed by the same procedure as the previous experiment are shown in Figs. 11(b)-(h). To emphasize the spatial displacement, the images were binarized with a suitable threshold value selected from their gray-level histograms [7]. The images were trimmed to 200×200 pixels to exclude the edge pixels which contain noise caused by the separation walls.
The centroid of the white square of the target and the individual wavelengths were calculated as a metric for the evaluation. The centroid (Gx,Gy) is given by
where g(x,y) and T, are the reconstructed image, and the sum of the pixel value, respectively.
The calculated results are shown in Table 1.
The maximum displacement of the centroid between the target and the reconstructed image is one pixel in the horizontal and vertical directions except the image of 400 nm band. The image of 400 nm is extremely degraded due to low transmission of the filter and poor sensitivity of the image sensor at the wavelength. Consequently, we confirm that the spatial displacement of the observation area is considerably small and that the proposed method provides good performance in spatial identity.
Table 1. Centroid for target chart and wavelengths
6. Summary
In this paper, a multispectral TOMBO (thin observation module by bound optics) system capable of observing the same point by multiple photodetectors has been presented as one application of the TOMBO system. An extended reconstruction method specifically suited for this system has been proposed. The validity of the proposed reconstruction method was confirmed by image capturing experiments. Seven spectral images corresponding to seven spectral filters were retrieved from a multispectral compound-eye image. A comparison between a reference image and the spectral images demonstrates the effectiveness of the proposed method. The results of this study are expected to contribute to development of compact multispectral imaging systems.
Acknowledgments
This research was supported by an “Ultra-Thin Image Capturing Module” grant awarded by the Japan Science and Technology Agency under the Science and Technology Incubation Program in Advanced Region.
References and links
1. W. Ford and M.S. Baum, Secure Electronic Commerce: Building the Infrastructure for Digital Signatures and Encryption (Prentice Hall Inc., Upper Saddle River, NJ, 2001).
2. Masahiro Nishibori, “The Role of Multispectral Imaging in Medicine,” in Proc. of International Symposium on Multispectral Imaging and Color Representation for Digital Archives (1999), pp. 114–116.
3. M. Yamaguchi, T. Teraji, K. Ohsawa, T. Uchiyama, H. Motomura, Y. Murakami, and N. Ohyama, “Color image reproduction based on the multispectral and multiprimary imaging: Experimental evaluation,” Color Imaging: Device-Independent Color, Color Hardcopy, and Applications VII, Proc. SPIE 4663, 15–26 (2002).
4. J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T. Morimoto, N. Kondou, D. Miyazaki, and Y. Ichioka, “Thin observation module by bound optics (TOMBO): concept and experimental verification,” Appl. Opt. 40, 1806–1813 (2001). [CrossRef]
5. J. Tanida, Y. Kitamura, K. Yamada, S. Miyatake, M. Miyamoto, T. Morimoto, Y. Masaki, N. Kondou, D. Miyazaki, and Y. Ichioka, “Compact image capturing system based on compound imaging and digital reconstruction,” Micro- and Nano-optics for Optical Interconnection and Information Processing, Proc. SPIE 4455, 34–41 (2001). [CrossRef]
6. J. Tanida, R. Shogenji, Y. Kitamura, K. Yamada, M. Miyamoto, and S. Miyatake, “Color imaging with an integrated compound imaging system,” Opt. Express 11, 2109–2117 (2003), http://www.opticsexpress.org/abstract.cfm?URI=OPEX-11-18-2109. [CrossRef] [PubMed]
7. Nobuyuki Otsu, “A Threshold Selection Method from Gray-Level Histograms,” IEEE Trans. Sys., Man, and Cybernetics SMC-9, No.1, 62–66 (1979).
8. M. Irani and S. Peleg, “Improving resolution by image registration,” CVGIP: Graphical Models and Image Processing 53, 231–239 (1991). [CrossRef]
