## Abstract

In this paper the influence of the number of lenslets on the performance of image restoration algorithms for the thin observation module by bound optics (TOMBO) imaging system was investigated, and the lenslet number was optimized to achieve thin system and high imaging performance. Subimages with different numbers of lenslets were generated following the TOMBO observation model, and image restoration algorithms were applied to evaluate the imaging performance of the TOMBO system. The optimal lenslet number was determined via theoretical performance optimization and verified via experimental comparisons of angular resolutions of two TOMBO systems and a conventional single-lens system.

© 2014 Optical Society of America

## 1. Introduction

Thin observation module by bound optics (TOMBO) based on a combination of imaging techniques and efficient image processing algorithms is a fast-developing area of computational photography [1–3]. The TOMBO imaging system is an optical system that achieves thinness and high-resolution imaging by replacing a conventional full-aperture lens with a lenslet array [4]. The optical arrangement of the TOMBO imaging system is shown in Fig. 1. Regarding thinness, the focal length of each lenslet can be 1/*n* of the conventional single-lens system with equal *f* number, with *n* being the number of subimages of the scene in one direction across the image sensor. The reduction in the focal length of the lenslet (keeping the *f* number the same) would keep the focal resolution unaffected with the compromise of reduction in angular resolution [5]. In the view of high resolution, recovering the lost angular resolution by collecting subimages can be achieved. The lenslet array accumulates a series of subimages from the same scene which are undersampled. Each subimage resolution is determined by pixel size rather than the optical performance of the lenslet, and these subimages can be processed in a manner similar to multiframe superresolution processing to obtain a fully upsampled, high-resolution image [6]. A number of TOMBO image restoration algorithms have been proposed and proven to be effective in the literature [7–12].

In this paper we focus on the optimization of the number of lenslets to achieve a thin system with high image restoration performance for the TOMBO system. For one TOMBO system with a fixed image sensor pixel number, a larger lenslet number leads to more subimages and fewer pixels for each subimage. More subimages only improve the resolution marginally if a special set of low-resolution pixels has been captured. On the other hand, subimages with fewer pixels demand a larger magnification factor, which leads to the performance deterioration of existing algorithms [13]. Therefore, optimizing the lenslet number is an important practical problem regarding the design of TOMBO. However, due to the specific architecture of TOMBO, to the best of our knowledge, this problem has not been adequately discussed.

In the past, the TOMBO system has been analyzed when image restoration is based on Moire magnification [2]. The performance limits of reconstruction-based superresolution algorithms and the fundamental requirements of low-resolution images with unlimited numbers have been discussed in great detail [13–15]. Unlike previous work, this paper bases the image restoration on multiframe superresolution algorithms which could increase the high-frequency components and remove the degradations caused by the imaging process. The number of low-resolution images is constrained by the pixel number of each low-resolution image in the TOMBO system, and this constraint determines the maximum magnification factor of superresolution algorithms and translation requirements between low-resolution images. Moreover, we focus on balancing the low-resolution image numbers and image restoration performance for practical TOMBO systems (more low-resolution images correspond to thinner system dimensions, and higher image restoration performance leads to angular resolution closer to that of the single-lens system).

In this paper, the optimization of the number of lenslets for the TOMBO system is performed by analyzing the influence of lenslet number on the image restoration performance. The imaging process (comprising image global translation, blur, downsampling, and noise) of each subimaging system with different lenslet numbers is presented. Then iterative backprojection (IBP) [8], bilateral total variance (BTV) [16], and l1 norm combined with simultaneous auto regressive (l1-SAR) [17] are applied to restore high-resolution images with actual translation parameters as well as translation parameters estimated by scale invariant feature transform [18] and random sample consensus [19] (SIFT-RANSAC), respectively. Mean square error (MSE) and peak signal-to-noise ratio (PSNR) of restored images are calculated for image quality assessment. The optimal lenslet number is obtained by analyzing the influence of lenslet number on the restored images with 100 synthetic and real images as input. Experimental comparison of angular resolutions among two TOMBO systems and a single-lens system is performed to validate the optimal lenslet number.

## 2. Performance simulation

Consider a TOMBO system with *n* × *n* subimaging units. Each captured subimage can be modeled as [12]

*L*represents the blurred, noisy, and low-resolution output image captured by subimaging system in the

_{i,j}*i*row of the

*j*column of TOMBO (

*i*,

*j*= 1, 2, …,

*n*);

*H*is the input high-resolution image;

*b*is a two-dimensional PSF representing the channel blur for each imaging unit;

_{i,j}*t*(

_{i,j}*r*) is a global translation shift operator which has

_{i,j}*r*= [Δ

_{i,j}*x*, Δ

_{i}*y*] translation with respect to the input image;

_{j}*↓D*is the downsampling operator; and

*v*is the noise, such as fixed-detector thermal noise, signal-dependent shot noise, and background noise [20].

_{i,j}In our simulation, a rotationally symmetric Gaussian low-pass filter of size [5] with standard deviation σ^{2} = 1 is used as channel blur *b _{i,j}*, variance σ

^{2}of two-dimensional zero mean white Gaussian noise is set to 2, and downsampling factor

*d*equals the lenslet number

*n*.

Because the sufficient number of low-resolution subimages is *M*^{2} when the translation between adjacent subimages is 1/*d* (*M* is an integer magnification factor of superresolution algorithms) [13], global translation for input image is set to *r _{i,j}* = [(

*i*– 1), (

*j*– 1)] (i.e., Δ

*x*=

_{i}*i*– 1 and Δ

*y*=

_{j}*j*– 1), and magnification factor

*M*equals the lenslet number

*n*in our simulation. For instance, translation between two low-resolution pixels is 1/3 in horizontal and vertical as shown in Fig. 2 when downsampling factor

*d*is 3. Take the system in [5]. as an example: the translation between adjacent subimages is manufactured as 70 μm, and this corresponds to 2 1/3 pixels of the sensor. Figure 3 shows simulated subimages with different lenslet numbers. All these subimages are resized to resolution of the input image with nearest interpolation.

Figure 4 presents the restored images using the l1-SAR algorithm with actual translation parameters. In the l1-SAR algorithm, the magnification factor *M* equals the lenslet number *n*, and the number of iterations is 20. Furthermore, for consideration of practical applications, restored images using the same algorithm with estimated translation parameters by SIFT-RANSAC are shown in Fig. 5. Matched pixel pairs between any two subimages are generated by SIFT, and then RANSAC is used to remove incorrect matches.

Restored images with 100 synthetic and real images as input are tested for assessment of the influence of the lenslet number. IBP, BTV, and l1-SAR are utilized as restored algorithms. Moreover, MSE and PSNR of restored images are averaged for quantitative evaluation of the influence. PSNR is defined as

where MAX is the maximum pixel intensity of an input image and MSE is the mean square error between the input image and the restored image. For the sake of simplicity, temporal noise is not taken into account in Eq. (2).The influence of the lenslet number on the averaged image quality (MSE and PSNR) of restored images is present in Figs. 6, 7, 8, and 9. As shown in these figures, l1-SAR has higher restoration performance than BTV, whereas BTV is better than IBP. Moreover, under the actual translation conditions, the restoration performance remains almost unchanged with the increasing lenslet number. On the other hand, under the estimated translation conditions the performance deteriorates with the increasing lenslet number. The registration error is a key factor that leads to the deterioration of restoration performances.

Figures 10 and 11 present the averaged horizontal and vertical registration errors estimated using maximum likelihood [7] and SIFT-RANSAC. As shown in these two figures, in general SIFT-RANSAC has smaller registration errors than maximum likelihood. However, the registration errors of both registration algorithms increase as the lenslet number increases. On the contrary, in the image restoration procedure, as actual translation parameter is equal to 1/*n*, higher registration accuracy is required as the lenslet number increases. Under this condition, larger registration errors lead to restoration performance deterioration.

As a larger lenslet number leads to thinner system dimensions, and the simulation results show that MSE or PNSR with 4 × 4 lenslets is almost the same as that in the single-lens system, the optimal lenslet number for TOMBO system is determined to be 4 × 4.

## 3. Experiment

In this section, the optimal lenslet number for TOMBO systems is verified by experimental comparison of angular resolutions among a 4 × 4 lenslet TOMBO system, a 5 × 5 lenslet TOMBO system, and a conventional single-lens system using the same monochrome image sensor. The sensor size is 10.9 mm × 10.9 mm with 1024 × 1024 pixels. For the 4 × 4 lenslet TOMBO system, lenses are aligned with a pitch of 2.6 mm. The focal length *f _{l}* and the diameter of each lens are 20 mm and 2.6 mm, respectively. For the 5 × 5 lenslet TOMBO system, lenses are aligned with a pitch of 2 mm. The focal length

*f*and the diameter of each lens are 16 mm and 2 mm, respectively. For the single-lens system, we utilize Nikon Nikkor lens whose focal lens

_{l}*f*is 85 mm and

_{s}*f*number ranges from 1.8 to 16.

The angular resolutions of the three systems with approximately the same *f* number ( = 8) are measured by a collimator. In the collimator, an illuminated USAF 1951 resolution target is positioned at the front focal plane of the objective lens as shown in Fig. 12. With this configuration, all light beams passing a point in the resolution target plane form a collimated light bundle behind the objective lens. The focal length *f _{c}* and clear aperture of the collimator are 1000 mm and 100 mm, respectively. Figure 13 presents a photo of the experimental setup for a TOMBO system.

The data captured by the 4 × 4 and 5 × 5 lenslet TOMBO systems are shown in Fig. 14. Figure 15 shows the restored images and the image captured by the single-lens system. The restored images are acquired by SIFT-RANSAC and l1-SAR with magnification factors of 4 for Fig. 15(a) and 5 for Fig. 15(b). In contrast to that in Fig. 15(a), the images shown in Fig. 15(b) suffer deterioration of restoration performance as demonstrated in the simulation. Compared with Fig. 15(c), the contrast between white bar and black section is lower in Fig. 15(a), but according to Eq. (3) the angular resolutions of these two images are the same.

where*b*is the minimal interval of white bars that can be distinguished,

*f*is the focal length of the collimator, and the value 206,265 is the arcseconds for one radian. As shown in Figs. 15(a) and 15(c),

_{c}*b*in yellow squares corresponds to the resolution target’s group 1 element 3, which indicates 2.52 line pairs/mm. So the angular resolution for the 4 × 4 lenslet TOMBO system and the single-lens system is 81.85″, whereas the 5 × 5 lenslet TOMBO system has a poorer angular resolution performance. These results confirm that 4 × 4 is the optimal lenslet number for TOMBO systems, as the simulations showed.

## 4. Conclusion and future work

The subimage capturing processes of the TOMBO imaging system with different lenslet numbers have been simulated, and restoration algorithms have been applied to restore high-resolution images. MAE and PSNR of the restored images have been calculated for quantitative image quality assessment to optimize the lenslet number. The optimal lenslet number has been determined to be 4 × 4 by the simulations and confirmed by imaging experiments with 4 × 4 and 5 × 5 TOMBO imaging systems and a single-lens imaging system.

It is worth mentioning that the optimal lenslet number discussed in this paper is based on the applications in which imaging resolution and system thinness are of primary importance. However, for cases in which other performance parameters, such as sensitivity, are of primary concern, related analyses could still be explored by employing other performance metric functions (such as the capacity metric proposed in [20].) with parameters such as sensitivity, object scene field of view, and resolution.

Future studies for an observation model of the TOMBO imaging system and more accurate registration between subimages should be performed. An ideal imaging process was assumed in the observation models introduced in Section 2. The consideration of the optical aberration in a real imaging system could improve the model. Furthermore, larger optimal lenslet numbers could be expected for thinner system dimensions with more accurate registration.

## Acknowledgments

This work was supported by the Preeminent Youth Fund of Sichuan Province under grant 2012JQ0012 and the National Natural Science Foundation of China under grant 11173008.

## References and links

**1. **J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T. Morimoto, N. Kondou, D. Miyazaki, and Y. Ichioka, “Thin observation module by bound optics (TOMBO): concept and experimental verification,” Appl. Opt. **40**(11), 1806–1813 (2001). [CrossRef] [PubMed]

**2. **J. W. Duparré and F. C. Wippermann, “Micro-optical artificial compound eyes,” Bioinspir. Biomim. **1**(1), R1–R16 (2006). [CrossRef] [PubMed]

**3. **D. Mendlovic, “Toward a super imaging system,” Appl. Opt. **52**(4), 561–566 (2013). [CrossRef] [PubMed]

**4. **K. Choi and T. J. Schulz, “Signal-processing approaches for image-resolution restoration for TOMBO imagery,” Appl. Opt. **47**(10), B104–B116 (2008). [CrossRef] [PubMed]

**5. **M. Shankar, R. Willett, N. Pitsianis, T. Schulz, R. Gibbons, R. Te Kolste, J. Carriere, C. Chen, D. Prather, and D. Brady, “Thin infrared imaging systems through multichannel sampling,” Appl. Opt. **47**(10), B1–B10 (2008). [CrossRef] [PubMed]

**6. **A. V. Kanaev, D. A. Scribner, J. R. Ackerman, and E. F. Fleet, “Analysis and application of multiframe superresolution processing for conventional imaging systems and lenslet arrays,” Appl. Opt. **46**(20), 4320–4328 (2007). [CrossRef] [PubMed]

**7. **Y. Kitamura, R. Shogenji, K. Yamada, S. Miyatake, M. Miyamoto, T. Morimoto, Y. Masaki, N. Kondou, D. Miyazaki, J. Tanida, and Y. Ichioka, “Reconstruction of a high-resolution image on a compound-eye image-capturing system,” Appl. Opt. **43**(8), 1719–1727 (2004). [CrossRef] [PubMed]

**8. **A. Stern and B. Javidi, “Three-dimensional image sensing and reconstruction with time-division multiplexed computational integral imaging,” Appl. Opt. **42**(35), 7036–7042 (2003). [CrossRef] [PubMed]

**9. **R. Horisaki, S. Irie, Y. Ogura, and J. Tanida, “Three-dimensional information acqusition using a compound imaging system,” Opt. Rev. **14**(5), 347–350 (2007). [CrossRef]

**10. **A. V. Kanaev, J. R. Ackerman, E. F. Fleet, and D. A. Scribner, “TOMBO sensor with scene-independent superresolution processing,” Opt. Lett. **32**(19), 2855–2857 (2007). [CrossRef] [PubMed]

**11. **A. A. El-Sallam and F. Boussaid, “Spectral-based blind image restoration method for thin TOMBO imagers,” Sensors (Basel Switzerland) **8**(9), 6108–6124 (2008). [CrossRef]

**12. **S. Mendelowitz, I. Klapp, and D. Mendlovic, “Design of an image restoration algorithm for the TOMBO imaging system,” J. Opt. Soc. Am. A **30**(6), 1193–1204 (2013). [CrossRef] [PubMed]

**13. **Z. Lin and H. Y. Shum, “Fundamental limits of reconstruction-based superresolution algorithms under local translation,” IEEE Trans. Pattern Anal. Mach. Intell. **26**(1), 83–97 (2004). [CrossRef] [PubMed]

**14. **S. Baker and T. Kanade, “Limits on super-resolution and how to break them,” IEEE Trans. Pattern Anal. Mach. Intell. **24**(9), 1167–1183 (2002). [CrossRef]

**15. **D. Robinson and P. Milanfar, “Statistical performance analysis of super-resolution,” IEEE Trans. Image Process. **15**(6), 1413–1428 (2006). [CrossRef] [PubMed]

**16. **S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” IEEE Trans. Image Process. **13**(10), 1327–1344 (2004). [CrossRef] [PubMed]

**17. **S. Villena, M. Vega, S. D. Babaccan, R. Molina, and A. K. Katsaggelos, “Bayesian combination of sparse and non-sparse priors in image super resolution,” Digit. Signal Process. **23**(2), 530–541 (2013). [CrossRef]

**18. **D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. **60**(2), 91–110 (2004). [CrossRef]

**19. **M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Commun. ACM **24**(6), 381–395 (1981). [CrossRef]

**20. **M. W. Haney, “Performance scaling in flat imagers,” Appl. Opt. **45**(13), 2901–2910 (2006). [CrossRef] [PubMed]