1. Introduction

Compressive imaging [1] aims to optically compress images using fewer measurements, or measurement elements, than what is required to traditionally sample the images at a given resolution using a detector array. Inspired by compressed sensing theory [2], compressive imaging is possible if two conditions are met. Firstly, the image content needs to exist in a subspace of a noticeably smaller dimensionality in contrast to the full sampling dimension–that is–the image is natively sparse or it is sparse when projected into an adequate sparsifying basis or domain (e.g. Wavelet, Fourier, Gradient) [3]. Secondly, every compressed measurement should be a linear combination of hopefully independent projections of the whole scene–as incoherent as possible not only between them, but also to the sparsifying basis–which can be achieved by a random sampling scheme, for example.

Fortunately the first condition is easily satisfied, since the majority of what are called natural images, that are most likely dense in the image domain, are at least compressible (near sparse) in the Fourier/Wavelet domain–where most of the energy is concentrated into a few larger coefficients. This is what makes image compression possible (JPEG, JPEG2K). On the other hand, and although the second condition is not trivial, the existence of spatial light modulators such as the Digital Micromirror Device (DMD) make it possible to experimentally demonstrate what is known as the single-pixel camera (SPC) [4], where each measurement is a randomly coded and multiplexed projection of the whole scene integrated onto a bucket detector. Thus, and by using optimization algorithms based on sparsity constraints, images of N × N pixels can be reconstructed from a sequence of M << N² measurements.

In order to overcome the sequential acquisition process, the challenge for developing snapshot compressive imaging is the extension of the single-pixel camera idea to generate simultaneous optical projections that can still fulfill the compressed sensing reconstruction requirements. Unfortunately, this leads to unpractical architectures [5], such as an array of SPCs that can take all measurements at once. Nonetheless, detector arrays are available in many imaging modalities, bringing new possibilities such as it is proposed in [6, 7], where the DMD is spatially shared and splits the image FoV, reducing the amount of sequential measurements in proportion to the amount of detector elements (pixels) available. This system acts as a different array of SPCs that are only responsible of coding partial parts of the scene, but the reconstruction is done jointly.

On the other hand, an effective approach for single-shot compressive imaging has been the use of coded apertures [8] in conjunction with low resolution detector arrays. Both aperture and phase masks allow for the point spread function (PSF) engineering of the imaging system. The joint optical and detector PSF produce both coding and multiplexing of partial but different parts of the scene over every one of the pixels of the detector. However, and in contrast to the SPC, the projections are not arbitrary and may not comply with the incoherence requirements for successful reconstruction. Nonetheless, some theoretical relaxations have shown that convolution based sensing matrices can also work [9]. To enhance incoherence, in [10] a dual phase encoding system based on optical encription is proposed, while a mixed aperture/phase mask was successfully demonstrated as a compressive target tracking system in [11]. Lately, in [12] they propose a novel phase mask design to implement the concept of random convolution for compressive imaging.

One of the goals of compressive imaging is to make better usage of the measurement resources available [13]. However, as any computational imaging system, the benefit comes at the expense of calibration and computational resources, or even further, the availability of spatial modulators. Therefore, if we are in the presence of a costly imaging system with a low resolution detector in a sensing problem where not only the spatial but also the temporal sampling is an issue, then the usage of snapshot compressive imaging would be beneficial, especially if we are able to perform the needed modulation for coding and multiplexing the incoming field with minimal modifications to the imaging system. With this in mind, in this paper we want to study the effect of a simple way to provide phase modulation: aberrations. Although aberrations are traditionally a nuisance for imaging systems design, they have shown potential for computational imaging applications such as to extend the depth-of-field [14] and to reduce complexity in imaging systems [15]. As aberrations can be inherent to any imaging system or can actually be designed as phase masks to be introduced at the pupil plane, we want to understand whether aberrated low resolution, detector limited imaging systems can become effective snapshot compressive imaging devices. In this exploration, we add controlled aberrations to a detector limited imaging system–i.e. optical PSF smaller than detector PSF–and then evaluate the ability of each aberrated system to reconstruct a higher resolution image from a low resolution snapshot using sparse reconstruction algorithms.

2. Modeling and simulation

Starting with a clear aperture, the different imaging systems are represented by phase masks containing the different aberrations of interest. Assuming incoherent imaging, the corresponding optical PSF at the center of the field of the image plane is computed by squaring the Fourier transform of the complex pupil function. For the simulations of the forward model of the compressive imaging system, we assume that the incoming optical image x that we want to sample is discretized at a grid of N × N pixels, which is projected onto a low resolution detector array of M × M leading to the compressed image y. By assuming an isotropic behavior (spatial invariance), we can obtain the corresponding M² × N² (where M < N) compression system matrix H by projecting the optical PSF–sampled at the original image resolution–onto the low resolution detector for every possible pixel in the high resolution grid. Note that the system matrix combines the joint effects of the optical and detector PSFs. Therefore, the system matrix can be decomposed into a N² × N² optical PSF matrix H_O (a convolution matrix) and a M² × N² detector PSF matrix H_D (a downsampling matrix), leading to

The optical PSF matrix H_O is responsible of the coding while the detector PSF matrix H_D actually performs the compression by downsampling.

For the actual simulations, we use input images of 128 × 128 pixels and output images of 32 × 32 pixels using system matrices of 1024 × 16384, leading to a 16× compression ratio. We generated sampling matrices for the clear aperture, unaberrated, diffraction-limited system (Ab×0) and for several types of aberrations that correspond to every one of the first Zernike modes from z4 to z15 at different strengths (Ab×1, Ab×2 and Ab×3).

We use the Zernike two-dimensonal polynomials since they form an orthonormal basis in the unit circle, and are vastly used to characterize optical aberrations. In the nomenclature we adopted, z4 means defocus, z5 and z6 are primary astigmatisms, z7 and z8 are X and Y coma, z9 and z10 are X and Y trefoil, z11 is spherical aberration, z12 and z13 are secondary astigmatisms, and z14 and z15 are the X and Y tetrafoil. The strength, or weight, of the aberration is related to the maximum deflection of the phase element in proportion to the wavelength used. Therefore, the Ab×1, Ab×2 and Ab×3 aberrated systems are related to 1, 2 or 3 waves of maximum deviation for any of the particular Zernike modes of choice. In the particular case of our simulations, the different strengths may lead to PSFs that span from 5 to 15 pixels in diameter in the low resolution space, and 2 pixels in the unaberrated case.

In Fig. 1(a) we can observe different phase masks, which correspond to a clear aperture followed by different strengths of the secondary astigmatism z13. Then, in Fig. 1(b) we can see the corresponding PSF for each phase profile, which are used to finally generate the associated compressive systems matrices H, as seen in Fig. 1(c).

 

Fig. 1 Example of aberrations, its associated PSF and system matrix for the unaberrated system Ab×0 and the aberrated systems Ab×1, Ab×2 and Ab×3 with different degrees of secondary astigmatism z13. (a) phase profile; (b) PSF; (c) 16× compressive system matrix.

Download Full Size | PDF

Note that the unaberrated system delivers a diffraction limited PSF (barely noticeable) that leads to a system matrix that resembles almost a pure downsampling matrix. As the aberration gets stronger, the PSF gets broader and the system matrix becomes more populated. The role of the different optical PSFs–even the diffraction limited one–is to provide with coding to the system matrix so it transitions from a pure downsampling matrix. A pure downsampling (detector PSF) matrix H = H_D will have two distinctive characteristics. On one hand, there are several columns that are highly correlated, which correspond to the pixel positions in the high resolution space that are being averaged in the low resolution measurement. On the other hand, the columns that correspond to any other pixel than the ones involved in the downsampling process for a particular measurement are completely uncorrelated. When adding aberrations, the PSF is locally multiplexing and coding the information of the high resolution space into the low resolution measurements. As such, different PSFs will provide with more or less distinctive local columns to a particular compressive system matrix depending on its structure and size, playing a tradeoff of better local incoherence in contrast to more or less correlation with measurements that are near or even beyond the size of the PSF kernel.

Once the system matrices are computed, the compressed image y, in lexicographical order, can be obtained by applying the linear model

As input images to our simulations we use a set of 15 standard natural images taken from the USC-SIPI image database, resampled at 128 × 128 pixels. We also synthesized a natively sparse image (“Stars”’) that has 250 sparse objects, which is at the limit of the Donoho-Tanner phase transition for this compression ratio [16]. Note that if a signal is K-sparse, it can be reconstructed from the set of M > K log(N/K) measurements with high probability if the sensing matrix satisfies the restricted isometry property, which is vastly the case for a variety of random matrices.

Assuming that conditions for compressed sensing are met, reconstructions for the collection of natural images are obtained by solving the following total variation (TV) convex minimization problem:

We present an example of the simulated compressive acquisition and reconstruction process using the source image in Fig 2(a). Then, we show the 16× compressive measurements obtained for the unaberrated system Ab×0 in Fig. 2(c) and when using the z13 aberration with a strength of Ab×3 in Fig. 2(d). Also, we present a plain downsampled version of the source image in Fig. 2(b) for comparison purposes. Note that both the downsampled and unaberrated compressed version are very similar, but not equal. This is due the fact that the convolution PSF matrix H_O is not necessarily an identity matrix, as is the theoretical case for the downsampling. Note that the compressed measurement using aberrations is a blurry and almost unidentifiable version of the source image. The reconstruction results from the the compressed measurements are shown in Fig. 2(f) and Fig. 2(g), whereas a bicubic interpolated version of the downsampled image is presented in Fig. 2(e). Note that there is a clear gain in signal to noise when using sparse reconstruction and having knowledge of the PSF in the case of the unaberrated system, which can be further increased with the introduction of aberrations.

 

Fig. 2 Compressive acquisition and reconstruction examples. (a) 128 × 128 original image; (b) 32 × 32 downsampled version; (c) 32 × 32 compressed version with the Ab×0 system matrix; (d) 32 × 32 compressed version with the z13 Ab×3 system matrix; (e) 128 × 128 bicubic interpolation of (b); (f) 128 × 128 sparse reconstruction (TV) of (f); (g) 128 × 128 sparse reconstruction (TV) of (g).

Download Full Size | PDF

In Fig. 3 we present the average reconstruction error, quantified in terms of the peak signal-to-noise ratio (PSNR) defined as

 

Fig. 3 Average performance in PSNR of the reconstructed images from compressive measurements using the unaberrated system Ab×0 and aberrated systems with Zernike modes from z4 to z15 at different strength levels Ab×1, Ab×2 and Ab×3.

Download Full Size | PDF

The reconstruction errors in terms of PSNR for all images and aberration types are then averaged for each of the strengths from Ab×0 to Ab×3 and summarized in Table 1, together with the results for equivalent SPC systems implemented with: binary coding, SPC (1/0); uniform distribution grayscale coding, SPC(u); or a theoretical Gaussian distribution coding, SPC(g). As suggested by the best performing aberrations in the plot of Fig.3, we can note that the performance of the aberrated systems improve as more strength is applied. Also, the average error obtained when using any aberration, on average, is always lower than the unaberrated system. Surprisingly, even the unaberrated system Ab×0 outperforms any of the SPC. In addition, all SPC based systems reach very similar performances no matter what coding distribution is used.

Table 1. Average reconstruction PSNR for 16× simulated compressive imaging systems. The average is performed over all images and Zernike modes for a particular strength.

View Table | View all tables in this article

We have to mention that we also simulated with even more aberration strength, but the gain in performance starts to be marginal. In addition, and for practical reasons, broad PSFs may present difficulties in signal-to-noise due to light efficiency since most of the energy will be spread outside the detector area. In Fig. 4 we show reconstruction samples for five of the images in the test set obtained with the secondary astigmatism z13 systems at different aberration strengths shown in Fig. 1. We should note that the performance is always image dependent. For the natural images, we can note that more details and textures starts to appear as more strength is applied, especially in contrast to the Ab×0 system, which is in agreement with the quantitative results reported in Fig. 3. A similar analysis can be made for the sparse “Stars” image, where point objects start to become better resolved and positioned as more strength is applied. Also, even a bit of aberration, such as Ab×1, noticeably improves over the unaberrated reconstructed image with Ab×0.

 

Fig. 4 Reconstruction samples for the 16× simulated snapshot compressive imaging systems using the secondary astigmatism z13 aberration at different strengths Ab×0, Ab×1, Ab×2 and Ab×3.

Download Full Size | PDF

There is a good amount of research devoted to the design of optimal sensing matrices for compressed sensing. However, we are in a different situation where we can design or synthesize a coded aperture or phase mask, but we do not have arbitrary control to deterministically design the sensing matrix. Still, since evaluating the restricted isometry property is unpractical, people often evaluate the ability of a sensing matrix to perform as a compressed sensing matrix by the analysis of the mutual coherence parameter [18], defined as follows:

We computed μ for our sensing matrices, and the results are displayed in Fig. 5. A first observation is that all of the aberrated systems present values near the unity, which is the worse value possible as obtained by the unaberrated system Ab×0. Nevertheless, the fact of not being exactly one seems as a good indication of better performance since some of the systems outperform the Ab×0 system in the quantitative PSNR analysis. Although there is not a perfect correlation, most of the relatively minimum coherence values found are associated with some of the best performing aberrated systems such as both primary and secondary astigmatism and defocus. However, if we analyze z4, z5 and z6, we see that the coherence value decreases as the strength increases, which is the opposite to what was seen with the PSNR trend. On the other hand, the secondary astigmatism z13 achieves it lowest coherence values when increasing the aberration strength.

 

Fig. 5 Average mutual coherence of the unaberrated system Ab×0 and aberrated systems with Zernike modes from z4 to z15 at different strength levels Ab×1, Ab×2 and Ab×3.

Download Full Size | PDF

The mutual coherence results that are averaged by strength of the aberrations are summarized in Table 2. As seen in the plot of Fig. 5, we do not see the same overall trend as with the PSNR, and therefore on average the coherence seems to be stable with aberration strength. However, we can state two things for sure for a particular selection of aberration strength: if we use the system with the largest coherence, its performance is certainly no the best, but if use the system with the lowest coherence, this system will most likely provide with one of the best recovery performances.

Table 2. Average mutual coherence for simulated 16× compressive imaging systems.

View Table | View all tables in this article

Nonetheless, if the comparison is now made with the coherence obtained by the SPC, the story is even more confusing since the SPC have smaller coherence but worse performance. In addition, if we compare the different variations of the SPC when using binary (1/0) or uniform grayscale amplitude modulation (u), there is a clear difference although their reconstruction performance from Table 1 is basically the same. For the same performance in theoretical reconstruction, the difference in μ is even more dramatic in contrast with a theoretical SPC that would implement Gaussian random sampling matrix (g), where the mutual coherence approaches the Welch bound, which for this particular system matrix geometry is μ = 0.0303.

From the compressed sensing theory, a coherence value so close to 1 may indicate that a system should not work at all as a compressive imaging system–which may be the case for most compressive imaging systems based on some sort of convolution. However, and given our results, we believe further research should be done to understand the trade-off between the size of the convolution kernel and the image statistics of the imaging problem of interest, and how this can be related to a sort of local mutual coherence metric instead of the traditional mutual coherence. Another interesting aspect to consider is the effect on the coherence of having a strictly positive sensing matrix. As seen in the case of the SPC when implements a binary or grayscale sampling scheme, the coherence values change dramatically in comparison with the theoretical Gaussian sampling, for almost the same performance.

3. Experiment

To verify the fact that aberrations may boost the acquisition and recovery of more details than a normal camera system–that is designed to be a faithful imager–can do, we have devised the following optical setup. As seen in the schematic presented in Fig. 6, the experimental setup mainly consists of a main objective lens that produce an image of the object, and subsequent 4f system that has a phase modulating element at 2f, and that relays the modified image plane at a detector array. As the imaging target source at the object plane we located a 128 × 128 OLED screen (OLED-128-G2, 4D Systems), which was used for calibration and performing the imaging experiments. The intermediate image plane is formed by a main 16mm objective lens (Computar M1614-MP2) which is relayed by a pair of 75mm Fourier transforming lenses (Thorlabs AC254-075-A-ML). With the help of a beamsplitter (BS, Thorlabs CCM1-BS013), we placed a deformable mirror (DM, Thorlabs DMP40-P01) at the pupil plane at a distance of 2f = 150mm from the intermediate image plane. A Full-HD CMOS camera (Pointgrey GS3-U3-23S6M-C) sampled the final image plane at a distance of 2f = 150mm from the deformable mirror.

 

Fig. 6 Schematic of the proposed snapshot compressive imaging system. (Inset) Picture of the experimental setup.

Download Full Size | PDF

The inclusion of a deformable mirror into the setup allowed to add any desired aberration to the optical system without touching it. After selecting a particular aberration within the first Zernike moments from z4 to z15 available by the existing deformable mirror, or even if we would like to apply any linear combination of Zernike modes to end up with random aberrations, we need to perform a calibration procedure. For this case, calibration means obtaining the experimental system matrix H. This can be accomplished by capturing the response of the imaging system, or the PSF, for every pixel source of the high resolution target. Every image captured at the compressed resolution at the different input pixel positions is stacked in lexicographical order as a column until fulfilling the system matrix. In this particular situation where we want to compress source images of 128 × 128, we need to perform 16384 calibration measurements at the desired output resolution of our detector, which is at 32 × 32 to achieve the 16× compression ratio used in the simulations. However, as or most aberrations we never obtained a PSF that goes beyond 6 pixels of diameter at the low resolution image detector when setting the Zernike at the maximum deformable mirror stroke, we spatially multiplexed the acquisition of the PSFs illuminating a 4 × 4 array of point objects at the screen. Since there is not overlap between the different PSFs, we were able to retrieve each corresponding system response projection from the multiplexed image by cropping each corresponding piece. This is repeated until completing the whole system matrix, which implied a 16× reduction in the calibration time. Once the calibration was finish, we projected all the same set of test images used in our simulations.

Every image was background subtracted, and a median filter was applied to remove hot pixels. Then, since the images of the OLED screen were being projected at an approximately 600 × 600 pixel area, we performed digital subsampling to reach the desired resolution of 32 × 32 as our compressive imaging measurements. We performed the calibration and compressive imaging acquisition for systems with no aberrations and any of the Zernike modes of interest at an unique strength given the maximum stroke of the deformable mirror, which we denominated Ab×1. From the rendered reconstructions, we noticed a similar trend in terms of qualitative results in comparison with the simulations. In our experiment, astigmatisms were again more favorable for better and richer image reconstructions. Sample images comparing the reconstruction results for the unaberrated system, primary astigmatism z6 and secondary astigmatism z13 are shown in Fig. 7.

 

Fig. 7 Comparison of reconstruction results from 16× experimental compressive imaging measurements from the Lena, Peppers, House, Man and Stars images. (Left to right) the original high resolution image is compared with reconstructions from systems with no aberrations, primary astigmatism z6 and secondary astigmatism z13.

Download Full Size | PDF

At a first sight, the results from the unaberrated system Ab×0 seem cleaner. The aberrated systems presented some sort of ringing artifacts, which are most likely due to problems with the system matrix calibration, which may contain noise contributions that do not belong to the system itself. This is a common issue of compressive imaging systems as stated in [19]. For example, pixel sources used during calibration are not real point sources, and display fluctuations, detector readout and quantization noise can contribute noisy PSF estimates for every incoming position.

From the reconstructed images we can note that all of them present more details and textures when adding aberrations, especially with the secondary astigmatism. This can be further verified when inspecting the zoomed-in version of two of the images shown in Fig. 8. In the first image, which contains the window of the house image, it is clear how the unaberrated image was unable to resolve the window partition, while the other two versions succeeded. When checking the sparse image, the difference is more noticeable since the point sources are better resolved in both astigmatic cases. It is important to note that no aberrated system is necessarily superior to the other everywhere and in all the images, since this is very image content specific. As an additional insight, note that even the unaberrated system present better resolving point sources in the sparse image in contrast to what was originally expected from the simulations. This is due the fact that even this system is not really free from aberrations. Although they are very minor, they are still able to contribute towards embedding a bit more of the high resolution image information into the compressed measurements.

 

Fig. 8 Zoomed-in versions of the highlighted patches of the reconstructed House and Stars images in Fig. 7: (Left to Right) original, unaberrated, primary astigmatism z6, and secondary astigmatism z13.

Download Full Size | PDF

4. Conclusion

In summary, we proposed an architecture for single-shot compressive imaging by introducing aberrations at the pupil plane of a low resolution digital imaging system. In contrast to the single-pixel camera, and similarly to other coded-aperture approaches, aberrated PSFs are able to simultaneously code and multiplex different parts of the scene if projected onto an array detector. A qualitative analysis from the reconstructions results obtained from our 16× compressive imaging systems reveal that more details and textures can be recovered when larger aberrations are introduced. In addition, a quantitative analysis based on reconstruction error indicates that some aberrations effectively decrease the average error for a set of test natural and sparse images in contrast to unaberrated measurements, as is the case of the use of primary and secondary astigmatisms. Even for the unaberrated system, reconstruction results are superior to what can be accomplished by an equivalent SPC using random binary projections. This is explained due to the fact that the recovery from largely multiplexed measurements are harder to disambiguate, although random measurements are able to resolve better details in cases where the information content is really sparse and it is not evenly spatially distributed. We also analyzed the mutual coherence of the compressive system matrices, and we surprisingly realized that even though our aberrated systems present a very high level of coherence near the maximum value of one, they still provide with even better results than the SPC, which has a smaller coherence. Nonetheless, we were able to find some degree of correlation between the mutual coherence parameter and the performance, where astigmatic systems often presented the lowest coherence among aberrated systems and the best performance overall. As the compressive imaging system based on aberrations encodes only partial parts of the scene per measurement, it acts–and also has a similar system matrix structure–as the parallel approach that shares the DMD in [6,7]. However, the gain on simultaneity of the measurement process has the drawback of not being able to design and perform arbitrary projections that could potentially have lower coherence. On the other hand, its local compressive behavior may suit better the compression of natural images that are often not sparse, but whose details are spatially spread.

We have also corroborated the potential for aberrations to convey more informative compressive imaging measurements with the development of an experimental setup that was able to introduce and test different aberrations using a deformable mirror. Although the reconstructions presented artifacts for the aberrated systems, the experimental results confirmed the findings of the simulations where systems with either primary and secondary astigmatism are able to recover more details and textures in the case of natural scenes, and also have more accuracy in the recovery of sparse objects in sparse images. Therefore, and contrary to the idea that aberrations jeopardize the image quality, in this case they are successfully used in benefit of optical image compression. We are currently working on improving calibration procedures and also developing a better metric in order to predict the performance of compressive imaging systems.