Active computational imaging for circumventing resolution limits at macroscopic scales

Prasanna Rangarajan; Indranil Sinharoy; Predrag Milojkovic; Marc P. Christensen

doi:10.1364/AO.56.000D84

1. INTRODUCTION

Since the invention of the microscope and telescope, engineers and scientists alike have embarked on a perpetual quest to improve the resolving power of imaging instruments. Ernst Abbe was the first to recognize that the resolving power of an imaging instrument is fundamentally limited by diffraction. Following Abbe’s seminal work in 1873, the scientific community has made numerous attempts to circumvent the diffraction limit. The most significant breakthroughs came in the form of aperture synthesis and super-resolution microscopy, each of which has revolutionized imaging at the astronomical and microscopic scales, respectively.

Despite breakthroughs, there exists a continuum of image scales between the microscopic and the astronomical, where diffraction continues to limit the resolving power of imaging instruments. Existing approaches to improve the resolution of macroscopic imagers are centered on the notion of increasing the numerical aperture while concomitantly shrinking the pixel size. The former serves to improve the optical resolution by means of the diffraction limit, while the latter serves to improve the sensor resolution by means of the Nyquist limit. Attempts to break away from the endless cycle of creating better optics and smaller pixels are fraught with problems. These range from the poor light throughput of smaller pixels to the need for expertly designed optics featuring sophisticated aberration correction.

Recent innovations, such as compressed sensing [1–3], permit us to break away from the Nyquist sampling limit, but diffraction remains an unsurpassed barrier. The resulting loss of spatial detail is evidenced in the image of Fig. 1 wherein the imager is unable to resolve the fine spatial detail in the inner segments of the spoke target.

Fig. 1. Illustrating the limits of macroscopic imaging.

Download Full Size | PDF

A third limit that affects imaging at macroscopic scales stems from the loss of dimensionality (3D world to 2D image) intrinsic to the geometry of image formation. It manifests as a loss of absolute size and shape information (perspective foreshortening), as evidenced in the image size of the chess pieces and the warped outline of the chessboard.

A. Concept Imager

The suite of methods outlined in this work afford macroscopic imagers the unique ability to circumvent the diffraction limit while also recovering depth and range information. Such disparate capabilities are realized by spatially patterning the light incident on the object volume prior to image acquisition. Subsequent processing of the images acquired under patterned illumination provides images with enhanced transverse and longitudinal resolution.

It is worth emphasizing that the idea of imaging under patterned illumination is not unique to the presented work. The microscopy community has exploited illumination diversity to circumvent the diffraction limit. Likewise, stereoscopic techniques in computer vision rely on illumination diversity to recover depth. An assimilation of these ideas into a broader framework for active computational imaging at macroscopic scales is one of the original contributions of this work.

Figure 2 provides a conceptual illustration of the proposed imager concept. It extends the notion of computational imaging to accommodate illumination diversity, thereby attempting to re-distribute the burden of imaging across the imaging, illumination, sensing, and reconstruction components. The aforementioned notion contrasts the current practice of recording high fidelity imagery using expertly designed optics and flood illumination.

Fig. 2. Active computational imaging concept.

Download Full Size | PDF

The imager concept introduced above is a precursor to a new class of computational imagers dubbed “active computational imagers.” These imagers afford capabilities that are difficult to achieve using classical approaches. Examples include building computational imagers whose resolving power is fully decoupled from the constraints imposed by the collection optics and the sensor. Other features include support for multiscale reconstruction and ranging in a variety of monostatic and bistatic arrangements, and task specific engineering of the imager PSF [4].

The combined ability to range and super-resolve, while accommodating aberrations and aliasing, and support for multiscale reconstruction are capabilities that are unique to this work.

B. Principle

The dual objectives of surpassing the diffraction limit and recovering depth information are realized by processing images acquired under spatially patterned illumination. The Moiré fringes arising from the heterodyning of object detail and patterned illumination encapsulate spatial frequencies lost to optical blurring. The deformations in the phase of the detected illumination pattern arising from parallax encode depth information.

1. Surpassing the Diffraction Limit by Heterodyning

The relevance of heterodyning to super-resolution is not apparent in the insets of Fig. 3. The notion is explored further in the example of Fig. 4, wherein the imager is unable to resolve fine spatial detail in the target due to the limited bandwidth of the imaging optics. Illuminating the target with a sinusoidal pattern produces sum and difference frequency patterns (Moiré fringes), the latter of which survives optical blurring. Knowledge of the frequency of the illumination pattern could be used to demodulate the fringes, yielding spatial frequencies beyond the optical cutoff. The demodulated image, when combined with the diffraction-limited image of the target, yields a super-resolved image whose bandwidth exceeds the diffraction limit. Closer examination of Fig. 4 indicates that the target is modulated by multiple patterns that are phase-shifted replicas of a master pattern. The modulation diversity afforded by phase-shifting is used to restore the heterodyned spatial frequencies in an unambiguous manner.

Fig. 3. Mechanism underlying dual objectives surpassing the diffraction limit and recovering depth information.

Download Full Size | PDF

Fig. 4. Surpassing the diffraction limit by heterodyning.

Download Full Size | PDF

Lukosz and Marchand [5] were the first to recognize the potential use of Moiré fringes in extending the spatial frequency bandwidth of an imaging system. Their insight has spawned an extensive body of work [6–24] comprising active and passive techniques for super-resolution of extended objects. These techniques, however, restrict attention to improving the resolution of well-corrected optics characterized by space-invariant blur. The assumption of space invariance contrasts the space variance observed in practice (on account of defocus and/or aberrations). This work relaxes the space invariance requirement and establishes that linearity of the imaging process is the only requirement for super-resolution using heterodyning.

2. Depth Estimation Using Parallax

The depth map in Fig. 5 was obtained by decoding the phase deformations in the recorded illumination pattern. The deformations stem from the viewpoint difference between the illumination source and the imager. This fact is routinely exploited in laser stripe scanning [25–28], Structured light scanning [29–32], and phase measurement profilometry [33–38].

Fig. 5. Recovering depth using phase modulation.

Download Full Size | PDF

In general, the image of a scene acquired under patterned illumination exhibits both amplitude and phase modulation. The former is useful in recovering spatial detail lost to optical blur while the latter is useful in recovering depth information. In brief, images acquired under spatially patterned illumination may be processed to recover an image with enhanced transverse and longitudinal resolution.

C. Assumptions

The present work assumes the availability of incoherent light sources with temporal fluctuations that are largely uncorrelated over the time scale of integration. It is also assumed that the patterning of light is strictly confined to the irradiance component, and that the light sensitive elements in the illumination module are individually addressable as pixels. This assumption will be relaxed later.

D. Highlights

Figure 6 provides a snapshot of results compiled from various experiments at SMU. Details pertaining to each experiment are disclosed in upcoming sections. The examples highlight the scope of the proposed approach, especially the latitude in selecting the optics and the sensor. In each instance, a distinct illumination pattern and reconstruction strategy is used to realize a specific task. In all instances, multiple images of the scene are acquired under patterned illumination. The images, while monochrome in appearance, represent light integrated over the full spectral bandwidth of the detector. The example on the top-left highlights the combined ability to super-resolve and recover depth when using sinusoidal illumination patterns. The example on the top-right relies on a stochastic illumination pattern (spread spectrum modulation) and a correlation receiver to surpass the diffraction limit of a well-corrected optic observing multiple targets.

Fig. 6. Select experimental results highlighting specific capabilities of proposed active computational imagers. Please note that in each example the super-resolved image contains spatial detail beyond the diffraction limit of the collection optics.

Download Full Size | PDF

The example in the bottom-left demonstrates the ability to computationally engineer imagers with resolving power that is fully decoupled from the constraints imposed by the collection optics. The collection optics in this example are a single lens element exhibiting severe anisotropy (or space variance) in resolving power. The results support the claim that space invariance is not a prerequisite for super-resolution using heterodyning. The illumination/reconstruction strategy adopted in the example may be readily extended to accommodate detectors with varying degrees of aliasing (ranging from a single pixel to focal plane array with arbitrary pixel pitch). The example in the bottom-right demonstrates the multiscale reconstruction capability afforded by the selection of a lattice of light spots as the illumination pattern.

In summary, resolution gains in excess of four times the diffraction limit and sub-micrometer range resolution over 1.1 m were achieved under lab settings. In all cases, the resolution gain was limited by the resolving power of the illumination optics.

The discussions to date have presented “active computational imaging” as a viable approach to circumvent resolution limits at the macroscopic scale. The remainder of this work attempts to delve deeper into the mechanics of circumventing these limits. Our inquiry begins with an attempt to develop a comprehensive model for irradiance transport in an active computational imager.

2. MODEL FOR IMAGING UNDER PATTERNED ILLUMINATION

Current models for irradiance transport in structured light systems limit their attention to well-corrected optics characterized by space-invariant blur. The subset of models concerned with macroscopic scales rely on a thin-lens approximation to describe the geometry of image formation. As a result, these models cannot accommodate the use of collection optics featuring multiple lens elements and apertures.

The present work outlines a general model for imaging under patterned illumination that simultaneously accommodates multi-lens systems and spatially varying optical blur. The model draws upon established principles in optics and computer vision, including

• a pupil-centric thick-lens model to describe the geometry of image formation [39,40];
• the use of epipolar geometry to describe the geometric relation between corresponding points in the illumination and image fields [41,42];
• physical optics models to describe the spatially varying camera and projector optical blur [43].

The principal objective of this section is to develop the mathematical relation between the detector irradiance and the scene radiance originating from patterned illumination. In the interest of clarity, initial efforts are restricted to the analysis of sinusoidal illumination patterns.

The notional stereo arrangement of Fig. 7 is used to introduce relevant concepts. It is comprised of an imaging device (or camera) and an illumination source (or projector) that are operable to acquire images of a 3D scene under patterned illumination. Henceforth, the term “active stereo setup” will be used to describe such stereo arrangements.

Fig. 7. Notional active stereo arrangement used to develop model for imaging under patterned illumination.

Download Full Size | PDF

The points $O_{cam}$ and $O_{ill}$ in Fig. 7, represent the center-of-perspective (COP) of the imaging and illumination modules, respectively. It is assumed without loss of generality that the world coordinate system $(X Y Z)$ is aligned with the camera coordinate system centered at $O_{cam}$ . The points $(c_{x}, c_{y})$ and $({\overset{´}{c}}_{x}, {\overset{´}{c}}_{y})$ represent the non-zero pixel coordinates corresponding to the intersection of the camera and projector optical axes with the respective sensor and display planes. The vector $\vec{O_{cam} O_{ill}}$ joining the camera and projector COP is referred to as the baseline vector or baseline, in keeping with nomenclature in computer vision.

The mathematical expressions developed in this section will oftentimes simultaneously accommodate spatial and spatial frequency coordinates. To minimize confusion, a standard notation has been adopted. For example, accented and unaccented coordinates denote points in the projector and camera image field, respectively. A more exhaustive list is compiled below:

3D coordinates	Coordinates of scene point, e.g., $(X, Y, Z)$
Unaccented 2D coordinates	Pixel coordinates of field point in image sensor, e.g., $(x, y)$
Accented 2D coordinates	Pixel coordinates of field point in display, e.g., $(\overset{´}{x}, \overset{´}{y})$
Uppercase boldface	Matrices $M_{i j}$ : element in $i$ th row, $j$ th column
Lowercase boldface	Column vectors
$F {\dots}$	Fourier transform operator
$Caligraphic typeface$	Fourier transform of a spatial pattern or 2D function
Greek symbols $ξ, η$	Spatial frequency coordinates expressed in the units $\frac{cycles}{mm}$ or $\frac{cycles}{pixel}$
$‖ b ‖$	$L_{2}$ norm of vector

A. Geometric Model

In the interest of simplicity, it is assumed that the collection optics in the notional imager of Fig. 7 are comprised of a double Gauss lens, such as the one illustrated in Fig. 8. The assumption is primarily intended to aid in the development of geometric concepts, and it may be replaced with any other combinations of lenses and apertures.

Fig. 8. Illustration of imaging model for a canonical collection optic.

Download Full Size | PDF

The relationship between the scene point $P \overset{def}{=} (X_{0}, Y_{0}, Z_{0})$ , and corresponding image coordinates $p \overset{def}{=} (x_{0}, y_{0})$ may be derived using the pupil-centric model for image formation documented in the literature [39,40]. The model assumes that lens elements and apertures may be aggregated into a single unit comprised of two terminals: an “entrance pupil” representing the finite aperture through which light-rays enters the optical elements and an “exit pupil” representing the finite aperture through which light-rays exit the optical elements en route to the sensor. The passage of light through the unit is described using the standard paraxial model for thick-lenses.

In the illustration of Fig. 8, the COPs of the illumination and imaging modules ( $O_{cam}$ and $O_{ill}$ ) correspond to the center of the entrance and exit pupil planes of the imaging and illumination modules.

The exact relation between the scene point $P \overset{def}{=} (X_{0}, Y_{0}, Z_{0})$ , and corresponding image $p \overset{def}{=} (x_{0}, y_{0})$ , is disclosed in Eq. (1). The negative sign in the expressions accommodate the image reversal observed at the sensor.

x_{0} = \frac{- m_{p}}{Δ} \frac{Z_{d}}{Z_{o}} X_{0} + c_{x}, y_{0} = \frac{- m_{p}}{Δ} \frac{Z_{d}}{Z_{0}} Y_{0} + c_{y} .

The above expression is obtained by relating the direction cosines of the incoming segment of the chief ray $o_{cam} p$ with those of the outgoing segment $O_{cam} P$ by means of the pupil magnification $m_{p}$ . In the simpler case of unity pupil magnification, the chief ray angles on the object and image side are identical, and the resulting expressions reduce to the imaging model frequently encountered in computer vision.

Please note that the values associated with $x_{0}$ and $y_{0}$ are expressed in pixel units and do not represent field coordinates. This slight abuse of notation enables us to readily incorporate standard results from epipolar geometry in computer vision.

The pixel offsets $c_{x}$ and $c_{y}$ in Eq. (1) accommodate the fact that the origin of the sensor pixel coordinate system is located at the top-left corner of the sensor modules. The term $Δ$ represents the pixel pitch of the image sensor, while the term $Z_{d}$ represents the distance between the camera exit pupil plane and the image sensor. Note that object-side and image-side distances are specified with reference to the entrance and exit pupil planes, respectively.

The relationship disclosed in Eq. (1) may be re-formulated as the matrix product shown below:

[\begin{matrix} x_{0} \\ y_{0} \\ 1 \end{matrix}] = \frac{1}{Z_{0}} \underset{K}{\underset{⏟}{[\begin{matrix} - m_{p} Z_{d} / Δ & 0 & c_{x} \\ 0 & - m_{p} Z_{d} / Δ & c_{y} \\ 0 & 0 & 1 \end{matrix}]}} [\begin{matrix} X_{0} \\ Y_{0} \\ Z_{0} \end{matrix}] .

Readers familiar with computer vision will recognize the matrix

K

as the camera intrinsic matrix.

The imaging model disclosed above may also be used to identify the relation between the scene point $(X_{0}, Y_{0}, Z_{0})$ and the pixel counterpart $({\overset{´}{x}}_{0}, {\overset{´}{y}}_{0})$ in the projector field. The result is expressed as the matrix product shown below:

[\begin{matrix} {\overset{´}{x}}_{0} \\ {\overset{´}{y}}_{0} \\ 1 \end{matrix}] = \overset{´}{γ} \underset{K}{\underset{⏟}{[\begin{matrix} - {\overset{´}{m}}_{p} {\overset{´}{Z}}_{d} / \overset{´}{Δ} & 0 & {\overset{´}{c}}_{x} \\ 0 & - {\overset{´}{m}}_{p} {\overset{´}{Z}}_{d} / \overset{´}{Δ} & {\overset{´}{c}}_{y} \\ 0 & 0 & 1 \end{matrix}]}} \underset{R}{\underset{⏟}{[\begin{matrix} {\overset{´}{r}}_{11} & {\overset{´}{r}}_{12} & {\overset{´}{r}}_{13} \\ {\overset{´}{r}}_{21} & {\overset{´}{r}}_{22} & {\overset{´}{r}}_{23} \\ {\overset{´}{r}}_{31} & {\overset{´}{r}}_{32} & {\overset{´}{r}}_{33} \end{matrix}]}} [\begin{matrix} X_{0} - b_{X} \\ Y_{0} - b_{Y} \\ Z_{0} - b_{Z} \end{matrix}],

where

\overset{´}{γ} = \frac{1}{{\overset{´}{r}}_{31} (X_{0} - b_{X}) + {\overset{´}{r}}_{32} (Y_{0} - b_{Y}) + {\overset{´}{r}}_{33} (Z_{0} - b_{Z})}

. The term

\overset{´}{Δ}

represents the pixel pitch of the projector while the term

{\overset{´}{Z}}_{d}

represents the distance between the projector entrance pupil plane and the light sensitive module.

The relation disclosed in Eq. (3) is derived by transforming the world coordinates of the scene point into an intermediate Cartesian coordinate frame centered at $O_{ill}$ , such that the $Z$ -axis is aligned with the projector optical axis. The vector ${[X_{0} - b_{X}, Y_{0} - b_{Y}, Z_{0} - b_{Z}]}^{T}$ accommodates the translation of the origin of the intermediate coordinate frame with respect to the world coordinate frame. Multiplication by $\overset{´}{R}$ accommodates the difference in the orientation of the world coordinate frame and the intermediate coordinate system centered at $O_{ill}$ .

1. Mapping from Projector $\leftrightarrow$ Camera Pixel Coordinates

Using the object-image and display-image relations disclosed in Eqs. (2) and (3), one can attempt to identify the geometric mapping between the camera and projector pixel coordinates. The process begins with the elimination of the vector ${[X_{0}, Y_{0}, Z_{0}]}^{T}$ from Eqs. (2) and (3). The resulting vector identities are disclosed below:

\begin{matrix} [\begin{matrix} {\overset{´}{x}}_{0} \\ {\overset{´}{y}}_{0} \\ 1 \end{matrix}] = \overset{´}{γ} Z_{0} \overset{´}{K} \overset{´}{R} K^{- 1} [\begin{matrix} x_{0} \\ y_{0} \\ 1 \end{matrix}] - \overset{´}{γ} \overset{´}{K} \overset{´}{R} K^{- 1} K [\begin{matrix} b_{X} \\ b_{Y} \\ b_{Z} \end{matrix}], \end{matrix}

[\begin{matrix} x_{0} \\ y_{0} \\ 1 \end{matrix}] = \frac{1}{\overset{´}{γ}} \frac{1}{Z_{0}} K {\overset{´}{R}}^{T} {\overset{´}{K}}^{- 1} [\begin{matrix} {\overset{´}{x}}_{0} \\ {\overset{´}{y}}_{0} \\ 1 \end{matrix}] + \frac{1}{Z_{0}} K [\begin{matrix} b_{X} \\ b_{Y} \\ b_{Z} \end{matrix}] .

The non-singular matrices

\overset{´}{K} \overset{´}{R} K^{- 1}

and

K {\overset{´}{R}}^{T} {\overset{´}{K}}^{- 1}

define a bi-linear mapping (homography/projective transform) between the pixel coordinates of the camera and projector, for objects at infinity

(Z_{0} \to \infty)

. These matrices are referred to as the infinite homography in vision literature. The vector

K {[b_{X}, b_{Y}, b_{Z}]}^{T}

represents the geometric image of the COP of the projector as seen from the camera viewpoint, and it is referred to as epipole in vision literature.

In the interest of notational brevity, the matrices $\overset{´}{K} \overset{´}{R} K^{- 1}$ and $K {\overset{´}{R}}^{T} {\overset{´}{K}}^{- 1}$ and the vector $K {[b_{X}, b_{Y}, b_{Z}]}^{T}$ will henceforth be abbreviated as $H^{\infty}$ , ${\overset{´}{H}}^{\infty}$ , and $t$ , respectively. These abbreviations allow us to express Eqs. (4) and (5) in the compact form disclosed in Eqs. (6) and (7).

[\begin{matrix} {\overset{´}{x}}_{0} \\ {\overset{´}{y}}_{0} \\ 1 \end{matrix}] = \overset{´}{γ} Z_{0} H^{\infty} [\begin{matrix} x_{0} \\ y_{0} \\ 1 \end{matrix}] - \overset{´}{γ} H^{\infty} [\begin{matrix} t_{X} \\ t_{Y} \\ t_{Z} \end{matrix}],

[\begin{matrix} x_{0} \\ y_{0} \\ 1 \end{matrix}] = \frac{1}{\overset{´}{γ}} \frac{1}{Z_{0}} {\overset{´}{H}}^{\infty} [\begin{matrix} {\overset{´}{x}}_{0} \\ {\overset{´}{y}}_{0} \\ 1 \end{matrix}] + \frac{1}{Z_{0}} [\begin{matrix} t_{X} \\ t_{Y} \\ t_{Z} \end{matrix}] .

It is worth emphasizing that the infinite homography matrices

H^{\infty}

and

{\overset{´}{H}}^{\infty}

and the epipole

t

are completely independent of scene geometry and depend exclusively on the placement of the camera and projector.

Equating the third element on either side of Eq. (6) enables us to solve for $\overset{´}{γ}$ . Substituting the resulting expression for $\overset{´}{γ}$ back into Eq. (6), yields the following $camera \to projector$ coordinate mapping:

{\overset{´}{x}}_{0} = \frac{h_{11}^{\infty} (Z_{0} x_{0} - t_{X}) + h_{12}^{\infty} (Z_{0} y_{0} - t_{Y}) + h_{13}^{\infty} (Z_{0} - t_{Z})}{h_{31}^{\infty} (Z_{0} x_{0} - t_{X}) + h_{32}^{\infty} (Z_{0} y_{0} - t_{Y}) + h_{33}^{\infty} (Z_{0} - t_{Z})}, {\overset{´}{y}}_{0} = \frac{h_{21}^{\infty} (Z_{0} x_{0} - t_{X}) + h_{22}^{\infty} (Z_{0} y_{0} - t_{Y}) + h_{23}^{\infty} (Z_{0} - t_{Z})}{h_{31}^{\infty} (Z_{0} x_{0} - t_{X}) + h_{32}^{\infty} (Z_{0} y_{0} - t_{Y}) + h_{33}^{\infty} (Z_{0} - t_{Z})} .

Alternately, one can solve for

\overset{´}{γ}

by equating the third element on either side of the vector-identity of Eq. (7). Substituting the resulting expression for

\overset{´}{γ}

yields the following

projector \to camera

coordinate mapping:

x_{0} = (\frac{Z_{0} - t_{Z}}{Z_{0}}) (\frac{{\overset{´}{h}}_{11}^{\infty} {\overset{´}{x}}_{0} + {\overset{´}{h}}_{12}^{\infty} {\overset{´}{y}}_{0} + {\overset{´}{h}}_{13}^{\infty}}{{\overset{´}{h}}_{31}^{\infty} {\overset{´}{x}}_{0} + {\overset{´}{h}}_{32}^{\infty} {\overset{´}{y}}_{0} + {\overset{´}{h}}_{33}^{\infty}}) + \frac{1}{Z_{0}} t_{X}, y_{0} = (\frac{Z_{0} - t_{Z}}{Z_{0}}) (\frac{{\overset{´}{h}}_{21}^{\infty} {\overset{´}{x}}_{0} + {\overset{´}{h}}_{22}^{\infty} {\overset{´}{y}}_{0} + {\overset{´}{h}}_{23}^{\infty}}{{\overset{´}{h}}_{31}^{\infty} {\overset{´}{x}}_{0} + {\overset{´}{h}}_{32}^{\infty} {\overset{´}{y}}_{0} + {\overset{´}{h}}_{33}^{\infty}}) + \frac{1}{Z_{0}} t_{Y} .

The expression for the

projector \leftrightarrow camera

coordinate mapping disclosed in Eqs. (8) and (9) is essential to the process of transporting irradiance from the projector to the scene and on to the camera. Variants of the expression were originally identified by the authors in [44]. A more detailed derivation of the current expression for the

projector \leftrightarrow camera

coordinate mapping is available in [4].

2. Limitations

The imaging model disclosed in Eqs. (2) and (3) assumes central projection consistent with the assumptions of paraxial imaging. Deviations from this behavior encountered in practice are accommodated as a source of space variance in the optical blur. As an example, radial distortion such as pincushion/barrel may be modeled as a source of field-dependent aberrations through the $W_{311}$ Seidel aberration term [43,45].

B. Blur Model

Accurate modeling of the light transport from the projector to the camera requires simultaneous consideration of the $projector \leftrightarrow camera$ coordinate mapping and the blurring induced by the imaging and illumination optics. Blurring on the image side limits the ability to resolve increasingly fine detail, while blurring on the illumination side limits the ability to project high frequency spatial patterns.

The remainder of this section is devoted to assimilating the geometric model developed in the previous section with physical optics models to describe optical blur in the imaging and illumination paths.

1. Light Transport from the Scene to the Detector

Our inquiry begins with an attempt to transport light from the scene to the detector. To this end, the illuminated object volume is interpreted as a collection of secondary point emitters/sources. The optical image of each point source is a blurry spot commonly referred to as the point-spread function (PSF). The PSF neatly encapsulates the effect of blurring due to diffraction, optical aberrations, and defocus. The spatial extent of the PSF imposes a firm limit on the resolving power of the imaging system.

Suppose that the camera in Fig. 7 observes a point source located at $(U_{0}, V_{0}, W_{0})$ in the object space. Suppose that $(u_{0}, v_{0})$ represents the transverse coordinates of the Gaussian image of the scene point $(U_{0}, V_{0}, W_{0})$ . A real-valued function of the form $h_{cam} (x - u_{0}, y - v_{0}; u_{0}, v_{0})$ encapsulates the light distribution at the ( $x$ , $y$ )th detector pixel in response to the point source at $(U_{0}, V_{0}, W_{0})$ . The arguments $(u_{0}, v_{0})$ in $h_{cam} (x - u_{0}, y - v_{0}; u_{0}, v_{0})$ capture the field dependence of the PSF with respect to the location of the point source. Lohmann and Paris [46] extended the above notion to an arbitrary collection of point sources using the superposition integral, as shown below:

i (x, y) = \iint p (u, v) h_{cam} (x - u, y - v; u, v) d u d v .

The term

p (u, v)

represents the intensity associated with a point source at

(U, V, W)

, whose Gaussian image coordinates are

(u, v)

.

Interpretation of the illuminated object volume as a collection of secondary point sources enables the use of Eq. (10) as a blueprint to identify the expression for the camera image $i_{θ} (x, y)$ . The corresponding expression for $i_{θ} (x, y)$ is included below:

i_{θ} (x, y) = \iint (\begin{matrix} s (u, v) \times \\ r (u, v) \end{matrix}) h_{cam} (x - u, y - v; u, v) d u d v .

The term

s (u, v)

represents the camera image of the illumination pattern, while

r (u, v)

represents the geometric image of the scene under uniform illumination.

2. Light Transport from the Projector to the Scene

Without loss of generality, the expression for the detected intensity disclosed in Eq. (11) may be adapted to obtain the expression for the radiance at the scene point $(X, Y, Z)$ arising from the illumination pattern $p_{θ} (\overset{´}{x}, \overset{´}{y})$ . The corresponding expression is provided below:

s (X, Y, Z) \propto \iint p_{θ} (\overset{´}{u}, \overset{´}{v}) h_{ill} (\overset{´}{x} - \overset{´}{u}, \overset{´}{y} - \overset{´}{v}; \overset{´}{u}, \overset{´}{v}) d \overset{´}{u} d \overset{´}{v} .

The term

p_{θ} (\overset{´}{u}, \overset{´}{v})

represents the intensity of intensity of the (

\overset{´}{u}

,

\overset{´}{v}

)th pixel in the illumination pattern, while

h_{ill} (\overset{´}{x}, \overset{´}{y}; \overset{´}{u}, \overset{´}{v})

represents the spatially varying blur associated with the illumination optics.

3. Light Transport from the Projector to the Camera

The expressions for the incident and detected intensities disclosed in Eqs. (11) and (12) provide a blueprint for the design of a comprehensive model for light transport in the active stereo arrangement of Fig. 7.

Subsequent discussions on the topic are restricted to the following sinusoidal patterns:

Periodic sinusoidal pattern:

p_{θ} (\overset{´}{x}, \overset{´}{y}) = \overset{´}{A} + \overset{´}{B} \sin (2 π (ξ_{0} \overset{´}{x} + η_{0} \overset{´}{y}) + θ) .

Warped sinusoidal pattern:

p_{θ} (\overset{´}{x}, \overset{´}{y}) = \overset{´}{A} + \overset{´}{B} \sin (2 π (ξ_{0} \frac{{\overset{´}{h}}_{11}^{\infty} \overset{´}{x} + {\overset{´}{h}}_{12}^{\infty} \overset{´}{y} + {\overset{´}{h}}_{13}^{\infty}}{{\overset{´}{h}}_{31}^{\infty} \overset{´}{x} + {\overset{´}{h}}_{32}^{\infty} \overset{´}{y} + {\overset{´}{h}}_{33}^{\infty}} + η_{0} \frac{{\overset{´}{h}}_{21}^{\infty} \overset{´}{x} + {\overset{´}{h}}_{22}^{\infty} \overset{´}{y} + {\overset{´}{h}}_{23}^{\infty}}{{\overset{´}{h}}_{31}^{\infty} \overset{´}{x} + {\overset{´}{h}}_{32}^{\infty} \overset{´}{y} + {\overset{´}{h}}_{33}^{\infty}}) + θ) .

The terms

ξ_{0}

and

η_{0}

represent the spatial frequency of the illumination pattern, as expressed in cycles/pixel. The terms

\overset{´}{A}

and

\overset{´}{B}

represent the average intensity and peak intensity excursion of the illumination pattern. The term

θ

represents the phase shift associated with the illumination pattern. The term

{\overset{´}{h}}_{i j}^{\infty}

represents the (

i

,

j

)^th entry of the infinite homography

{\overset{´}{H}}^{\infty} = K {\overset{´}{R}}^{T} {\overset{´}{K}}^{- 1}

.

The periodic sinusoidal pattern of Eq. (13) is the pattern of choice in super-resolution microscopy and phase measurement profilometry. The use of a warped sinusoidal pattern [Eq. (14)] is unique to this work. It is the outcome of a joint effort to recover scene geometry and spatial detail lost to blurring.

Light transport under periodic sinusoidal illumination

The transport model of Eq. (12) may be used to identify the incident intensity at the scene point $(X, Y, Z)$ in response to the periodic sinusoidal pattern of Eq. (13). The result is provided below:

s (X, Y, Z) = \overset{´}{A} (\overset{´}{x}, \overset{´}{y}) + | \overset{´}{B} (\overset{´}{x}, \overset{´}{y}) | \sin (\begin{matrix} 2 π (ξ_{0} \overset{´}{x} + η_{0} \overset{´}{y}) + \\ ∠ \overset{´}{B} (\overset{´}{u}, \overset{´}{v}) + θ \end{matrix}) .

The terms

\overset{´}{A} (\overset{´}{x}, \overset{´}{y})

,

| \overset{´}{B} (\overset{´}{x}, \overset{´}{y}) |

, and

∠ \overset{´}{B} (\overset{´}{x}, \overset{´}{y})

encapsulate the effect of the illumination optics on the contrast of the projected pattern and the instantaneous phase of the projected pattern.

A fraction of the light incident on $(X, Y, Z)$ reaches the ( $x$ , $y$ )^th camera pixel, following albedo loss and optical blur. The expression for the intensity of the ( $x$ , $y$ )^th pixel is obtained by aggregating the contribution of every scene point $(U, V, W)$ to the ( $x$ , $y$ )^th pixel, just as in Eq. (11). The resulting expression for the detector irradiance is disclosed below

i_{θ} (x, y) = \iint {\begin{matrix} r (u, v) [\overset{´}{A} (\overset{´}{u}, \overset{´}{v}) + | \overset{´}{B} (\overset{´}{u}, \overset{´}{v}) | \sin (\begin{matrix} 2 π (ξ_{0} \overset{´}{u} + η_{0} \overset{´}{v}) + \\ ∠ \overset{´}{B} (\overset{´}{u}, \overset{´}{v}) + θ \end{matrix})] \\ h_{cam} (x - u, y - v; u, v) \end{matrix}} d u d v .

The term

r (u, v)

in Eq. (16) represents the geometric image of the scene obtained under uniform illumination.

The coordinate mapping $(\overset{´}{x}, \overset{´}{y}) \mapsto (x, y)$ identified in Eq. (8) may be used to recast the expression for the detector irradiance in the form:

i_{θ} (x, y) = \iint {\begin{matrix} r (u, v) [A (u, v) + | B (u, v) | \sin (\begin{matrix} φ (u, v) + θ \\ + ∠ B (u, v) \end{matrix})] \\ h_{cam} (x - u, y - v; u, v) \end{matrix}} d u d v .

The definition of the various terms in Eq. (17) is included below:

\begin{array}{l} φ (u, v) \overset{def}{=} 2 π ξ_{0} \frac{h_{11}^{\infty} (W u - t_{X}) + h_{12}^{\infty} (W v - t_{Y}) + h_{13}^{\infty} (W - t_{Z})}{h_{31}^{\infty} (W u - t_{X}) + h_{32}^{\infty} (W v - t_{Y}) + h_{33}^{\infty} (W - t_{Z})} + 2 π η_{0} \frac{h_{21}^{\infty} (W u - t_{X}) + h_{22}^{\infty} (W v - t_{Y}) + h_{23}^{\infty} (W - t_{Z})}{h_{31}^{\infty} (W u - t_{X}) + h_{32}^{\infty} (W v - t_{Y}) + h_{33}^{\infty} (W - t_{Z})}, \\ A (u, v) \overset{def}{=} \overset{´}{A} \iint h_{ill} (\overset{´}{u} - \overset{´}{u}, \overset{´}{v} - \overset{´}{v}; \overset{´}{u}, \overset{´}{v}) d \overset{´}{u} d \overset{´}{v}, \\ B (u, v) \overset{def}{=} \iint {\begin{matrix} \exp (- j 2 π (ξ_{0} (\overset{´}{u} - \overset{´}{u}) + η_{0} (\overset{´}{v} - \overset{´}{v}))) \\ \times h_{ill} (\overset{´}{u} - \overset{´}{u}, \overset{´}{v} - \overset{´}{v}; \overset{´}{u}, \overset{´}{v}) \end{matrix}} d \overset{´}{u} d \overset{´}{v} . \end{array}]

The term

φ (u, v)

represents the parallax-induced phase distortion experienced by the illumination pattern, as observed from the camera perspective. Likewise, the terms

A (x, y)

,

| B (x, y) |

, and

∠ B (x, y)

represent the blur-induced distortion in the contrast and phase of the detected illumination pattern.

A comprehensive derivation of Eqs. (15) and (17) is available in Section 3.4 of [4].

Light transport under warped sinusoidal illumination

The transport model of Eq. (12) may be used to may be used to identify the incident intensity at the scene point $(X, Y, Z)$ in response to the warped sinusoidal pattern of Eq. (13). The resulting expression is identical to Eq. (15).

A fraction of the light incident on $(X, Y, Z)$ reaches the ( $x$ , $y$ )th camera pixel, following albedo loss and optical blur. The expression for the intensity of the ( $x$ , $y$ )th pixel is obtained by aggregating the contribution of every scene point $(U, V, W)$ to the ( $x$ , $y$ )th pixel, just as in Eq. (11). The resulting expression for the detector irradiance is disclosed below

i_{θ} (x, y) = \iint {\begin{matrix} r (u, v) [A (u, v) + | B (u, v) | \sin (\begin{matrix} φ (u, v) + θ \\ + ∠ B (u, v) \end{matrix})] \\ h_{cam} (x - u, y - v; u, v) \end{matrix}} d u d v .

The definition of the various terms in Eq. (19) is included below

\begin{array}{l} φ (u, v) \overset{def}{=} 2 π (\frac{W - t_{Z}}{W}) (ξ_{0} u + η_{0} v) - (\frac{2 π}{W - t_{Z}}) (ξ_{0} t_{X} + η_{0} t_{Y}) \\ A (u, v) \overset{def}{=} \overset{´}{A} \iint h_{ill} (\overset{´}{u} - \overset{´}{u}, \overset{´}{v} - \overset{´}{v}; \overset{´}{u}, \overset{´}{v}) d \overset{´}{u} d \overset{´}{v} \\ B (u, v) \overset{def}{=} \iint {\begin{matrix} \exp (- j φ (\overset{´}{u} - \overset{´}{u}, \overset{´}{v} - \overset{´}{v})) \\ \times h_{ill} (\overset{´}{u} - \overset{´}{u}, \overset{´}{v} - \overset{´}{v}; \overset{´}{u}, \overset{´}{v}) \end{matrix}} d \overset{´}{u} d \overset{´}{v} \end{array}] .

The derivation employs the coordinate mapping

(x, y) \mapsto (\overset{´}{x}, \overset{´}{y})

identified in Eq. (9). A comprehensive derivation of Eqs. (19) and (20) is available in Section 3.5 of [4].

The term $φ (u, v)$ in the expression for the detector irradiance [Eq. (19)] represents the parallax-induced phase distortion experienced by the illumination pattern, when observed from the camera perspective.

Likewise, the terms $A (x, y)$ , $| B (x, y) |$ , and $∠ B (x, y)$ represent the blur-induced distortion in the contrast and phase of the detected illumination pattern.

4. Highlights

Equations (17)–(20) furnish a rigorous model for imaging under sinusoidal illumination that simultaneously accommodates space variant blur in the imaging/illumination paths and topographic variation in the scene.

On account of the linearity of the light transport mechanism, these expressions for the detector irradiance may be readily extended to accommodate illumination patterns that are expressible as a superposition of sinusoids or warped sinusoids.

5. Key Findings

Figure 9 succinctly captures two key results that emerge from the analysis of light transport in structured light systems.

Fig. 9. Expression for detector irradiance in an active stereo setup.

Download Full Size | PDF

First, the camera perceives a scene-dependent amplitude and phase variation in the projected pattern. This finding is consistent with practical observations.

Second, the camera image is a blurred representation of the ideal geometric image of the scene, subject to amplitude and phase modulation by a pattern whose functional form is identical to the functional form of the illumination pattern.

Super-resolution by heterodyning is predicated in the observation of unresolved spatial frequencies in the camera images acquired under sinusoidal illumination. The presence of such frequencies may be confirmed by analyzing the Fourier transform of the product term $r (u, v) [A (u, v) + | B (u, v) | \sin (φ (u, v) + ∠ B (u, v) + θ)]$ . The field-dependent amplitude distortions arising from $A (u, v)$ and $| B (u, v) |$ diminish the modulation strength of the projected pattern, limiting the ability to heterodyne increasingly fine object details.

Depth estimation is predicated in the observation of scene-dependent phase deformations in the camera images of the projected illumination pattern. The scene-dependent phase modulation term $φ (u, v)$ serves this purpose and permits recovery of depth information $W$ from $φ (u, v)$ , through Eqs. (18) and (20). The additional phase distortion $∠ B (u, v)$ attributed to the illumination blur results in an over/under estimation of the scene depth $W$ . The field-dependent amplitude distortions arising from $A (u, v)$ and $| B (u, v) |$ diminish the modulation strength of the projected pattern, resulting in a loss of range resolution. These issues are examined in detail in [4].

6. Limitations

The light transport model developed in this work disregards cast shadows and irradiance contributions from indirect light paths involving inter-reflections and/or scattering in the scene. Such behavior is expected to introduce bias in the recovered scene geometry and artifacts in the super-resolved image. A rigorous treatment of the topic is beyond the scope of this paper.

C. Useful Restrictions

It is evident from the discussion in Section 2.B.5 that neither space variance in the optical blur nor topographic variations in a macroscopic scene preclude the possibility of super-resolution using patterned illumination.

At first glance, it appears that the heterodyned frequencies can be restored to the correct positions outside the optical passband using an AM (amplitude modulation) receiver that is tuned to the spatial frequency $ξ_{0}, η_{0}$ of the illumination pattern $p_{θ} (\overset{´}{x}, \overset{´}{y})$ . In practice, it is observed that exact restoration is only supported by select camera and projector arrangements. This inconsistency stems from the fact that the detected sinusoidal pattern exhibits scene-dependent phase distortion, that is, $φ (u, v)$ depends on scene depth $W$ . Such distortions are evident in the image of the chess pieces in Fig. 10 and at the interface of the vertical planes and the ground plane. The scene-dependence, while desirable from a depth estimation standpoint, poses problems for super-resolution. Failure to compensate for these scene-dependent distortions produces aliasing-like artifacts in the demodulated/super-resolved image.

Fig. 10. Camera image of scene under periodic sinusoidal illumination in the active stereo arrangement of Fig. 7.

Download Full Size | PDF

A formal discussion of the mechanics of super-resolution and depth estimation will be deferred until Section 3. The remainder of this section is devoted to the study of combinations of stereo arrangements and illumination patterns, for which the camera image of the projected pattern is devoid of scene-dependent phase distortions, that is, $φ (u, v)$ is independent of scene depth.

1. Monostatic Arrangement for Periodic Sinusoidal Illumination

The expressions for the detector irradiance $i_{θ} (x, y)$ [listed in Eq. (17)] and the phase $φ (u, v)$ [listed in Eq. (18)] provide clues to eliminate scene-dependent distortions when using periodic sinusoidal patterns.

It follows from Eq. (18) that $φ (u, v)$ exhibits a bi-linear dependence with the scene depth $W$ . This bi-linear dependence may be reduced to a strict inverse dependence under a specific choice of infinite homography $H^{\infty}$ , epipole $t$ and pattern orientation $ξ_{0}, η_{0}$ . The resulting expression for $φ (u, v)$ is disclosed below:

φ (u, v) = 2 π κ_{o} (ξ_{0} u + η_{0} v) + φ_{o} + \frac{1}{W} 2 π κ_{d} (ξ_{0} b_{X} + η_{0} b_{Y}),

wherein

k_{o} \overset{def}{=} (\frac{{\overset{´}{m}}_{p}}{m_{p}} \frac{{\overset{´}{Z}}_{d}}{Z_{d}} \frac{Δ}{\overset{´}{Δ}}), k_{d} \overset{def}{=} \frac{{\overset{´}{m}}_{p} {\overset{´}{Z}}_{d}}{\overset{´}{Δ}}, φ_{o} \overset{def}{=} 2 π [\begin{matrix} ξ_{0} ({\overset{´}{c}}_{x} - k_{o} c_{x}) \\ η_{0} ({\overset{´}{c}}_{y} - k_{o} c_{y}) \end{matrix}] .

The term

κ_{o}

accommodates differences in the transverse magnification of the illumination and imaging paths, while

φ_{o}

accommodates differences in the sampling phase of the sensor and display pixel grids.

Constraints on stereo arrangement

• Infinite homography $H^{\infty}$ reduces to an affine transform, that is, $h_{31}^{\infty} = h_{32}^{\infty} = 0$ and $h_{33}^{\infty} = 1$
• $Z$ -component of the epipole $t_{z} = 0$ .

These constraints are simultaneously satisfied when the camera and projector optical axes are parallel and the baseline vector is perpendicular to both optical axes, that is,

\overset{´}{R} = I

and

b_{Z} = 0

. This stereo arrangement is referred to as the canonical stereo arrangement in vision literature. In such arrangements, the entrance pupil plane of the camera and the exit pupil plane of the projector are coplanar, so that there is no

Z

-displacement between the corresponding COPs. The net result is that the camera and projector side epipoles are both at infinity.

Constraints on pattern orientation

The phase function $φ (u, v)$ simplifies to a pure carrier frequency favored by super-resolution when the pattern orientation satisfies the constraint $ξ_{0} b_{X} + η_{0} b_{Y} = 0$ . In such cases, the iso-intensity lines in the illumination pattern are aligned with the camera-side epipolar lines [47,48]. Violation of the constraint $ξ_{0} b_{X} + η_{0} b_{Y} = 0$ produces depth-dependent phase distortions that are useful in estimating scene depth. The aforementioned characteristics are observed and verified in experiments. Figure 11 illustrates these characteristics for a canonical stereo arrangement with a vertical baseline.

Fig. 11. Canonical active stereo arrangement with epipoles at infinity.

Download Full Size | PDF

The discussion thus far seems to suggest that the ability that to super-resolve is confined to a single orientation. The problem is easily addressed by employing multiple canonical stereo arrangements that share a common imager. Alternatively, one could rotate a single canonical stereo arrangement about the camera COP. Details are available in [4].

The chief limitation of the canonical stereo arrangement is the inability to accommodate a large baseline that is necessary to improve the depth resolution of the computational imager. The above fact is evident in the linear/inverse dependence of the phase $φ (u, v)$ with the baseline/scene depth in Eq. (21). Attempts to increase the baseline in canonical stereo arrangements are likely to fail on account of the limited overlap between the illumination and imager field-of-view (FOV).

2. Monostatic Arrangement for Warped Sinusoidal Illumination

The expressions for the detector irradiance $i_{θ} (x, y)$ [listed in Eq. (19)] and the phase $φ (u, v)$ [listed in Eq. (20)] provide clues to eliminate scene-dependent distortions when using periodic sinusoidal patterns.

It follows from Eq. (20) that $φ (u, v)$ exhibits a bi-linear dependence with the scene depth $W$ . This bi-linear dependence may be reduced to a strict inverse dependence under a specific choice of the epipole $t$ and pattern orientation $ξ_{0}, η_{0}$ . The resulting expression for $φ (u, v)$ is disclosed below:

φ (u, v) = 2 π κ_{o} (ξ_{0} u + η_{0} v) + φ_{o} + \frac{1}{W} 2 π κ_{d} (ξ_{0} b_{X} + η_{0} b_{Y}),

wherein

κ_{o} \overset{def}{=} 1, κ_{d} \overset{def}{=} 1, φ_{o} \overset{def}{=} 0 .

Constraints on stereo arrangement

• $Z$ -component of the epipole $t_{z} = 0$ .

This above constraint is satisfied when the projector COP lies in the entrance pupil plane of the camera. The net result is that the camera-side epipole is at infinity. Such stereo arrangements are referred to as collocated stereo arrangements in this work.

Constraints on pattern orientation

The phase function $φ (u, v)$ simplifies to a pure carrier frequency favored by super-resolution when the pattern orientation satisfies the constraint $ξ_{0} b_{X} + η_{0} b_{Y} = 0$ . In such cases, the iso-intensity lines in the illumination pattern are aligned with the camera-side epipolar lines [44]. Violation of the constraint $ξ_{0} b_{X} + η_{0} b_{Y} = 0$ produces depth-dependent phase distortions useful for estimating depth.

Both characteristics are observed and verified in experiments. Figure 12 illustrates these characteristics for a canonical stereo arrangement with a horizontal baseline.

Fig. 12. Collocated stereo arrangement with camera epipole at infinity.

Download Full Size | PDF

The principal difference between the canonical and collocated stereo arrangements lies in the non-coplanar structure of the pupil planes and the non-zero angle between the optical axes. The warped nature of the sinusoidal illumination pattern accounts for the relative rotation between the camera and projector optical axes. As a result, the warped sinusoidal illumination pattern appears as a periodic sinusoid to the camera. In the absence of warping, a pure sinusoidal pattern appears as a chirp pattern to the camera. The supplementary material [49] includes a video that illustrates the benefit of warping.

Broader impact

It is observed that the stereo arrangements in commodity structured light (SL) scanners tend to operate in near-collocated configurations. This suggests that commodity SL scanners can be adapted (with minimal modification) to super-resolve texture, in addition to recovering topographic information. This unique capability has hitherto remained undocumented in the optics and vision literature.

Lastly, the canonical stereo arrangement of Section 2 may be interpreted as a special case of the collocated stereo arrangement, so that one could attempt to pre-warp the periodic sinusoidal illumination pattern in a canonical stereo arrangement. In this case, pre-warping compensates for the difference in the relative magnification of the camera and projector and also the difference in the sampling phase of the camera and projector sampling grids.

3. Monostatic Arrangement for Any Illumination Pattern

Attempts to super-resolve in multiple orientations using the canonical/collocated stereo configurations require the use of multiple projectors or mechanical movement. These issues may be overcome by examining stereo arrangements for which the phase function $φ (u, v)$ [listed in Eq. (18)] is independent of scene depth. Direct substitution can be used to verify that when the camera and projector share the same viewpoint $(\overset{´}{R} = identity matrix, b_{X} = b_{Y} = b_{Z} = 0)$ , the phase function $φ (u, v)$ is completely independent of scene depth. The resulting expression for $φ (u, v)$ is disclosed below:

φ (u, v) = 2 π κ_{o} (ξ_{0} u + η_{0} v) + φ_{o},

wherein

κ_{o} \overset{def}{=} (\frac{{\overset{´}{m}}_{p}}{m_{p}} \frac{{\overset{´}{Z}}_{d}}{Z_{d}} \frac{Δ}{\overset{´}{Δ}}), φ_{o} \overset{def}{=} 2 π [\begin{matrix} ξ_{0} ({\overset{´}{c}}_{x} - κ_{o} c_{x}) \\ η_{0} ({\overset{´}{c}}_{y} - κ_{o} c_{y}) \end{matrix} +] .

In practice, a 45° beam splitter is used to ensure that the camera and projector share a common viewpoint. The process entails alignment of the optical axes and the COPs of the camera and projector. The aforementioned stereo arrangement is referred to as coincident stereo setup, because the two COPs coincide with each other.

An attractive feature of the coincident stereo arrangement is that any illumination pattern (sinusoidal or otherwise) appears devoid of phase deviations from the camera perspective. For this reason, it is the preferred embodiment to realize super-resolution using any spatial pattern.

Intuition suggests that the absence of scene-dependent phase distortions in the phase function $φ (u, v)$ would impede attempts to recover topographic information in a coincident stereo arrangement. In such cases, the finite depth of field of the projector may be exploited to recover depth information from the modulation strength $(given by \frac{A (u, v)}{| B (u, v) |})$ of the detected pattern [50].

4. Bistatic Arrangement for Periodic Sinusoidal Patterns and Superpositions of Periodic Sinusoids

The canonical/collocated active stereo arrangements impose specific restrictions on the placement $(b_{X}, b_{Y}, b_{Z})$ of the camera and projector. The common restriction $b_{Z} = 0$ appears to automatically rule out bistatic arrangements. Further analysis indicates that these restrictions may be lifted when attempting to super-resolve a single planar facet that is plane-parallel to both the camera and projector pupil planes. Henceforth, the term 2D scene is used to describe the planar facet that is plane-parallel to both the camera and projector pupil planes. The nomenclature reflects the fact that these scenes do not exhibit topographical variation from the camera perspective.

In the aforementioned case, it is observed that the relative rotation $\overset{´}{R} = identity matrix$ , and the absolute depth of each scene point reduces to a constant $W_{0}$ that is independent of the field coordinates $(u, v)$ . Incorporating these values into Eq. (18) yields the following expression for phase function $φ (u, v)$ :

φ (u, v) = 2 π κ_{o} (ξ_{0} u + η_{0} v) + φ_{o},

wherein

κ_{o} \overset{def}{=} (\frac{{\overset{´}{m}}_{p}}{m_{p}} \frac{{\overset{´}{Z}}_{d}}{Z_{d}} \frac{Δ}{\overset{´}{Δ}}) \frac{W_{0}}{W_{0} - t_{Z}}

and

φ_{o} \overset{def}{=} [2 π ξ_{0} ({\overset{´}{c}}_{x} - (\frac{{\overset{´}{m}}_{p}}{m_{p}} \frac{{\overset{´}{Z}}_{d}}{Z_{d}} \frac{Δ}{\overset{´}{Δ}}) c_{x}) + 2 π η_{0} ({\overset{´}{c}}_{y} - (\frac{{\overset{´}{m}}_{p}}{m_{p}} \frac{{\overset{´}{Z}}_{d}}{Z_{d}} \frac{Δ}{\overset{´}{Δ}}) c_{y}) - 2 π \frac{κ_{o}}{W_{0}} (ξ_{0} t_{X} + η_{0} t_{Y})] .

Notice that the linear phase profile of

φ (u, v)

is consistent with the requirements of super-resolution.

A bistatic active stereo arrangement was used to super-resolve the 2D scenes in Panels 2–4 of Fig. 6.

3. SUPER-RESOLUTION USING SINUSOIDAL PATTERNS

As stated previously, active super-resolution is predicated on the prospect of observing unresolved spatial frequencies in the camera images acquired under sinusoidal illumination. It was established in Section 2.C that a select combination of stereo arrangements and sinusoidal patterns supports pure AM by a carrier, as favored by super-resolution. However, it remains to be established that the modulated spatial frequencies can be unambiguously restored, effecting an improvement in the resolving power/bandwidth of the collection optics.

A demodulation strategy utilizing trigonometric identities to process images acquired under sinusoidal illumination is outlined below.

A. Demodulation Strategy

The strategy adopted in this work is encapsulated in the expression for the reconstructed image, disclosed below:

i_{recon} (x, y) \overset{def}{=} i_{b b} (x, y) + (\begin{matrix} \cos (2 π κ_{o} (ξ_{0} x + η_{0} y) + φ_{0}) i_{\cos} (x, y) \\ + \sin (2 π κ_{o} (ξ_{0} x + η_{0} y) + φ_{0}) i_{\sin} (x, y) \end{matrix}) .

The term

i_{b b} (x, y)

dubbed the “baseband image,” represents the camera image acquired under uniform illumination. The terms

i_{\cos} (x, y)

and

i_{\sin} (x, y)

represent images that are acquired under strictly cosine and sine illumination, respectively. The scale factor

κ_{o}

accommodates differences in the transverse magnification of the illumination and imaging paths. The phase offset

φ_{o}

accommodates differences in the sampling phase of the detector pixel grid and the projector pixel grid.

The images $i_{b b}, i_{\cos}$ , and $i_{\sin}$ are derived from the set of camera images ${i_{θ} (x, y)}$ acquired under the phase-shifted illumination pattern ${p_{θ} (\overset{´}{x}, \overset{´}{y}) : θ \in [0, π / 2, π, 3 π / 2]}$ , using Eqs. (26)–(28).

i_{b b} (x, y) \overset{def}{=} \frac{1}{4} [i_{0} (x, y) + i_{π / 2} (x, y) + i_{π} (x, y) + i_{3 π / 2} (x, y)],

i_{\cos} (x, y) \overset{def}{=} \frac{1}{2} [i_{π / 2} (x, y) - i_{3 π / 2} (x, y)],

i_{\sin} (x, y) \overset{def}{=} \frac{1}{2} [i_{0} (x, y) - i_{π} (x, y)] .

The expression for the reconstructed image

i_{recon} (x, y)

may be simplified by utilizing the trigonometric identity

\cos (P - Q) = \cos P \cos Q + \sin P \sin Q

. The simplified expression for the reconstructed image

i_{recon} (x, y)

is disclosed below:

i_{recon} (x, y) = \iint [r (u, v) o (x - u, y - v; u, v) h_{cam} (x - u, y - v; u, v)] d u d v,

wherein

o (x, y; u, v) \overset{def}{=} A (u, v) + | B (u, v) | \cos (\begin{matrix} 2 π κ_{o} (ξ_{0} x + η_{0} y) \\ - ∠ B (u, v) \end{matrix}) .

Notice that the reconstructed image

i_{recon} (x, y)

bears a strong resemblance to the image acquired under the computationally engineered PSF,

h_{engd} (x, y; u, v) \overset{def}{=} o (x, y; u, v) \times h_{cam} (x, y; u, v) .

1. Frequency Domain Interpretation of Super-Resolution

The terms $A (u, v), B (u, v)$ , and $h_{cam} (x, y; u, v)$ embedded within the expression for the engineered PSF remain fixed values for a fixed field location $(u, v)$ . The deterministic nature of these terms enables us to associate a field-dependent transfer function to the imager and the computationally engineered imager. The standard practice of identifying the transfer function by computing the Fourier transform of the PSF can be utilized to obtain the following result:

F {h_{engd} (x, y; u, v)} = F {o (x, y; u, v)} ⊛ F {h_{cam} (x, y; u, v)} .

The operator

F

denotes the 2D Fourier transform while

⊛

denotes the convolution operator.

It is observed that the convolution operation in Eq. (32) produces replicas of the field-dependent camera OTF, which are centered at the frequencies (0, 0) and $\pm (\frac{κ_{o} ξ_{0}}{\overset{´}{Δ}}, \frac{κ_{o} η_{0}}{\overset{´}{Δ}}) \frac{cycles}{mm}$ . The replication in the frequency domain serves to increase the bandwidth of the computationally engineered OTF and the underlying computational imager.

2. Spatial Domain Interpretation of Super-Resolution

Intuition suggests that an increase in the bandwidth of the computationally engineered OTF is likely accompanied by a reduction in the size of the resolvable spot. An accepted criterion to identify the size of the resolvable spot is the width of the central lobe. It is identified as the separation of the intensity minima lying on either side of the intensity maximum. The multiplicative structure of Eq. (31) guarantees that the engineered PSF shares the minima of the camera PSF and the oscillatory pattern. In theory, a tightening of the central lobe should result, when two or more minima of the oscillatory pattern $o (x, y; u, v)$ are accommodated within the central lobe of the camera PSF $h_{cam} (x, y; u, v)$ . We proceed to verify this claim using an example.

The field dependence of the terms $A (u, v)$ and $B (u, v)$ complicates the interpretation of PSF engineering. So we assume (strictly for the purpose of this discussion) that the illumination blur is space invariant, that is, $A (u, v) = A, B (u, v) = B$ , and $κ_{o} = 1$ .

Figure 13 illustrates the prospect of super-resolving a diffraction-limited imager, using periodic sinusoidal illumination. The plots labeled “optical PSF” represent cross-sections of an Airy disk with a cutoff frequency $ρ_{0} = 193.69 cyc / mm$ . The red lines depict the spacing between the first two nulls. The plots labeled “engineered PSF” represent cross-sections of the engineered PSF in two principal directions: one aligned with the modulation direction and the other orthogonal to the modulation direction. A modulation frequency of $ξ_{0} = 0$ , $η_{0} = ρ_{0} = 193.69 cyc / mm$ was chosen for the purpose of this example. Notice that the resolution gain in the modulation direction manifests as a reduction in the central lobe width of the engineered PSF. This finding is consistent with the claims made in previous paragraphs. It is observed that four consecutive intensity minima of the oscillatory pattern can be accommodated within the central lobe of the Airy disk. Each intensity minimum marks the beginning of a side lobe in the engineered PSF.

Fig. 13. Super-resolution of a diffraction-limited imager affected by computational PSF engineering.

Download Full Size | PDF

Intuition suggests that a larger resolution gain should result when multiple intensity minima of the oscillatory patterns are accommodated within the central lobe of the Airy disk. In practice, it is observed that the side lobes manifest as ghost artifacts in the reconstructed image. The artifacts are attributed to the highly repetitive structure of the sinusoidal illumination pattern and the associated oscillatory pattern.

Figure 14 illustrates the prospect of engineering the PSF of a 19 mm biconvex lens using a periodic sinusoidal pattern with frequency $ξ_{0} = 0$ , $η_{0} = 125 cyc / mm$ . The diffraction-limited cutoff frequency of the singlet is $ρ_{0} = 193.69 cyc / mm$ .

Fig. 14. Super-resolution of a 19 mm biconvex lens affected by computational PSF engineering.

Download Full Size | PDF

The red insets represent the on-axis PSF, while the green, blue and magenta insets represent PSFs at the edges of the image field: (0.76 mm, 0), (0, 0.76 mm), and (0.76 mm, 0.76 mm).

Inspection of the engineered PSFs confirms resolution improvement in the modulation direction. Closer inspection of the off-axis engineered PSFs reveals pronounced side lobes. The presence of these lobes is attributed to the increased number of cycles of the oscillatory pattern that can be accommodated within the central lobe of the off-axis PSF’s.

The examples discussed thus far support the notion that super-resolution is affected by computational engineering of the transverse PSF. The notion generalizes the viewpoint espoused in [51] for space-invariant blurs, and it is an original contribution of this work.

B. Aliasing Management

The analysis of super-resolution thus far has implicitly assumed that the detector Nyquist limit exceeds the diffraction-limited cutoff frequency of the imaging optics. The assumption may be violated in practice. Our super-resolution technique accommodates mild under sampling (e.g., Nyquist frequency $< 0.75 \times optical$ cutoff) by digitally low-pass filtering (DLPF) the recorded images prior to demodulation. The cutoff frequency of the DLPF is chosen to block aliased frequency components ( $> 0.25 \times optical$ cutoff in the example) in the recorded image. The additional loss in resolution stemming from DLPF is overcome by using additional sinusoidal illumination patterns.

Severe under sampling in the recorded images introduces irreversible ambiguities in the position of the heterodyned frequencies. The issue is overcome in Section 5 by adopting a different modulation (illumination pattern) and reconstruction strategy.

Even in the absence of under sampling, the finite pixel pitch of the detector could present problems during demodulation. A subset of the demodulated frequencies may exceed the sensor Nyquist frequency producing aliasing artifacts in the reconstructed image $i_{recon} (x, y)$ . This issue was observed and reported by the authors in [52]. It was observed that resampling the images $i_{b b} (x, y), i_{\cos} (x, y)$ , and $i_{\sin} (x, y)$ resolves the problem. Spatial domain interpolation techniques such as Lanczos interpolation may be employed for the purpose. The results in Panel 1 of Fig. 6 were produced with appropriate resampling prior to demodulation.

C. Experimental Validation

In the interest of brevity, the manuscript describes two experiments that corroborate the findings of this work and demonstrate super-resolution using heterodyning. A more exhaustive list of experiments validating the proposed approach in a variety of monostatic and bistatic arrangements is compiled in [4,44] and [48].

1. Super-Resolving Well-Corrected Optics

Figure 15 provides details of the apparatus used to demonstrate resolution enhancement of a well-corrected optic that observes a macroscopic scene exhibiting topographic variations.

Fig. 15. Super-resolution in a collocated active stereo arrangement.

Download Full Size | PDF

Results from the experiment are included in Panel 1 of Fig. 6. We briefly outline the procedure used to set up the collocated stereo arrangement and identify the warped sinusoidal pattern.

The three optical posts of increasing height embedded within the yellow rectangle of Fig. 15 are used to align the camera optical axis with a line on the optical table and to ascertain that the camera rotates about this line, through the COP. A thin and sharp tipped object, such as the lead of a mechanical pencil, is used to determine whether the projector COP is contained within the camera entrance pupil plane. Collocation in the horizontal direction is achieved when the tip of the lead and the shadow under flood illumination from the projector are lined up horizontally. Details are available in the supplementary materials.

The warped sinusoidal pattern required for super-resolution is identified using Eq. (14). The process requires knowledge of the infinite homography ${\overset{´}{H}}^{\infty}$ , whose computation is tedious. Previous work [44] by the authors has established that the homography $\overset{´}{Π}$ induced by a planar facet that is plane-parallel to the projector exit pupil may be used to identify the warped sinusoidal pattern warp in lieu of the infinite homography. The facet labeled $π$ in Fig. 15 serves this purpose.

The homography is computed using established techniques in computer vision [53] that rely on matching corresponding features in the camera image of a reference projector pattern, such as a grid of squares.

The warped sinusoidal patterns obtained using the aforementioned procedure are included in the image insets of Fig. 12

The process of super-resolution begins with the acquisition of images $i_{θ} (x, y)$ under the phase-shifted illumination patterns shown below:

p_{θ} (\overset{´}{x}, \overset{´}{y}) = \frac{1}{2} + \frac{1}{2} \sin (2 π η_{0} (\frac{{\overset{´}{Π}}_{21} \overset{´}{x} + {\overset{´}{Π}}_{22} \overset{´}{y} + {\overset{´}{Π}}_{23}}{{\overset{´}{Π}}_{31} \overset{´}{x} + {\overset{´}{Π}}_{32} \overset{´}{y} + {\overset{´}{Π}}_{33}}) + θ),

wherein

η_{0} = \frac{1}{6} \frac{cyc}{pixel}, θ \in [0, π / 2, π, 3 π / 2], {\begin{matrix} x \in [1,1400] \\ \overset{´}{y} \in [1,1050] \end{matrix}}

. The camera images acquired under the aforementioned illumination patterns are processed using the demodulation strategy outlined in Section 3.A to obtain a super-resolved image of the scene. The result is disclosed in Panel 1 of Fig. 6. Inspection of the image insets confirms the ability to super-resolve spatial detail in a macroscopic scene exhibiting topographic variations.

The spatial frequency response (SFR) plots of Fig. 16 aid in the quantitative assessment of resolution gain. Comparison of the red and blue plots suggest a marked and marginal resolution gain in the modulation direction and orthogonal orientation, respectively. The marginal gain is an artifact of the dynamic range adjustment process used to match the appearance of the reconstructed and baseband images.

Fig. 16. Spatial frequency response (SFR) plots.

Download Full Size | PDF

A resolution gain of 1.6602 is observed in the modulation direction. The number is obtained as the ratio of the cutoff frequency prior to and subsequent to super-resolution. The cutoff frequency is defined as the spatial frequency for which the modulation strength falls to 0.02.

2. Super-Resolving Optics with Spatially Varying Blur

Details of the coincident stereo apparatus used to demonstrate resolution improvement in a collection optic afflicted with space variance are included in Fig. 17.

Fig. 17. Super-resolving singlet using a coincident stereo arrangement.

Download Full Size | PDF

A 45° pellicle beam splitter and standard alignment techniques are used to ensure that the COPs of the camera and projector are lined up. Details on the alignment procedure adopted in this work are available in the supplementary materials.

The image insets of Fig. 18 illustrate the severity of space variance in the camera optical blur. The images represent the response of the collection optics to a grid-of-dots displayed on a 24 inch LCD monitor. The size of the each spot dot is approximately $270 μm \times 270 μm$ .

Fig. 18. Measured PSF at select image field locations.

Download Full Size | PDF

The process of super-resolution begins with the acquisition of images $i_{θ} (x, y)$ recorded under the phase-shifted illumination patterns disclosed in Eq. (34).

p_{θ} (\overset{´}{x}, \overset{´}{y}) = \frac{1}{2} + \frac{1}{2} \sin (2 π ξ_{0} \overset{´}{x} + θ), p_{θ} (\overset{´}{x}, \overset{´}{y}) = \frac{1}{2} + \frac{1}{2} \sin (2 π η_{0} \overset{´}{y} + θ),

wherein

ξ_{0}, η_{0} = \frac{1}{6} \frac{cyc}{pixel}, θ \in [0, 2 π / 3, 4 π / 3], {\begin{matrix} x \in [1,1920] \\ \overset{´}{y} \in [1,1080] \end{matrix}}

. The camera images acquired under the aforementioned illumination patterns are processed using the demodulation strategy outlined in Section 3.A to obtain a super-resolved image of the resolution target. The result is disclosed in Fig. 19.

Fig. 19. Results from attempt to super-resolve a singlet using sinusoidal patterns in the coincident stereo arrangement of Fig. 16.

Download Full Size | PDF

Inspection of the image insets confirms that space variance in the optical blur is not an impediment to super-resolution. It also confirms that linearity (not space invariance) is the principal requirement for super-resolution using heterodyning. The aforementioned view broadens the scope of super-resolution techniques to accommodate a much larger class of imagers not limited to well-corrected optics.

A visual assessment of the number of resolvable bars in each USAF target hints at a resolution gain of four elements. The knowledge that successive elements in the USAF target differ in resolution by $2^{1 / 6}$ may be used to ascertain an empirical resolution gain of 1.5874.

Key observations

Field-dependent resolution gain: A comparison of the modulation strength of element-6 in each colored inset confirms the field dependence on the resolving power of the engineered imager. The most visible improvement appears near the optical axis (blue inset), while the least visible improvement appears at the edge of the image field (red and green insets). The anisotropy in the resolving power can be traced back to the field dependence in the resolving power of the baseband PSF.

Uneven brightness: This may be confirmed by comparing the brightness of the central portion of the super-resolved image to the surroundings. This behavior is attributed to the field dependence of the DC (zero-frequency) response of the camera optical blur. It is observed that the baseband image of a constant albedo target exhibits intensity variations in accord with the severity of space variance in the camera optical blur.

Ghost artifacts: The impact of ghosting on the quality of reconstruction is most apparent in the red insets. The severity of ghosting increases with increasing distance from the optical axis. In addition, the direction of ghosting appears to depend on the modulation direction. Both observations are consistent with the findings reported in Section 3.A.2.

In summary, space variance in the optical blur introduces

• field dependence on the resolving power of the computationally engineered imager, and
• ghost artifacts in the super-resolved image.

In Section 5, we examine alternative illumination and reconstruction strategies to simultaneously overcome the above issues.

3. Limitations

A constant criticism levied against heterodyning-based super-resolution schemes is the purported need for the cutoff frequency of the illumination optics to exceed the cutoff frequency of the imaging optics. This begs the obvious question: “why not use the illumination optics for imaging?”

We observe that the criticism is ill-conceived as the optical design tradeoff associated with the illumination optics are markedly different from those associated with the imaging optics. The illumination optics need only project a specific set of frequencies, a task that can be accomplished using a variety of techniques such as sparse aperture synthesis and Talbot projection. In contrast, the imaging optics are by design required to resolve extended objects with spatial frequencies spanning the entire optical passband.

4. DEPTH ESTIMATION USING SINUSOIDAL PATTERNS

The proposed approach to recovering depth is a straightforward adaptation of phase measurement profilometry (PMP) techniques [35] to macroscopic scales. The principal difference lies in the additional restrictions $(b_{Z} = 0)$ that are imposed on the placement of the camera and projector. The restrictions help expand the scope of PMP techniques to accommodate a broader class of illumination patterns, such as warped sinusoids.

PMP approaches recover densely sampled topographic information from images of a scene illuminated by one or more periodic sinusoidal patterns. Depth estimation is predicated on the observation of scene-dependent phase deformations in the camera image of the projected illumination pattern. It was established in Section 2.C that a select combination of stereo arrangements and sinusoidal patterns support phase modulation of a carrier [Eqs. (21) and (22)], as favored by PMP.

An examination of Eqs. (21) and (22) confirms that the instantaneous phase $φ (u, v)$ of the detected illumination pattern is proportional to the inverse of the scene depth ( $W^{- 1}$ ). This observation constitutes the basis of a depth estimation strategy. Topographic information is recovered from the phase map $φ$ by eliminating the linear phase term and the constant phase offset $φ_{0}$ .

The process begins with the acquisition of images under phase-shifted sinusoidal or warped sinusoidal patterns ${p_{θ} (\overset{´}{x}, \overset{´}{y}) : θ \in [0, π / 2, π, 3 π / 2]}$ of frequency $ξ_{0}, η_{0}$ . The camera images are then digitally recombined to obtain the cosine/sine modulated images of the scene, as outlined in Eqs. (26)–(28). The resulting expressions are disclosed below:

{\hat{i}}_{\cos} (x, y) \overset{def}{=} | B (x, y) | \cos (\hat{φ} (x, y)) r (x, y),

{\hat{i}}_{\sin} (x, y) \approx | B (x, y) | \sin (\hat{φ} (x, y)) r (x, y),

wherein

\hat{φ} (x, y) \approx 2 π κ_{o} (ξ_{0} x + η_{0} y) + φ_{0} + 2 π {\hat{Z}}^{- 1} κ_{d} (ξ_{0} b_{X} + η_{0} b_{Y}) .

Closer inspection of the expression for the cosine/sine images in Eqs. (35) and (36) exposes notable omissions: blurring due to the imaging optics and the phase distortion due to the illumination blur.

As is common practice in PMP, the effect of camera optical blur is disregarded in our approach. The practice is admissible when the primary objective is to recover qualitative instead of quantitative depth information. The assumption of well-corrected illumination optics enables us to omit phase distortion due to the illumination blur.

The trigonometric identity $\tan^{- 1} (\frac{\sin (Q - P)}{\cos (Q - P)}) = \mod (Q - P, 2 π)$ may be used to isolate the scene-dependent component of the phase from the full phase map $\hat{φ}$ . Substituting $P = 2 π κ_{o} (ξ_{0} x + η_{0} y) + φ_{0}$ and $Q = \hat{φ} (x, y) = P + [2 π κ_{d} {\hat{Z}}^{- 1} (ξ_{0} b_{X} + η_{0} b_{Y})]$ into the above trigonometric identity, yields the following expression for the scene-dependent phase:

{\hat{φ}}_{wrapped} (x, y) \overset{def}{=} \mod (2 π κ_{d} {\hat{Z}}^{- 1} (ξ_{0} b_{X} + η_{0} b_{Y}), 2 π) .

The

2 π

phase ambiguity in Eq. (37) emerges from the phase ambiguity associated with the arctangent function. In order to resolve the phase ambiguity, we employ two sets of illumination patterns with harmonically related spatial frequencies. The first set of sinusoidal patterns (spatial frequency

ξ_{low}, η_{low}

) are designed to yield a phase map

{\hat{φ}}_{low} (x, y)

that is devoid of phase wrapping artifacts. The second set of sinusoidal patterns (spatial frequency

ξ_{high} = F ξ_{low}, η_{high} = F ξ_{low}

) yields a second phase map

{\hat{φ}}_{high} (x, y)

that is unwrapped using

{\hat{φ}}_{low} (x, y)

, as shown below:

{\hat{φ}}_{unwrapped} = {\hat{φ}}_{high} + 2 π round (\frac{F {\hat{φ}}_{low} - {\hat{φ}}_{high}}{2 π}) .

The above unwrapping scheme relies on the fact that the scene-dependent phase deformations induced by parallax vary linearly with the spatial frequency of the illuminating pattern. A topographic map of the scene can be obtained by computing the per-pixel inverse of the unwrapped phase map, that is,

\hat{Z} \propto {({\hat{φ}}_{unwrapped} (x, y))}^{- 1}

. Figure 20 summarizes the key steps in the proposed workflow for estimating scene depth in canonical/collocated stereo arrangements.

Fig. 20. Estimating depth in canonical/collocated stereo arrangement.

Download Full Size | PDF

A. Advantages Over PMP

As stated previously, the additional restrictions on camera and projector placement imposed by the proposed approach enables the use of a broader class of patterns for depth estimation. An example is the use of warped sinusoidal patterns in collocated stereo arrangements. The warped structure of the sinusoidal illumination pattern compensates for the relative rotation between the camera and projector optical axes. In practice, it is observed that the warping helps avoid fringe bunching/aliasing in the detected illumination pattern. It also helps circumvent the need for an extensive intrinsic/extrinsic calibration of the stereo arrangement, as in standard PMP. Additional details are available in Section 6.3 of [4].

B. Experimental Validation

In the interest of brevity, a single experimental result demonstrating the feasibility of depth estimation using warped sinusoidal patterns is examined below. A more exhaustive list of experiments validating the proposed approach in a variety of monostatic and bistatic arrangements is compiled in [4,44] and [48].

The collocated stereo apparatus of Fig. 15 is tasked with estimating scene depth. The process begins with the acquisition of images $i_{θ} (x, y)$ under the phase-shifted illumination patterns shown below:

p_{θ}^{low} (\overset{´}{x}, \overset{´}{y}) = \frac{1}{2} + \frac{1}{2} \sin (2 π ξ_{0}^{low} (\frac{{\overset{´}{π}}_{11} \overset{´}{x} + {\overset{´}{π}}_{12} \overset{´}{y} + {\overset{´}{π}}_{13}}{{\overset{´}{π}}_{31} \overset{´}{x} + {\overset{´}{π}}_{32} \overset{´}{y} + {\overset{´}{π}}_{33}}) + θ), p_{θ}^{high} (\overset{´}{x}, \overset{´}{y}) = \frac{1}{2} + \frac{1}{2} \sin (2 π ξ_{0}^{high} (\frac{{\overset{´}{π}}_{11} \overset{´}{x} + {\overset{´}{π}}_{12} \overset{´}{y} + {\overset{´}{π}}_{13}}{{\overset{´}{π}}_{31} \overset{´}{x} + {\overset{´}{π}}_{32} \overset{´}{y} + {\overset{´}{π}}_{33}}) + θ),

wherein

{\begin{matrix} ξ_{0}^{low} = \frac{1}{2592} \frac{cyc}{pixel} \\ ξ_{0}^{high} = \frac{1}{18} \frac{cyc}{pixel} \end{matrix}}, θ \in [0, π / 2, π, 3 π / 2], {\begin{matrix} x \in [1,1400] \\ \overset{´}{y} \in [1,1050] \end{matrix}} .

The camera images acquired under the aforementioned set of illumination patterns are processed using the workflow of Fig. 20.

The frequency of the demodulation matches the frequency of the illumination pattern $(ξ_{0}, η_{0})$ , since the warping of the sinusoidal illumination pattern accounts for differences in the magnification of the imaging and illumination paths, that is, $κ_{o} = 1$ in Eq. (36).

The first set of sinusoidal patterns with spatial frequency $(ξ_{0}^{low}, 0)$ yield a phase map ${\hat{φ}}_{low} (x, y)$ that is devoid of phase wrapping artifacts. The second set of sinusoidal patterns with spatial frequency $(ξ_{0}^{high}, 0)$ yields a second phase map ${\hat{φ}}_{high} (x, y)$ , which is unwrapped using ${\hat{φ}}_{low} (x, y)$ . The topographic map of the scene is derived from the unwrapped phase is illustrated in Panel 1 of Fig. 6.

The range resolution of a topographic measurement technique is an important figure-of-merit that quantifies the smallest discernible difference in scene depth $(Δ Z)$ . The image insets in Panel 1 of Fig. 6 provide a visual assessment of the range resolution of the collocated stereo apparatus. The red stripe overlaid on the camera image depicts one of the many isophase contours of the sinusoidal illumination pattern. Inspection of the blue inset confirms the presence of phase distortion in a single stripe pattern, as it strikes a 1 mm thick LEGO fence at a standoff distance of 1 m from the camera COP.

The analytical expression for the range resolution of the proposed depth estimation strategy is available in Appendix B of [4].

C. Why Not Sinusoids?

The overwhelming popularity of sinusoidal illumination patterns in super-resolution using heterodyning stems from the inherent simplicity of the reconstruction algorithm and the intuition associated with the modulation and demodulation processes. In practice, the presence of artifacts in the reconstructed image limits the utility of sinusoids to well-corrected optics characterized by largely space-invariant blur. As stated previously, the use of countably finite sinusoidal patterns in conjunction with space variance in the optical blur introduces:

• field dependence in the resolving power of the computationally engineered imager, and
• ghost artifacts in the super-resolved image.

The next section examines alternative illumination and reconstruction strategies to overcome the above issues.

5. ALTERNATIVES TO SINUSOIDAL ILLUMINATION

Heterodyning is an inescapable consequence of spatially patterning the illumination that is incident on the object volume. The fact may be confirmed by re-examining the light transport model derived in Section 2.B. Inspection of the expression for the detector irradiance in Eq. (19) indicates that the camera image is a blurred representation of the geometric image of the scene that is subject to AM.

On account of the linearity of the light transport mechanism, the expression for detector irradiance disclosed in Eq. (19) may be readily extended to accommodate any combination of sinusoids and, in general, arbitrary patterns. This suggests that any spatial pattern besides flood illumination will likely heterodyne object information into the passband of the collection optics. The observation serves as the guiding principle for our subsequent efforts to design illumination patterns. Ideas from coding theory and communication theory were in the design and selection of illumination patterns with desirable properties. The carefully chosen patterns help restore the heterodyned frequencies, which are otherwise smeared across the imager passband, to the correct position outside the passband. The smearing results from the modulation of object detail with the multitude of frequencies in the illumination pattern.

A. Assumptions

The remainder of this document focuses on super-resolving macroscopic scenes in multiple orientations. This imposes restrictions on the placement of the camera and projector. Previous discussions inform us of two stereo arrangements that support super-resolution in multiple orientations. These include a coincident stereo arrangement and a bistatic stereo arrangement observing 2D scenes lacking depth variation from the camera and projector perspective.

Without loss of generality, subsequent discussions assume that the illumination pattern incident on the target has a finite extent $(M \overset{´}{Δ} \times N \overset{´}{Δ})$ and is expressible as a superposition of non-overlapping spots of size $\overset{´}{Δ} \times \overset{´}{Δ} μm$ . Such a pattern admits the following decomposition:

p (\overset{´}{x}, \overset{´}{y}) = \sum_{k, ℓ = 0}^{\binom{M - 1}{N - 1}} ψ [k, ℓ] g (\overset{´}{x} - k \overset{´}{Δ}, \overset{´}{y} - ℓ \overset{´}{Δ}),

wherein the term

g

represents a top-hat function (rect function) with intensity that is modulated by the discrete sequence

ψ

.

The aforementioned constraints on the illumination pattern may be satisfied using multiple approaches including projection of the spatial pattern using well-corrected optics, the use of diffractive optical elements as pattern generators, and rapid scanning of a modulated Gaussian beam across the imager field of regard. The last two options are attractive in that they allow us to dispense with the need for wide-field projection optics with exacting spot size requirements.

B. Mathematical Preliminaries

Our analysis begins by re-examining the expressions disclosed in Eqs. (23) and (24) for the detector radiance in a coincident and bistatic stereo arrangement. These expressions may be generalized to accommodate an arbitrary superposition of sinusoids on account of the linearity of the light transport mechanism. The result is included below:

i (x, y) = \iint p (u, v) r (u, v) h_{cam} (x - u, y - v; u, v) d u d v .

The term

p (u, v)

represents the camera image of the projected pattern while

r (u, v)

represents the geometric image of the scene under uniform illumination. Please note in this case the term

p (u, v)

subsumes the effect of blurring due to the illumination optics. This blur is space invariant on account of the constraints imposed in previous paragraphs. The space variance arising from defocus may be disregarded when dealing with 2D scenes. Additionally, the use of beam scanning solutions to illuminate the object volume enable us to disregard space variance in the illumination blur because of the limited beam divergence at standoff.

In the event that the image sensor out-resolves the imaging optics, the camera image of Eq. (41) may be resampled so that the inter-sample-spacing matches that of the illumination pattern $(\overset{´}{Δ})$ , and the respective sampling grids are aligned. The expression for the resampled camera image may be recast in the projector coordinates using the coordinate mapping of Eq. (9) and the infinite homography ${\overset{´}{H}}^{\infty} = K {\overset{´}{K}}^{- 1}$ . The choice reflects restrictions imposed by the coincident and bistatic stereo arrangements considered in this discussion.

The final expression for the resampled camera image is provided below:

i (\overset{´}{x}, \overset{´}{y}) = κ_{o}^{- 2} \iint p (\overset{´}{u}, \overset{´}{v}) \tilde{r} (\overset{´}{u}, \overset{´}{v}) {\tilde{h}}_{cam} (\overset{´}{x} - \overset{´}{u}, \overset{´}{y} - \overset{´}{v}; \overset{´}{u}, \overset{´}{v}) d \overset{´}{u} d \overset{´}{v} .

The terms

\tilde{r}

and

{\tilde{h}}_{cam}

represent the geometric image of the scene and the camera optical blur, as expressed in the projector coordinates. The operations encapsulated in Eqs. (41) and (42) are realized in practice by warping the camera image using the infinite homography

{\overset{´}{H}}^{\infty} = K {\overset{´}{K}}^{- 1}

. The warp accommodates differences in the transverse magnification of the imaging and illumination paths and differences in the sampling phase of the imaging and illumination grids.

The above expression is the starting point for the investigation into alternative illumination patterns and reconstruction strategies.

C. Super-Resolution by Spatio-Temporal Coding

The first approach draws inspiration from the spread spectrum modulation schemes in communication theory. The concept is illustrated in Fig. 21. An intuitive description of the approach is provided first, before delving into the mechanics of operation.

Fig. 21. Super-resolution by spread spectrum modulation.

Download Full Size | PDF

The idea is to assign a temporally unique code to neighboring scene points by patterning the illumination. Following optical blurring, these codes will smear into each other. The smearing is undone by demodulating each pixel with the corresponding code.

The success of the approach hinges on the uniqueness of the code. Codes, such as code division multiple access (CDMA) spreading codes and perfect binary sequences, that exhibit compact correlation properties (delta correlated to be precise) are suitable candidates. The smearing of codes resulting from optical blurring can be undone by exploiting the fact that the codes are uncorrelated with translated copies. This property forms the basis of our reconstruction strategy.

As stated previously, super-resolution using heterodyning is predicated by the observation of unresolved spatial frequencies in the image $i (\overset{´}{x}, \overset{´}{y})$ obtained under patterned illumination. Fourier analysis of the product $p (\overset{´}{u}, \overset{´}{v}) \tilde{r} (\overset{´}{u}, \overset{´}{v})$ in Eq. (42) confirms that modulation smears the object spectrum across the optical passband. The challenge lies in undoing the smearing induced by AM. The proposed solution exploits the modulation diversity afforded by circularly shifting the illumination pattern $p (\overset{´}{u}, \overset{´}{v})$ in increments of the feature size $\overset{´}{Δ}$ . The super-resolved image is assembled by demodulating the temporal sequence of images acquired under translations of the illumination pattern $p (\overset{´}{u}, \overset{´}{v})$ and accumulating the result.

The reconstruction process outlined above mimics the operation of a correlation receiver. The expression for the reconstructed image is disclosed below:

i_{recon} (\overset{´}{x}, \overset{´}{y}) = κ_{o}^{- 2} \sum_{s, t = 0}^{\binom{M - 1}{N - 1}} {\begin{matrix} p_{s, t} (\overset{´}{x}, \overset{´}{y}) \\ [\iint p_{s, t} (\overset{´}{u}, \overset{´}{v}) \tilde{r} (\overset{´}{u}, \overset{´}{v}) {\tilde{h}}_{cam} (\overset{´}{x} - \overset{´}{u}, \overset{´}{y} - \overset{´}{v}; \overset{´}{u}, \overset{´}{v}) d \overset{´}{u} d \overset{´}{v}] \end{matrix}},

wherein

p_{s, t} (\overset{´}{x}, \overset{´}{y}) = \sum_{k, ℓ = 0}^{\binom{M - 1}{N - 1}} ψ [\begin{matrix} \mod (k - s, M) \\ \mod (ℓ - t, N) \end{matrix},] g (\overset{´}{x} - k \overset{´}{Δ}, \overset{´}{y} - ℓ \overset{´}{Δ}) .

The summation in Eq. (43) may be simplified by incorporating the definition of

p_{s, t} (\overset{´}{x}, \overset{´}{y})

from Eq. (40). The result is disclosed below:

\sum_{s, t = 0}^{\binom{M - 1}{N - 1}} p_{s, t} (\overset{´}{x}, \overset{´}{y}) p_{s, t} (\overset{´}{u}, \overset{´}{v}) = \sum_{k, ℓ = 0}^{\binom{M - 1}{N - 1}} \sum_{m, n = 0}^{\binom{M - 1}{N - 1}} {\begin{matrix} (\sum_{s, t = 0}^{\binom{M - 1}{N - 1}} ψ [\begin{matrix} \mod (k - s, M) \\ \mod (ℓ - t, N) \end{matrix},] ψ [\begin{matrix} \mod (m - s, M) \\ \mod (n - t, N) \end{matrix},]) \\ g (\overset{´}{x} - m \overset{´}{Δ}, \overset{´}{y} - n \overset{´}{Δ}) g (\overset{´}{u} - k \overset{´}{Δ}, \overset{´}{v} - ℓ \overset{´}{Δ}) \end{matrix}},

The term enclosed in round brackets represents the circular correlation of the discrete sequence

ψ

. In the special case that the discrete sequence is delta correlated, Eq. (44) simplifies as follows:

\sum_{s, t = 0}^{\binom{M - 1}{N - 1}} p (\overset{´}{x} - s \overset{´}{Δ}, \overset{´}{y} - t \overset{´}{Δ}) p (\overset{´}{u} - s \overset{´}{Δ}, \overset{´}{v} - t \overset{´}{Δ}) = \sum_{k, ℓ = 0}^{\binom{M - 1}{N - 1}} g (\overset{´}{x} - k \overset{´}{Δ}, \overset{´}{y} - ℓ \overset{´}{Δ}) g (\overset{´}{u} - k \overset{´}{Δ}, \overset{´}{v} - ℓ \overset{´}{Δ}) .

Further simplification is possible by exploiting the finite extent of the top-hat function

g

. It is observed that the terms

g (\overset{´}{x} - k \overset{´}{Δ}, \overset{´}{y} - ℓ \overset{´}{Δ})

,

g (\overset{´}{u} - k \overset{´}{Δ}, \overset{´}{v} - ℓ \overset{´}{Δ})

within the summand are strictly non-zero over the intervals

\overset{´}{x} \in (k \overset{´}{Δ} \pm \frac{1}{2} \overset{´}{Δ})

,

\overset{´}{y} \in (ℓ \overset{´}{Δ} \pm \frac{1}{2} \overset{´}{Δ})

and

\overset{´}{u} \in (k \overset{´}{Δ} \pm \frac{1}{2} \overset{´}{Δ})

,

\overset{´}{v} \in (q \overset{´}{Δ} \pm \frac{1}{2} \overset{´}{Δ})

. As a result, the product of these functions is non-zero over

| \overset{´}{x} - \overset{´}{u} | \leq Δ, | \overset{´}{y} - \overset{´}{v} | \leq Δ

. The above finding may be recast in terms of the definition of the top-hat function

g

to obtain the following result:

\sum_{s, t = 0}^{\binom{M - 1}{N - 1}} p (\overset{´}{x} - s \overset{´}{Δ}, \overset{´}{y} - t \overset{´}{Δ}) p (\overset{´}{u} - s \overset{´}{Δ}, \overset{´}{v} - t \overset{´}{Δ}) = g (\frac{\overset{´}{x} - \overset{´}{u}}{2}, \frac{\overset{´}{y} - \overset{´}{v}}{2}) .

Incorporating the above result into the expression for the reconstructed image yields the following revised expression for the reconstructed image:

i_{recon} (\overset{´}{x}, \overset{´}{y}) = \iint \tilde{r} (\overset{´}{u}, \overset{´}{v}) g (\frac{\overset{´}{x} - \overset{´}{u}}{2}, \frac{\overset{´}{y} - \overset{´}{v}}{2}) {\tilde{h}}_{cam} (\overset{´}{x} - \overset{´}{u}, \overset{´}{y} - \overset{´}{v}; \overset{´}{u}, \overset{´}{v}) d \overset{´}{u} d \overset{´}{v} .

Notice that the reconstructed image bears a strong resemblance to the image acquired with the engineered PSF:

{\tilde{h}}_{engd} (\overset{´}{x}, \overset{´}{y}; \overset{´}{u}, \overset{´}{v}) = g (\frac{\overset{´}{x}}{2}, \frac{\overset{´}{y}}{2}) {\tilde{h}}_{cam} (\overset{´}{x}, \overset{´}{y}; \overset{´}{u}, \overset{´}{v}) \approx g (\frac{\overset{´}{x}}{2}, \frac{\overset{´}{y}}{2}) .

The approximation in Eq. (48) from the fact that the transverse extent of the top-hat function

g

is almost always smaller than the transverse extent of the optical PSF

{\tilde{h}}_{cam}

. This suggests that the computationally engineered imager has near isotropic resolving power throughout the image field. Consequently, the resolving power of our computational imager is decoupled from the resolving power of the collection optics.

According to Eq. (48), the resolving power of the computationally engineered imager is twice the support of the top-hat function $g$ , that is, $2 \overset{´}{Δ}$ . This claim is consistent with the fact that the finest frequency in the illumination pattern is limited to $0.5 {\overset{´}{Δ}}^{- 1}$ .

1. Limitations

In practice, the number of scene points in the imager field of regard can quickly outnumber the available number of unique codes. In such cases, it is recommended that the code be reused, in accord with the worst-case spot size of the camera optical blur. A simple method for code reuse involves tiling the code to span the illumination field. The tiling introduces a periodicity in the illumination pattern so that the illumination pattern admits a Fourier series decomposition. Illuminating the scene with the said pattern amounts to simultaneous modulation with a set of harmonically related sinusoids in multiple orientations. The challenge lies in disambiguating the contributions of each of the carrier components. The use of a stochastic code with compactly supported correlation helps isolate the contribution of each carrier component.

2. Experimental Validation

The feasibility of super-resolution using spatio-temporal coding is verified by attempting to improve the resolving power of a well-corrected optic observing an ISO chart. The camera and projector used in the experiment are identical to those used in Fig. 15. The difference lies in the use of bistatic stereo arrangement for the present experiment.

The standoff distance from the camera to the target is approximately 0.67 m, while that from the projector is approximately 0.21 m. The short throw of the projector necessitated the movement of the projector closer to the target to realize higher resolution gains.

The ISO target is super-resolved by exploiting the modulation diversity afforded by the use of a single pseudo-random binary pattern and a continuum of phase shifts. The pseudo-random pattern is obtained by tiling a $15 \times 17$ minimum length sequence across the illumination field. Care is taken to ensure that the size of the binary pattern exceeded the spot size of the camera optical blur.

A super-resolved image is assembled by carefully demodulating the temporal sequence of images captured under $17 \times 15 = 255$ shifts of the master illumination pattern. The exposure time associated with each measurement is 62.5 ms. An animation of the reconstruction process is included in the supplementary material [43].

Results from the experiment are tabulated in Fig. 22. Closer inspection of the resolution chart elements indicates a resolution gain in excess of 3.5. A more accurate estimate of $3.8 \times$ is obtained from the SFR plots of Fig. 23. Examination of the SFR plots reveals variations in the saggital and meridional SFR of the engineered imager. It is observed that the anisotropy stems from the anisotropy in the SFR of the projection optics.

Fig. 22. Results from attempt to super-resolve a well-corrected optic using a stochastic illumination pattern. (Zoom-in for a closer view.)

Download Full Size | PDF

Fig. 23. Spatial frequency response (SFR) plots.

Download Full Size | PDF

The images in Panel 2 of Fig. 6 were also obtained as part of this experiment.

3. Broader Impact

A key ingredient in super-resolution using stochastic illumination is the correlation receiver used to assemble the super-resolved image. This notion exploits the modulation diversity afforded by the use of a single spatial pattern and a continuum of phase shifts. This notion can be readily extended to support any spatial pattern suggesting the possibility of super-resolution using any spatial pattern. The corresponding expression for the engineered PSF is given by ${\tilde{h}}_{engd} (\overset{´}{x}, \overset{´}{y}; \overset{´}{u}, \overset{´}{v}) = χ_{p p} (\overset{´}{x} - \overset{´}{u}, \overset{´}{y} - \overset{´}{v}) {\tilde{h}}_{cam} (\overset{´}{x}, \overset{´}{y}; \overset{´}{u}, \overset{´}{v})$ , where $χ_{p p}$ denotes the auto-correlation of the master illumination pattern. In the special case that the master illumination pattern is a pure sinusoid, the auto-correlation is also a sinusoid with the same periodicity. In this sense, the correlation receiver may be viewed as a general reconstruction strategy supporting a wide class of patterns ranging from sinusoids to stochastic patterns.

It is observed that the resolving power of the aforementioned computational imager is limited by the support of the auto-correlation of the master illumination pattern. This claim is consistent with the observation that flood illumination yields no improvement in resolution, and delta-correlation yields maximal improvement in resolution.

D. Super-Resolution Using Lattice Illumination

Migrating from sinusoidal illumination patterns to stochastic illumination patterns has allowed us to engineer computational imagers with resolving power that is decoupled from that of the collection optics. Still, undersampling at the sensor presents a challenge, even more so in the case of spread spectrum modulation. The final approach examined in this work tackles the aforementioned problem.

It is common knowledge that the resolution requirements to image a sparse collection of point emitters is substantially weaker than the requirements for imaging extended objects. In the former case, the point emitters need only be separated by the two-point resolution for exact reconstruction. In the latter case, the imaging optics needs to resolve all spatial frequencies that make up the extended object.

The aforementioned distinction between imaging sparse/extended objects forms the basis of our most expansive super-resolution strategy.

The idea is to convert the extended object volume into a sparse collection of secondary point emitters using patterned illumination. The pattern of choice is a single spot or a collection thereof, which is referred to as lattice illumination in this work. In the case of lattice illumination, care must be taken to ensure that the lattice spacing avoids overlap between the image of adjacent spots.

Following optical blurring, light from each illuminated object spot is distributed over one or more sensor pixels. By reintegrating the distributed light, we can simultaneously decouple the imager resolving power from the constraints imposed by the collection optics and the detector. Figure 24 illustrates the concept for a single spot.

Fig. 24. Super-resolution by spot(s) scanning/lattice illumination.

Download Full Size | PDF

The resolving power of the aforementioned computational imager is strictly limited by the spot size of the illumination optics. The expression for the reconstructed image, in the case of spot scanning, is disclosed in Eq. (49).

i_{recon} [s, t] \propto \sum_{s, t = 0}^{\binom{M - 1}{N - 1}} \iint {\begin{matrix} p_{s, t} (\overset{´}{u}, \overset{´}{v}) \tilde{r} (\overset{´}{u}, \overset{´}{v}) \\ \begin{matrix} (\iint {\tilde{h}}_{cam} (\overset{´}{x} - \overset{´}{u}, \overset{´}{y} - \overset{´}{v}; \overset{´}{u}, \overset{´}{v}) d \overset{´}{x} d \overset{´}{y}) \end{matrix} \end{matrix}} d \overset{´}{u} d \overset{´}{v},

wherein

p_{s, t} (\overset{´}{u}, \overset{´}{v}) \overset{def}{=} {\begin{matrix} g (s \overset{´}{Δ}, t \overset{´}{Δ}) & \overset{´}{u} \in (s \overset{´}{Δ} \pm \frac{1}{2} \overset{´}{Δ}), \overset{´}{v} \in (t \overset{´}{Δ} \pm \frac{1}{2} \overset{´}{Δ}) \\ 0 & otherwise \end{matrix} .

The inner integral represents light aggregation over multiple detector pixels, subsequent to imaging. In practice, the limits of integration need only be limited to an area larger than the worst-case spot size of the collection optics. The operation mimics light integration at a single pixel detector with an active area that exceeds the worst-case spot size.

The inner integral in Eq. (49) represents the integrated intensity of the camera optical PSF, which by definition is a field-independent constant. Incorporating this fact into the expression for the reconstructed image yields the following:

i_{recon} [s, t] \propto \iint g (\overset{´}{u} - s \overset{´}{Δ}, \overset{´}{v} - t \overset{´}{Δ}) \tilde{r} (\overset{´}{u}, \overset{´}{v}) d \overset{´}{u} d \overset{´}{v} .

Notice that the reconstructed image bears a strong resemblance to the image acquired with the engineered PSF, and

g

corresponds to the light distribution in a single spot.

The resolving power of the aforementioned computational imager is strictly limited by the feature size of the illumination pattern, which in this case is $\overset{´}{Δ} μm$ .

1. Connection with Super-Resolution Using Sinusoidal Patterns

Both the single spot and lattice illumination patterns admit a Fourier decomposition so that illumination of the scene amounts to simultaneous modulation with multiple sinusoids in different orientations. The challenge lies in disambiguating the contributions of each of the sinusoidal components. The modulation diversity afforded by scanning the spot and/or shifting the lattice helps isolate the contribution of each sinusoidal component.

2. Connection with Super-Resolution Using Spatio-Temporal Coding

The temporal sequence of lattice illumination patterns used to illuminate the scene may be interpreted as a unique binary temporal code that is assigned to each scene point in the illuminated field. The connection suggests that a correlation receiver may also be used to assemble the reconstructed image in lieu of the reconstruction strategy outlined above.

3. Connection with Super-Resolution Microscopy

The proposed technique could be viewed as an adaptation of super-resolution techniques [54–58], such as PALM/STORM, with the broader goal of accommodating space variance in the optical blur and detector undersampling.

4. Experimental Validation

The apparatus shown in Fig. 25 is tasked with the objective of producing high quality imagery using a 25 mm double convex lens and periodic pulse train illumination. The single lens imager is comprised of multiple optical components, as disclosed in the figure. The extension tube and spacers aid in controlling the position of the plane of sharp focus. The diameter of the iris mounted in front of the lens controls the severity of aberrations in the camera image.

Fig. 25. Super-resolving a singlet using lattice illumination.

Download Full Size | PDF

The use of a laser beam scanning (LBS) projector removes the need for high quality projection optics. The bistatic arrangement of the camera and the projector accommodates differences in the transverse magnification. Not shown in the figure is a moving diffuser (3M P/N: NVAG829233), which helps minimize the occurrence of speckle artifacts in the camera image.

The image in Fig. 26 illustrates the space variance in the PSF of the single lens imager over the illumination field. The image spans $499 \times 925 pixels (1.1 \times 2.03 mm)$ and represents the response of the single lens imager to a grid of squares (size of each $square = 3 \times 3$ projector pixels) projected by the LBS. The blur spots are asymmetrically distributed about the center of the image field. The asymmetry is attributed to the offset projection in the LBS.

Fig. 26. Space variance in the camera PSF over illuminated field.

Download Full Size | PDF

The process of super-resolution begins with the acquisition of images under integer pixel translations of the lattice illumination pattern disclosed below:

p_{s, t} (\overset{´}{x}, \overset{´}{y}) = \sum_{k = 1, ℓ = 1}^{848,480} ψ [k - s, ℓ - t] g (\overset{´}{x} - k \overset{´}{Δ}, \overset{´}{y} - ℓ \overset{´}{Δ}),

wherein

ψ [k, ℓ] \overset{def}{=} δ [\mod (k, 21), \mod (ℓ, 21)]

. The discrete sequence

ψ

represents a lattice of spots with a periodicity of 21 pixels in the horizontal and vertical directions. The lattice spacing of 21 pixels was empirically determined to be sufficient to avoid overlap in the image of adjacent light spots. The

441 (= 21 \times 21)

camera images acquired under lattice illumination are resampled using a geometric warp that accommodates difference in magnification of the imaging and illumination paths and compensates for field distortion in the projected pattern. The resampled camera images are multiplied with the respective illumination patterns, and the product accumulated to obtain the super-resolved image.

The outcome of super-resolution is tabulated in Panel 3 of Fig. 6. Inspection of the insets confirms that the proposed super-resolution strategy proposed here may be used to produce high quality imagery free of the annoying ghost artifacts that afflicted previous attempts (Section 3.C.2) to super-resolve a single lens imager.

The SFR plots of Fig. 27 help verify the claim of isotropic resolving power in the super-resolved image field. The saggital and tangential resolution of the engineered imager are largely field-independent. The asymmetry in the saggital and tangential resolving powers is attributed to the elliptical structure of the laser light spot.

Fig. 27. Spatial frequency response (SFR) plots.

Download Full Size | PDF

5. Multiscale Reconstruction

Unlike super-resolution using sinusoidal/stochastic patterns, each image acquired under lattice illumination may be independently reconstructed to recover scene information at select field locations. This implies that a single image acquired under lattice illumination may be reconstructed to directly provide a low-resolution (decimated to be more precise) reconstruction of the scene. The pixel pitch of the reconstruction is determined by the periodicity of the lattice, whereas the fill factor is determined by the size of the illuminated spot.

Additional images merely provide the sampling diversity needed to assemble partial reconstructions of increasing resolution. The process forms the basis of a multiscale reconstruction strategy, and it is illustrated in Panel 4 of Fig. 6.

The partial reconstructions in the image pyramid are obtained by progressively quadrupling the number of observations used to assemble the reconstructed image. The quadrupling represents an octave improvement in the sampling density.

Alternatively, one can digitally interpolate the intermediate images to obtain a synthetic reconstruction of the scene, albeit at full-resolution. However, the periodic structure of the lattice introduces sampling artifacts in the reconstruction. The use of a pseudo-random lattice pattern alleviates the problem. In practice, it is observed that as many as 50% of the observations are sufficient to obtain a high fidelity reconstruction of the scene. An example is provided in the insets of Panel 4. The peak signal to noise (PSNR) value of the mean squared error between the exact reconstruction and the interpolated partial reconstruction is also indicated for comparison.

Prior information about the target (such as sparsity in a known basis) could be used to lower the number of observations needed to assemble the high fidelity reconstruction.

6. Broader Impact

A unique characteristic of super-resolution using lattice illumination is that it is agnostic to the structure of the PSF of the collection optics. This provides us with an opportunity to encode novel information into the camera-side PSF. As an example, one could attempt to encode range information using the double helix PSF [59,60].

The monostatic variant of the stereo arrangement used in Fig. 25 bears a striking resemblance to scanning units in a variety of active EO (electro-optic) sensors such as LADAR and gated-viewing imagers. The principal distinction between these systems is the use of patterned illumination. It is envisioned that the capabilities of many EO sensors can be augmented to acquire high resolution imagery, with suitable modifications.

6. SUMMARY

The notion of active computational imaging espoused in this work is centered on the premise that imaging under pattern illumination affords capabilities not achieved using classical approaches. The present work has examined this notion within the context of circumventing resolution limits at macroscopic scales. The mathematical tools required to comprehend and extend the ideas described in this work have been furnished as part of the investigation. This includes a comprehensive model for light transport that assimilates established principles from physical optics and computer vision into a broader framework for imaging under patterned illumination.

A taxonomy of select active computational imaging concepts examined in this work are summarized in Fig. 28. The numerical code assigned to each branch in the map represents a specific combination of restrictions imposed on the illumination pattern, collection optics, and image sensor.

Fig. 28. Taxonomy of active computational imaging concepts explored in this work.

Download Full Size | PDF

Please note that the notion of engineering a prescribed PSF using sinusoidal illumination is not examined in this manuscript. Interested readers are referred to [4,61] for more details. Key takeaways from the broader investigation are summarized below: The camera image of an illumination pattern in any active stereo arrangement will exhibit amplitude and/or phase modulation. The AM in isolation is essential for super-resolution and contributes to improved transverse resolution. The phase modulation in isolation is essential for depth estimation and contributes to improved longitudinal resolution.

A variety of stereo arrangements support super-resolution and/or depth estimation. One embodiment includes collocated stereo arrangements wherein the projector COP is embedded within the camera entrance pupil plane. Illumination patterns whose iso-intensity lines are aligned with the epipolar lines on the camera side do not exhibit scene-dependent phase deformations, and they are therefore best suited for super-resolution. Alternatively, illumination patterns whose iso-intensity lines are not-aligned/perpendicular to the epipolar lines on the camera side exhibit scene-dependent phase deformations, and they are therefore best suited for depth estimation.

A second embodiment includes coincident stereo arrangements wherein the camera and projector optical axes and COP are aligned. These stereo arrangements support super-resolution in any orientation.

The collocated/coincident stereo arrangements discussed in this work bear a striking resemblance to the scanning units in electro-optics sensors such as LADAR, gated-viewing cameras, SL scanners, and profilometers. It is envisioned that the capabilities of these sensors may be augmented to acquire higher resolution imagery, with suitable modifications. Key takeaways from the investigation into super-resolution are summarized below: Space variance in the imager blur is not an impediment to super-resolution. The improvement in transverse resolution is affected by a computational engineering of the imager PSF. Resolution gain manifests as a reduction in the spot size or the central lobe width of the engineered PSF. In the case of sinusoidal patterns, maximal/zero resolution improvement occurs in/orthogonal to direction of modulation.

Any spatial pattern besides flood illumination will heterodyne unresolved spatial frequencies into the passband of the imaging optics. Exact restoration of the heterodyned frequencies is made possible by exploiting the modulation diversity afforded by the use of a single pattern and a continuum of phase shifts. The aforementioned reconstruction process mimics the operations of a correlation receiver.

A stochastic illumination pattern used in conjunction with a correlation receiver enables us to engineer computational imagers whose resolving power is decoupled from that of the collection optics. The resulting computational engineer exhibits isotropic resolving power throughout the image field.

Undersampling at the sensor complicates attempts to restore the heterodyned frequencies when using sinusoidal patterns paired with an AM receiver OR stochastic patterns paired with a correlation receiver.

Neither sinusoidal nor stochastic patterns support reconstruction from partial observations, as the reconstruction problem is ill-posed. The ill-posed nature of the reconstruction problem stems from the ambiguity in restoring the heterodyned frequencies to the correct positions outside the optical passband.

Super-resolution techniques relying on spot/lattice illumination can be used to engineer imagers whose resolving power is fully decoupled from the constraints imposed by the collection optics and sensor. The resulting computational engineer exhibits isotropic resolving power throughout the image field. Unlike other active super-resolution techniques, a single observation of the scene under lattice illumination yields a low-resolution representation of the scene. Lastly, super-resolution techniques relying on spot/lattice illumination support multiscale reconstruction and reconstruction from partial information.

A. Research Avenues

The principal shortcomings of our active computational imaging approach include: (a) sacrificing temporal bandwidth in an attempt to squeeze more information into the limited optical/detection bandwidth of the imager and (b) the resolution on target is fundamentally limited by the feature size of the illumination pattern, as seen on target.

The loss in bandwidth may be minimized by borrowing ideas from compressed sensing, specifically the notion of object sparsity in a basis set that is identified using the low-resolution image of the scene. It is anticipated that sparse-signal reconstruction techniques could be used in conjunction with lattice illumination to recover a super-resolved image of the scene, using far fewer observations than necessary for exact reconstruction.

Current efforts are devoted to tackling the second problem of feature size on target. Two distinct approaches to minimize the feature size on target are being examined. The first of these approaches exploits recent innovations in engineering optical beams that can squeeze light into a region smaller than the diffraction limit [62]. The reduction in spot size is accompanied by an increase in side lobes, which could be moved away from the central lobe, with some effort. The reduced spot size is useful for computational resolution enhancement, while the separation of the side lobes avoids coupling light into the image of the central spot. Both capabilities come at the expense of SNR, since an appreciable amount of light energy is redirected toward the side lobes.

The second approach to controlling the feature size on the target involves the use of a lensless projection scheme to shape the field distribution in the object volume. The approach contrasts the current practice of shaping the irradiance component. It exploits the notion that free space propagation preserves the bandwidth of a propagating field that is set up using a spatial light modulator (SLM). It can be shown that the feature size on target is limited only by the pixel pitch of the SLM.

A few other avenues for research are suggested below. There is a strong connection between the multiplicative structure of the engineered PSF in our work and the PSFs encountered in interferometric two-pupil synthesis [63] and sparse aperture synthesis [64]. Likewise, a non-physical/computational pupil function could be associated with our engineered PSF, suggesting a connection with pupil replication techniques [65,66]. Another avenue for research includes the examination of a holistic approach to the design of electro-optics systems featuring patterned illumination.

Funding

Army Research Laboratory (ARL) (W911NF-06-2-0035).

Acknowledgment

The authors wish to thank Dr. Vikrant Bhakta, Dr. Panos Papamichalis, and Dr. Manjunath Somayaji for their valuable help in shaping some of the early ideas described in the work.

The work described in this paper was sponsored by the ARL and was accomplished under Cooperative Agreement Number W911NF-06-2-0035. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the ARL or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon.

REFERENCES

1. D. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory 52, 1289–1306 (2006). [CrossRef]

2. E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Inf. Theory 52, 489–509 (2006). [CrossRef]

3. M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008). [CrossRef]

4. P. Rangarajan, “Pushing the limits of imaging using patterned illumination,” Ph.D. dissertation (Southern Methodist University, 2014).

5. W. Lukosz and M. Marchand, “Optischen Abbildung Unter Uberschreitung der Beugungsbedingten Auflosungsgrenze,” J. Mod. Opt. 10, 241–255 (1963).

6. M. G. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198, 82–87 (2000). [CrossRef]

7. M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. USA 102, 13081–13086 (2005). [CrossRef]

8. R. Heintzmann, T. M. Jovin, and C. Cremer, “Saturated patterned excitation microscopy–a concept for optical resolution improvement,” J. Opt. Soc. Am. A 19, 1599–1609 (2002). [CrossRef]

9. R. Heintzmann and G. Ficz, “Breaking the resolution limit in light microscopy,” Brief Funct. Genomic Proteomic 5, 289–301 (2006). [CrossRef]

10. S. A. Shroff, “Structured illumination imaging,” Ph.D. dissertation (University of Rochester, 2010).

11. J. T. Frohn, H. F. Knapp, and A. Stemmer, “True optical resolution beyond the Rayleigh limit achieved by standing wave illumination,” Proc. Natl. Acad. Sci. USA 97, 7232–7236 (2000). [CrossRef]

12. J. T. Frohn, “Super resolution fluorescence microscopy by structured light,” Ph.D. dissertation (Swiss Federal Institute of Technology, 2000).

13. J.-Y. Lin, R.-P. Huang, P.-S. Tsai, and C.-H. Lee, “Wide-field super resolution optical sectioning microscopy using a single spatial light modulator,” J. Opt. A 11, 015301 (2009). [CrossRef]

14. A. Stemmer, M. Beck, and R. Fiolka, “Widefield fluorescence microscopy with extended resolution,” Histochem. Cell Biol. 130, 807–817 (2008). [CrossRef]

15. A. Neumann, Y. Kuznetsova, and S. R. Brueck, “Structured illumination for the extension of imaging interferometric microscopy,” Opt. Express 16, 6785–6793 (2008). [CrossRef]

16. V. Micó, Z. Zalevsky, and J. García, “Optical superresolution: imaging beyond Abbe’s diffraction limit,” J. Hologr. Speckle 5, 110–123 (2009). [CrossRef]

17. M. G. L. Gustafsson, “Extended resolution fluorescence microscopy,” Curr. Opin. Struct. Biol. 9, 627–628 (1999). [CrossRef]

18. Z. Zalevsky and D. Mendlovic, Optical Superresolution (Springer, 2004).

19. E. Barrett, D. W. Tyler, P. M. Payton, K. Ip, and D. N. Christie, “New approaches to image super resolution beyond the diffraction limit,” Proc. SPIE 6712, 67120D (2007). [CrossRef]

20. D. W. Tyler and E. B. Barrett, “Simulation of a passive grating-heterodyne super resolution concept,” Proc. SPIE 7094, 709403 (2008). [CrossRef]

21. A. Mudassar, A. R. Harvey, A. H. Greenaway, and J. Jones, “Band pass active aperture synthesis using spatial frequency heterodyning,” J. Phys. 15, 290–295 (2005).

22. E. Sabo, Z. Zalevsky, D. Mendlovic, N. Konforti, and I. Kiryuschev, “Superresolution optical system with two fixed generalized Damman gratings,” Appl. Opt. 39, 5318–5325 (2000). [CrossRef]

23. E. Ben-Eliezer, N. Konforti, and E. Marom, “Super resolution imaging with noise reduction and aberration elimination via random structured illumination and processing,” Opt. Express 15, 3849–3863 (2007). [CrossRef]

24. E. Ben-Eliezer and E. Marom, “Aberration-free superresolution imaging via binary speckle pattern encoding and processing,” J. Opt. Soc. Am. A 24, 1003–1010 (2007). [CrossRef]

25. Y. Shirai, “Recognition of polyhedron with a range finder,” Pattern Recognit. 4, 243–250 (1972). [CrossRef]

26. R. Nevita and T. O. Binford, “Structured descriptions of complex objects,” in Proceedings 3rd International Joint Conference on Artificial Intelligence (1973), pp. 641–647.

27. R. I. Popplestone, C. M. Brown, A. P. Ambler, and G. F. Crawford, “Forming models of plane-and-cylinder faceted bodies from light stripes,” in Proceedings 4th International Joint Conference on Artificial Intelligence (1975), pp. 664–668.

28. A. Shpunt and Z. Zalevsky, “Three-dimensional sensing using speckle patterns,” U.S. patent US20090096783 A1 (16 April 2009).

29. Z. Zalevsky, A. Shpunt, A. Maizels, and J. Garcia, “Method and system for object reconstruction,” U.S. patent WO2007043036 A1 (19 April 2007).

30. J. Salvi, J. Pages, and J. Batlle, “Pattern codification strategies in structured light systems,” Pattern Recognit. 37, 827–849 (2004). [CrossRef]

31. M. D. Altschuler, B. R. Altschuler, and J. Taboada, “Laser electro-optic system for rapid three-dimensional (3-D) topo-graphic mapping of surfaces,” Opt. Eng. 20, 953–961 (1981). [CrossRef]

32. E. Horn and N. Kiryati, “Toward optimal structured light patterns,” Image Vis. Comput. 17, 87–97 (1999). [CrossRef]

33. M. Halioua and R. S. Krishnamurthy, “Projection Moire with moving gratings for automated 3-D topography,” Appl. Opt. 22, 850–855 (1983). [CrossRef]

34. M. Takeda and K. Mutoh, “Fourier transform profilometry for the automatic measurement of 3-D object shapes,” Appl. Opt. 22, 3977–3982 (1983). [CrossRef]

35. V. Srinivasan, H. C. Liu, and M. Halioua, “Automated phasemeasuring profilometry: a phase mapping approach,” Appl. Opt. 24, 185–188 (1985). [CrossRef]

36. M. Chang and C. S. Ho, “Phase-measuring profilometry using sinusoidal grating,” Exp. Mech. 33, 117–122 (1993). [CrossRef]

37. P. S. Huang, S. Zhang, and F. P. Chiang, “Trapezoidal phase-shifting method for 3-D shape measurement,” Proc. SPIE 5606, 142 (2004). [CrossRef]

38. P. Jia, J. Kofman, and C. English, “Two-step triangular-pattern phase-shifting method for three dimensional object-shape measurement,” Opt. Eng. 46, 083201 (2007). [CrossRef]

39. A. Hornberg, Handbook of Machine Vision (Wiley-VCH, 2006).

40. M. Aggarwal and N. Ahuja, “A pupil-centric model of image formation,” Int. J. Comput. Vis. 48, 195–214 (2002). [CrossRef]

41. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed. (Cambridge University, 2003).

42. Y. Ma, S. Soatto, and J. Koeck, An Invitation to 3-D Vision: From Images to Geometric Models (Springer-Verlag, 2004).

43. V. N. Mahajan, Optical Imaging and Aberrations. Part I: Ray Geometrical Optics (SPIE, 1998), Chaps. 4 and 5.

44. P. Rangarajan, I. Sinharoy, P. Papamichalis, and M. Christensen, “Pushing the limits of digital imaging using structured illumination,” in International Conference on Computer Vision (2011), pp. 1315–1322

45. E. Hecht, Optics (Addison Wesley, 2002), Chap. 6.

46. A. W. Lohmann and D. P. Paris, “Space-variant image formation,” J. Opt. Soc. Am. 55, 1007–1013 (1965). [CrossRef]

47. D. A. Vaquero, R. Raskar, R. S. Feris, and M. Turk, “A projector-camera setup for geometry-invariant frequency demultiplexing,” in Computer Vision and Pattern Recognition (2009), pp. 2082–2089

48. P. Rangarajan, V. Bhakta, M. Christensen, and P. Papamichalis, “Perspective imaging under structured light,” in European Conference on Computer Vision, September 2010, pp. 405–419.

49. Link to supplementary material: http://lyle.smu.edu/~prangara/pubs/AO_Manuscript/.

50. L. Zhang and S. K. Nayar, “Projection defocus analysis for scene capture and image display,” ACM Trans. Graph. 25, 907–915 (2006). [CrossRef]

51. P. So, H. Kwon, and C. Dong, “Resolution enhancement in standing-wave total internal reflection microscopy: a point-spread-function engineering approach,” J. Opt. Soc. Am. A 18, 2833–2845 (2001). [CrossRef]

52. P. Rangarajan, V. Bhakta, and M. Christensen, “Surpassing the diffraction limit of digital imaging systems using sinusoidal illumination patterns,” in Computational Optical Sensing and Imaging, OSA Technical Digest (CD) (Optical Society of America, 2009), paper CTuC4.

53. P. Rangarajan and P. Papamichalis, “Estimating homographies without normalization,” in 16th IEEE International Conference on Image Processing (ICIP), November , 2009, pp. 3517–3520.

54. C. J. R. Sheppard, “Super-resolution in confocal imaging,” Optik 80, 53–54 (1988).

55. T. Wilson, R. Juškaitis, M. A. A. Neil, and M. Kozubek, “Confocal microscopy by aperture correlation,” Opt. Lett. 21, 1879–1881 (1996). [CrossRef]

56. C. B. Müller and J. Enderlein, “Image scanning microscopy,” Phys. Rev. Lett. 104, 198101 (2010). [CrossRef]

57. A. G. York, S. H. Parekh, D. D. Nogare, R. S. Fischer, K. Temprine, M. Mione, A. B. Chitnis, C. A. Combs, and H. Shroff, “Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy,” Nat. Methods 9, 749–754 (2012). [CrossRef]

58. O. Schulz, C. Pieper, M. Clever, J. Pfaff, A. Ruhlandt, R. H. Kehlenbach, F. S. Wouters, J. Großhans, G. Bunt, and J. Enderlein, “Resolution doubling in fluorescence microscopy with confocal spinning-disk image scanning microscopy,” Proc. Natl. Acad. Sci. USA 110, 21000–21005 (2013). [CrossRef]

59. A. Greengard, Y. Y. Schechner, and R. Piestun, “Depth from diffracted rotation,” Opt. Lett. 31, 181–183 (2006). [CrossRef]

60. S. Quirin and R. Piestun, “Depth estimation and image recovery using broadband, incoherent illumination with engineered point spread functions,” Appl. Opt. 52, A367–A376 (2013). [CrossRef]

61. P. Rangarajan, M. Christensen, and P. Milojkovic, “Parsimony in PSF engineering using patterned illumination,” in Imaging and Applied Optics Postdeadline, OSA Postdeadline Paper Digest (Optical Society of America, 2013), paper CTh3C.1.

62. M. Mazilu, J. Baumgartl, S. Kosmeier, and K. Dholakia, “Optical Eigenmodes; exploiting the quadratic nature of the energy flux and of scattering interactions,” Opt. Express 19, 933–945 (2011). [CrossRef]

63. I. Leizerson, S. Lipson, and V. Sarafis, “Superresolution in far-field imaging,” J. Opt. Soc. Am. A 19, 436–443 (2002). [CrossRef]

64. N. Miller, M. Dierking, and B. Duncan, “Optical sparse aperture imaging,” Appl. Opt. 46, 5933–5943 (2007). [CrossRef]

65. B. Braunecker, R. Hauck, and W. Rhodes, “Pupil function replication in OTF synthesis,” Appl. Opt. 18, 44–51 (1979). [CrossRef]

66. J. Jahns and A. W. Lohmann, “OTF synthesis: low pass and band pass filtering realized by periodic replication of the pupil function,” Opt. Commun. 42, 231–236 (1982). [CrossRef]

Abstract

1. INTRODUCTION

A. Concept Imager

B. Principle

1. Surpassing the Diffraction Limit by Heterodyning

2. Depth Estimation Using Parallax

C. Assumptions

D. Highlights

2. MODEL FOR IMAGING UNDER PATTERNED ILLUMINATION

A. Geometric Model

1. Mapping from Projector ↔ Camera Pixel Coordinates

2. Limitations

B. Blur Model

1. Light Transport from the Scene to the Detector

2. Light Transport from the Projector to the Scene

3. Light Transport from the Projector to the Camera

Light transport under periodic sinusoidal illumination

Light transport under warped sinusoidal illumination

4. Highlights

5. Key Findings

6. Limitations

C. Useful Restrictions

1. Monostatic Arrangement for Periodic Sinusoidal Illumination

Constraints on stereo arrangement

Constraints on pattern orientation

2. Monostatic Arrangement for Warped Sinusoidal Illumination

Constraints on stereo arrangement

Constraints on pattern orientation

Broader impact

3. Monostatic Arrangement for Any Illumination Pattern

4. Bistatic Arrangement for Periodic Sinusoidal Patterns and Superpositions of Periodic Sinusoids

3. SUPER-RESOLUTION USING SINUSOIDAL PATTERNS

A. Demodulation Strategy

1. Frequency Domain Interpretation of Super-Resolution

2. Spatial Domain Interpretation of Super-Resolution

B. Aliasing Management

C. Experimental Validation

1. Super-Resolving Well-Corrected Optics

2. Super-Resolving Optics with Spatially Varying Blur

Key observations

3. Limitations

4. DEPTH ESTIMATION USING SINUSOIDAL PATTERNS

A. Advantages Over PMP

B. Experimental Validation

C. Why Not Sinusoids?

5. ALTERNATIVES TO SINUSOIDAL ILLUMINATION

A. Assumptions

B. Mathematical Preliminaries

C. Super-Resolution by Spatio-Temporal Coding

1. Limitations

2. Experimental Validation

3. Broader Impact

D. Super-Resolution Using Lattice Illumination

1. Connection with Super-Resolution Using Sinusoidal Patterns

2. Connection with Super-Resolution Using Spatio-Temporal Coding

3. Connection with Super-Resolution Microscopy

4. Experimental Validation

5. Multiscale Reconstruction

6. Broader Impact

6. SUMMARY

A. Research Avenues

Funding

Acknowledgment

REFERENCES

Cited By

Figures (28)

Equations (57)

Applied Optics

1. Mapping from Projector $\leftrightarrow$ Camera Pixel Coordinates