1. Introduction

Spatial filtering operations, where different frequency components of an image are selectively transmitted or blocked, play a key role in many high-impact applications in microscopy, photography, and computer vision [1]. In particular, edge detection by high-pass filtering allows for image sharpening as well as segmentation, to distill a highly compressed version of the original image that is easier to store, transmit, and process. In fact, the latter idea is at the core of the initial stage of the visual recognition process, where different filtered versions of the original image are produced for subsequent analysis. The same principle is also observed in the first layer of convolutional neural networks (CNNs), which have emerged as the leading algorithmic approach for many demanding applications in visual data processing such as image classification and object recognition [2]. These filtering operations can be readily implemented in the electronic digital domain – at the expense, however, of substantial power consumption and processing time. As a result, their adoption in many embedded and mobile edge-computing applications remains a significant challenge (e.g., in autonomous vehicles, augmented reality headsets, and robots, where power and bandwidth are highly constrained).

These considerations have created novel opportunities for optical computing solutions, which in fact are currently enjoying a substantial resurgence of interest [3]. Photonics intrinsically offers ultrafast processing bandwidths (essentially at the speed of light) and low power consumption (only limited by optical propagation losses). Of particular relevance in the context of spatial filtering are approaches based on Fourier optics [4], building on the well-known two-lens 4f imaging system. In this setup, the first lens projects the Fourier transform of the object field onto a pupil mask between the two lenses, where different spatial frequency components are multiplied by different transmission coefficients before they are recombined by the second lens to form a filtered image of the object. However, this system suffers from large form factor and strict alignment requirements, which again limit its portability. In recent years, several nanophotonic structures have been investigated as a means to provide similar functionalities (particularly image differentiation for edge detection) with more compact dimensions and enhanced design flexibility [5–18]. Specific examples include phase-shifted Bragg reflectors [6], plasmonic filters [8], gradient metasurfaces [7,12,13,16,17], diffraction gratings [10], and photonic crystal slabs [9,11,14,15], all designed to introduce a sharp in-plane-wavevector dependence in their free-space transmittance. The use of Fourier optical filters in conjunction with neural networks is also being explored extensively [19–21].

In the present work, we introduce a different approach where optical spatial filtering is achieved (on a pixel-by-pixel basis) with an image sensor array consisting of specially designed directional photodetectors. Specifically, we employ devices coated with plasmonic metasurfaces that only allow for the detection of light incident along a small set of directions (determined by the metasurface design), whereas light incident along all other directions is reflected. This novel capability has been demonstrated in recent work focused on a different imaging application [22], i.e., planar lensless compound-eye vision with ultrawide field of view. Similar devices can also be used as optical spatial filters, based on the notion that different spatial-frequency components of an illuminated object correspond to plane waves propagating from the object along different directions. Importantly, with this approach the filter transfer function can be tailored through the design of the metasurface, and different metasurfaces (i.e., different filters) can be applied on different adjacent pixels within the same image sensor array. As a result, multiple filtering operations can be performed simultaneously with the same pixel array. Furthermore, this approach does not require any external optical components other than a standard imaging lens, and therefore is particularly convenient in terms of system miniaturization and alignment simplicity.

In the directional image sensors described below, light incident at the target detection angles is selectively detected via resonant coupling to a guided plasmonic mode. As a result, sharp responsivity peaks at geometrically tunable angles are obtained, which are particularly well suited to engineer a wide range of transfer functions for high-contrast optical spatial filtering. In contrast, other devices previously used for angle-sensitive vision [23–25] feature a more gradual angular dependence and/or limited tunability. At the same time, the role of the guided modes in our directional photodetectors complicates the conceptual analogy with standard optical spatial filters. Therefore, in order to substantiate the image processing capabilities of these devices, here we develop a rigorous theoretical model that quantifies and clarifies the nature of their filter transfer function. Next, we combine this model with the experimental angle-resolved responsivity of prototype samples to demonstrate two examples of relevant spatial filtering functionalities.

In the first example, we use devices featuring high-pass filtering characteristics to visualize a transparent phase-only object, which would otherwise be invisible to standard image sensor arrays. Second, we address the task of spatial differentiation for edge detection of amplitude objects with incoherent (i.e., natural) illumination. It is well established from Fourier optics that, when the incident light is spatially incoherent, high-pass filtering is generally impossible with a single filter [4], which represents a key limitation of optical-domain image processing. A possible solution is to record two low-pass filtered images of the object of interest with different cutoff frequencies, and then compute their difference [26–28]. Here we present a particularly simple protocol to perform this task, based on a camera where low-pass-filtering directional photodetectors are combined with standard pixels in a checkerboard pattern. The same approach can be extended to implement more complex incoherent filtering operations that similarly require suppression of the low-frequency components, e.g., for object recognition.

2. Plasmonic directional image sensors

The physical structure and principle of operation of our devices are illustrated in Fig. 1(a). In these devices, the illumination window of a photodetector is coated with a composite metasurface consisting of an optically thick metal film (Au) stacked with a periodic array of Au rectangular nanostripes (grating lines) and perforated with sub-wavelength slits. Two dielectric layers (SiO₂) are also introduced immediately below and above the metal film, to provide electrical insulation from the active layer and to optimize the film-grating coupling, respectively [22]. Light incident at the desired detection angle is diffracted by the grating into surface plasmon polaritons (SPPs) at the top surface of the metal film. These guided waves are then scattered by the slits into radiation propagating predominantly into the absorbing active layer, similar to the phenomenon of extraordinary optical transmission through sub-wavelength apertures in metal films [29,30]. Correspondingly, a photocurrent signal is detected proportional to the SPP field intensity at the slit locations. Light incident along any other direction is instead either reflected or diffracted back into the air above.

 

Fig. 1. Plasmonic directional image sensors. (a) Schematic illustration of the physical structure and principle of operation for a device designed to provide angle-sensitive photodetection peaked at normal incidence. The device is illuminated through its top surface, as shown by the arrow over the grating. The polar plot shows the calculated optical transmission coefficient through the metasurface for p-polarized light at λ₀= 1550 nm versus angle of incidence θ on the x–z plane. (b) Scanning electron microscopy (SEM) image of an experimental sample (device A) showing a few periods of the grating and the adjacent slits. The scale bar is 2 µm. (c) Measured responsivity of the same device versus polar θ and azimuthal ϕ illumination angles at λ₀= 1550 nm, summed over two orthogonal polarizations. In this device, the two SiO₂ layers have a nominal thickness of 60 nm, the metal film consists of 5 nm of Ti and 100 nm of Au, and each grating line consists of 5 nm of Ti and 50 nm of Au with a width of 250 nm. The grating contains 15 lines with a period Λ = 1485 nm. Each slit section contains 5 slits with 200-nm width and 400-nm center-to-center spacing. (d) Same as (c) for a different sample (device B) featuring a symmetric double-peaked angular response with maximum photocurrent at θ = ±3.8°. The geometrical parameters of this device are nominally the same as in the sample of (c), except for a larger array period Λ = 1581 nm.

Download Full Size | PDF

Specifically, in the present work we use two devices where the grating is surrounded symmetrically by slits on both sides, leading to a symmetric angular response peaked either at normal incidence (device A) or at equal and opposite illumination angles (device B) depending on the grating period Λ. In passing, we note that the same design platform can also be used to produce an asymmetric angular response peaked at any desired off-axis angle, by replacing the slits on one side of the array with a suitable “reflector” unit [22]. While this configuration is not considered in the present work, it opens up additional filtering opportunities in the context of phase-contrast imaging, where asymmetric transfer functions are particularly beneficial [31].

The polar plot of Fig. 1(a) shows the p-polarized transmission coefficient at 1550-nm wavelength of the metasurface of device A, computed as a function of polar illumination angle θ on the x-z plane with finite difference time domain (FDTD) simulations. The device geometrical parameters are listed in the figure caption. A sharp transmission peak centered at θ = 0° (normal incidence) is observed in this plot, with full width at half maximum as small as 3° and peak value above 45%, originating from the excitation of SPPs propagating towards both sets of slits. For s-polarized incident light, the calculated transmission coefficient through the same metasurface is isotropic and significantly smaller, < 0.2% at all angles, consistent with the polarization properties of SPPs [22]. The angular response of Fig. 1(a), rescaled by a factor of about 0.5, therefore also applies to unpolarized illumination. Because of its reliance on diffraction, the device operation is also intrinsically wavelength dependent, and monochromatic light at λ₀ = 1550 nm is considered throughout this work. In practice, the same behavior can be obtained even under broadband illumination with the addition of a spectral filter on the top surface of the image sensor array. Additionally, it should be noted that both polarization-insensitive and achromatic broadband operation are also possible with the same general platform, by replacing the periodic grating with a gradient metasurface [32,33] or multilevel diffractive elements [34] and leveraging the enhanced design flexibility of such systems.

Our experimental samples consist of metal-semiconductor-metal (MSM) Ge photoconductors, with the metasurfaces just described patterned in the region between two metal contacts deposited on the top surface of a Ge substrate. Such photoconductors are particularly simple to fabricate, and the same results in terms of angular response can be expected with any other type of photodetectors (including image-sensor photodiodes). The metasurfaces were developed with the multi-step fabrication process described in Ref. 22, including electron-beam lithography for the slits and nanostripes [Fig. 1(b)]. The completed devices were characterized by measuring their photocurrent under laser light illumination at 1550-nm wavelength as a function of polar θ and azimuthal ϕ angles of incidence. In order to simplify these angle-resolved experiments (and to avoid the need for tightly focused incident light which would degrade the measurement angular resolution), relatively large devices were used, with a lateral dimension w of about 24 µm. In general, the value of this parameter controls the tradeoff between spatial and angular resolution of the filtered images described below.

The experimental results [Figs. 1(c) and 1(d)] show highly directional response in good agreement with theoretical expectations. In particular, the incident directions of high responsivity form a rather narrow distribution within the full hemisphere, consisting of two C-shaped regions of opposite curvature. The shape of this distribution is determined by the diffractive coupling of the incident light into different SPP modes, and the two C-shaped regions correspond to SPPs collected by the two slit sections surrounding the grating (see Supplement 1, Section S1). In sample A [Fig. 1(c)], these two regions overlap at θ = 0° so that a single peak is produced in the horizontal line cut of the angular response. This device can therefore provide low-pass spatial filtering along the x direction. In contrast, in sample B the two C-shaped regions are slightly offset from one another around θ = 0° [Fig. 1(d)], leading to two symmetrically located response peaks at θ = ±3.8°. In conjunction with an imaging lens of suitably small numerical aperture, the resulting transfer function corresponds to a high-pass filter. The experimental responsivities of these and similar devices were also compared to reference samples without any metasurface [22]. The results are generally consistent with the calculated metasurface transmission penalty (about 45% and 23% for p-polarized and unpolarized light, respectively, as discussed above), although large sample-to-sample variations were observed (even in the reference samples) due to fabrication imperfections.

3. Coherent transfer function and phase contrast imaging

In order to establish a connection between the angular response maps of Figs. 1(c) and 1(d) and the spatial filtering capabilities of the same devices, here we introduce and evaluate the corresponding filter transfer function. In a standard optical spatial filter, such as a 4f system or nanophotonic equivalent, the input and output signals are the optical field distributions E_in(r) and E_out(r) on the input and output planes, respectively, and the coherent transfer function (CTF) is defined as the ratio of their Fourier transforms $\textrm{t}({\mathbf k} )= {\textrm{E}_{\textrm{out}}}({\mathbf k} )/{\textrm{E}_{\textrm{in}}}({\mathbf k} )$. In contrast, an array of directional plasmonic image sensors converts its incident optical field distribution E_in(r) into a plasmonic field distribution E_SPP(r), which is then sampled at the slit locations through the slit-scattering/photodetection process illustrated in Fig. 1(a). With this in mind, we can take E_SPP(r) as the output signal of interest E_out(r), with the understanding that such signal is only meaningfully defined at the slit locations ${\mathbf r} = {\mathbf r}_{\textrm{sl}}^{\boldsymbol n}$, because it cannot be measured by the sensor array anywhere else (here ${\boldsymbol n} = \{{{n_\textrm{x}},{n_\textrm{y}}} \}$ denotes a pair of integers ${n_\textrm{x}}$ and ${n_\textrm{y}}$ that label the different pixels in the array, and the spatial variable ${\mathbf r}_{\textrm{sl}}^{\boldsymbol n}$ indicates position along the slits in the ${{\boldsymbol n}^{\textrm{th}}}$ pixel). A CTF can then again be defined as the ratio of the Fourier transforms of the (discrete-space) output and input signals ${\textrm{E}_{\textrm{out}}}({{\mathbf r}_{\textrm{sl}}^{\boldsymbol n}} )= {\textrm{E}_{\textrm{SPP}}}({{\mathbf r}_{\textrm{sl}}^{\boldsymbol n}} )$ and ${\textrm{E}_{\textrm{in}}}({{\mathbf r}_{\textrm{sl}}^{\boldsymbol n}} )$, and finally used to compute the image recorded by the sensor array.

To evaluate this CTF, we begin by considering the model structure shown in Fig. 2(a), which contains N nanostripes arranged periodically at positions x_l (l = 1, 2, 3, …, N) with period Λ = x_l – x_l-1, and one slit located to the right of the grating at x_sl+ (for simplicity in this discussion we omit the pixel-label superscript ${\boldsymbol n}$). The device is illuminated with a harmonic plane wave of in-plane wavevector component k = (2π/λ₀)sinθ along the x direction, so that the incident field distribution on the grating is ${\textrm{E}_{\textrm{in},\textrm{k}}}(\textrm{x} )= {\textrm{E}_{\textrm{in}}}(\textrm{k} ){\textrm{e}^{\textrm{ikx}}}$. The incident light is scattered by all nanostripes, and the scattered waves can excite SPPs on the underlying metal film if the requirements of energy and momentum conservation are satisfied. The resulting SPP field at the slit position can then be expressed as

 

Fig. 2. Coherent transfer function of the plasmonic directional image sensors of Fig. 1. (a) Schematic illustration of the physical model used to evaluate the CTF contribution t₊(k) from the slits on the right of the grating. The circles illustrate the phase relationship among the light waves scattered by different nanostripes into SPPs. (b), (c) Magnitude (normalized to unit peak value) (b) and phase (c) of t₊(k_x,k_y= 0) versus k_x, computed using Eqs. (3) and (4) with the parameter values of sample A. The dashed vertical lines indicate the range of k values (from –2π/λ₀ to +2π/λ₀) accessible with external illumination from air.

Download Full Size | PDF

Equation (1) can be simplified by using ${\textrm{E}_{\textrm{in},\textrm{k}}}({{\textrm{x}_l}} )= {\textrm{E}_{\textrm{in}}}(\textrm{k} ){\textrm{e}^{\textrm{ik}{\textrm{x}_l}}}$ and x_sl+ – x_l = (N – l)Λ + δ, where δ = x_sl+ – x_N is the distance between the slit and its nearest nanostripe [see Fig. 2(a)]. With these substitutions, we find that the SPP field at the slit can be expressed as ${\textrm{E}_{\textrm{SPP},\textrm{k}}}({{\textrm{x}_{\textrm{sl} + }}} )= {\textrm{t}_ + }(\textrm{k} ){\textrm{E}_{\textrm{in},\textrm{k}}}({{\textrm{x}_{\textrm{sl} + }}} )$, where

The CTF t_±(k_x,k_y= 0) of Eqs. (3) and (4) consists of a series of identical peaks centered at k_x = k_SPP – mg, where g = 2π/Λ, m is an integer, and each peak corresponds to a different order of diffraction. Figures 2(b) and 2(c) show, respectively, the magnitude (normalized to unit peak value) and phase of t₊(k_x,k_y= 0) computed with these equations. The corresponding plots for t₋(k) can be inferred directly from these traces using the relation t₋(k) = t₊(–k), which follows from Eq. (4) (see Supplement 1, Section S2) and is consistent with the symmetric device geometry under study. These calculations are based on the parameter values of sample A, including Λ = δ = 1485 nm, N = 15, λ₀ = 1550 nm, and k_SPP = 2π/Λ so that the m = 1 peak is centered at k_x = 0 as in Fig. 1(c). For the SPP propagation length we use L_SPP = 80 µm, selected with a numerical fit so that the peaks of |t_±(k_x,k_y= 0)|² have the same linewidth as in our measured responsivity data [the horizontal line cut of the color map in Fig. 1(c)]. The dashed vertical lines in Figs. 2(b) and 2(c) indicate the range of k values (from –2π/λ₀ to +2π/λ₀) accessible with external illumination from the air above the grating. The experimental data of Fig. 1(c) are well reproduced by the calculation results plotted in Fig. 2(b), including the fringes around the main peak which originate from incomplete cancellation of the scattered waves away from the Bragg condition in the presence of a finite number of grating lines.

As shown in Fig. 2(c), the phase response ϕ₊(k) = arg{t₊(k)} exhibits a linear dependence on k_x with negative slope dϕ₊(k)/dk_x = –α across the entire linewidth of the peak at k = 0, i.e., for all accessible values of k for which |t₊(k)| is non-negligible. The value of the slope parameter α inferred from this plot is 9.8 µm, which is relatively close to the distance between the slit and the center of the grating x_sl+ – x_c = 11.9 µm in device A. In fact, as shown in Supplement 1, Section S3, α becomes exactly equal to x_sl+ – x_c in the limit of large L_SPP. Detailed FDTD simulations (also presented in Supplement 1, Section S3) similarly indicate a linear phase profile for t₊(k_x,k_y= 0) near k_x = 0 with comparable slope parameter α = 12.2 µm. By the shifting property of Fourier transforms, this linear phase profile corresponds to a displacement in real space by the amount α in the negative x direction. We can therefore conclude that the SPP signal sampled by the slit at x_sl+ is an amplitude-filtered version of the light incident on the device at position x_sl+ – α, close to the center of the grating. Similar considerations apply to the phase of t₋(k) with positive slope α, so that the slit at x_sl− filters the input signal at x_sl− + α, also close to the center of the device. It should be noted that this sampling behavior is fundamentally different from the operation of standard photodetectors, which instead average the incident light across their entire illumination window.

When the same plasmonic devices are illuminated with an arbitrary incident field ${\textrm{E}_{\textrm{in}}}({\mathbf r} )= \smallint \textrm{d}{\mathbf k}\,{\textrm{E}_{\textrm{in}}}({\mathbf k} ){\textrm{e}^{\textrm{i}{\mathbf k}\cdot {\mathbf r}}}$, the measured photocurrent is proportional to the sum of the intensities of the corresponding SPP fields detected by the two slits ${\textrm{E}_{\textrm{SPP}}}({{{\mathbf r}_{\textrm{sl} \pm }}} )= \smallint \textrm{d}{\mathbf k}\,{\textrm{t}_ \pm }({\mathbf k} ){\textrm{E}_{\textrm{in}}}({\mathbf k} ){\textrm{e}^{\textrm{i}{\mathbf k}\cdot {{\mathbf r}_{\textrm{sl} \pm }}}}$, each averaged over the slit length along the y direction. This photocurrent signal can therefore be computed from the CTF t_±(k). To evaluate the filtering capabilities of our experimental samples, in the calculations presented below this CTF is expressed as

As an example, we simulate the “phase” object shown in Fig. 3(a) (e.g., a phase-grating set in a transparent glass plate of refractive index n_glass and variable thickness h, where the phase value ϕ displayed in the figure varies with position according to ϕ = 2π(n_glass – n_air)h/λ₀. In this case, the object field E_obj(r) has uniform amplitude across the entire field of view, and therefore the object could not be visualized using a standard imaging system with k-independent response, except for negligibly small diffraction fringes [Fig. 3(b)]. At the same time, as light is transmitted through the object, its local direction of propagation is deflected by an angle proportional to the local phase gradient. As a result, if the response of each pixel varies with angle of incidence, the recorded photocurrent signals acquire a dependence on the object phase gradient. Spatially varying features of the phase object (in this case, the edges of the grating lines) can therefore be resolved. The same behavior can also be described in the spatial frequency domain as edge enhancement caused by the CTF k dependence. These ideas have been explored extensively with different types of optical spatial filters [35–38], for applications ranging from label-free imaging of biological samples [39] to semiconductor wafer inspection [40]. By virtue of their intrinsic angular sensitivity, the directional image sensors under study can provide the same functionality without any external spatial filtering elements.

 

Fig. 3. Phase imaging simulation results. (a) Phase distribution of the object. (b) Image of the object of (a) computed for an array of 392×392 uncoated pixels [i.e., with k-independent CTF] combined with an NA = 0.13 imaging lens. (c) Image of the same object computed for an otherwise identical camera where every pixel is coated with the metasurface of device B [modeled using the experimental data of Fig. 1(d)]. The signal intensity in (b) and (c) is normalized to that of the uncoated devices when illuminated with the same plane wave incident on the object.

Download Full Size | PDF

To illustrate, we have computed the image of the phase gratings of Fig. 3(a) for an array of 392×392 pixels described by the experimental angular response map of device B [Fig. 1(d)], combined with an F/3.8 imaging lens (corresponding to a numerical aperture NA = 0.13 and a field of view of 15°). Following the prescriptions above, this image was obtained by summing the contributions from the two slit sections governed by the CTFs t_±(k) (see Supplement 1, Sections S5 and S6 for more details). The resulting plot is shown in Fig. 3(c), where the signal measured by each plasmonic pixel is normalized to that of an identical uncoated device under the same illumination conditions. The y-oriented edges of the grating lines are clearly visualized in this image, consistent with the strong k_x-dependence of the responsivity R(k) of device B at small angles of incidence. The x-oriented edges can also be discerned, but with significantly lower contrast, due to the weaker variations of R(k) with k_y (mostly related to the C shapes of the responsivity peaks). For comparison, the grating lines are essentially invisible in the image computed for an otherwise identical camera of standard pixels with k-independent CTF [Fig. 3(b)]. It should also be noted that isotropic phase imaging could similarly be achieved with this approach, using alternative metasurface designs featuring rotationally-invariant angular response, e.g., based on circular grating lines and slits.

4. Optical transfer function and incoherent edge enhancement

Next, we consider the frequency response of the same devices under natural, and therefore spatially incoherent, illumination. In this case, the incident light features a highly localized correlation function, which can be modeled as $\left\langle\mathrm{E}_{\operatorname{in}}^{*}\left(\mathbf{r}-\frac{\delta \mathbf{r}}{2}\right) \mathrm{E}_{\operatorname{in}}\left(\mathbf{r}+\frac{\delta \mathbf{r}}{2}\right)\right\rangle \propto {\textrm{I}_{\textrm{in}}}({\mathbf r} )\textrm{exp}\left( { - \frac{{\mathrm{\delta }{\textrm{r}^2}}}{{4{\mathrm{\Delta }^2}}}} \right)/\mathrm{\Delta }$, where the brackets 〈…〉 indicate an ensemble average and the transverse coherence length Δ is small compared to the size of the image. Under these conditions, the operation of any optical spatial filter is governed by its optical transfer function (OTF) $\textrm{T}({\mathbf q} )= {\textrm{I}_{\textrm{out}}}({\mathbf q} )/{\textrm{I}_{\textrm{in}}}({\mathbf q} )$, where I_in(q) and I_out(q) are the Fourier transforms of the input and output field-intensity distributions, respectively [4]. This function can be computed by expressing the output intensity $\mathrm{I}_{\text {out }}(\mathbf{r}) \propto\left\langle\mathrm{E}_{\text {out }}^{*}(\mathbf{r}) \mathrm{E}_{\text {out }}(\mathbf{r})\right\rangle$ in terms of the Fourier transform of the output field ${\textrm{E}_{\textrm{out}}}({\mathbf k} )= \textrm{t}({\mathbf k} ){\textrm{E}_{\textrm{in}}}({\mathbf k} )$, and then evaluating the ensemble average in the spatial-frequency domain. In the directional image-sensor arrays of interest in this work, I_out(r) is again only accessible at the slit locations ${\mathbf r}_{\textrm{sl} \pm }^{\boldsymbol n}$, but otherwise the OTF can be defined and computed in the same fashion. As detailed in Supplement 1, Section S7, the resulting expression is

Figures 4(a) and 4(b) show the calculated magnitude of the OTF T₊(q) of devices A and B, respectively. In these calculations, the parameter Δ is evaluated based on the Van Cittert – Zernike theorem [4]. Specifically, we use Δ = 6.7 µm, which corresponds to a fully incoherent object at a representative distance of 10 times its lateral size. The corresponding radius of coherence r_c on the sensor array is 18.9 µm (see Supplement 1, Section S8), which is sufficiently large (i.e., larger than the half-size of the grating) to ensure that the SPPs scattered by all the grating lines can properly interfere with one another. This value of 18.9 µm should also be regarded as a lower bound for r_c, because it neglects the finite coherence of the light at the object. The CTF t_±(k) in Eq. (6) is computed using Eq. (5), with the responsivity contributions from the two slits ${\textrm{R}_ \pm }({\mathbf k} )$ obtained from the experimental data of Figs. 1(c) and 1(d) as described in Supplement 1, Section S5. As expected, both transfer functions plotted in Fig. 4 are nonzero and near-maximum at q = 0, which again originates from the autocorrelation nature of Eq. (6). At the same time, the detailed shape of these OTFs is determined by the k-dependence of the corresponding CTFs (which in turn can be tailored through the metasurface design), combined with the windowing action of the exponential term in Eq. (6). More complex OTFs can therefore be envisioned, for example involving additional peaks at finite q values, with metasurfaces designed to produce multiple peaks in t₊(k) and t₋(k) individually.

 

Fig. 4. Optical transfer function of the plasmonic directional image sensors of Fig. 1. Panels (a) and (b) show the magnitude of T₊(q) for devices A and B, respectively, computed using their measured responsivity maps and normalized to unit peak value.

Download Full Size | PDF

The transfer function of Eq. (6) can be used to model the incoherent imaging capabilities of arrays of these plasmonic devices. To that purpose, we begin by noting that the photocurrent measured by the ${{\boldsymbol n}^{\textrm{th}}}$ pixel is proportional to ${\textrm{I}_{\textrm{meas}}}({\boldsymbol n} )= {\textrm{I}_{\textrm{meas} + }}({\boldsymbol n} )+ {\textrm{I}_{\textrm{meas} - }}({\boldsymbol n} )$, where

The geometrical tunability of the transfer function of these devices is particularly significant for use in sensor arrays partitioned into identical blocks of multiple adjacent pixels, each coated with a different metasurface. With this arrangement, a single camera could produce multiple filtered images of a same object simultaneously, which could then be exploited to perform specific visual processing tasks (e.g., object recognition) with reduced electronic computational cost. Within this framework, even the constraint that T_meas(q = 0) ≠ 0 under incoherent illumination can be effectively circumvented by subtracting the signals of different adjacent pixels within each block, to produce an overall response equal to the difference of their respective OTFs. In fact, a similar idea has already been explored to enable incoherent edge detection with a 4f system, where two masks of different cutoff frequencies are inserted sequentially at the Fourier plane and the resulting images are subtracted from one another [26]. Such setup, however, is particularly bulky and rather impractical. More recently, nanophotonic implementations have also been proposed based on wavelength or polarization multiplexing [27,28]. By virtue of their ability to enable multiple filtering operations simultaneously on a pixel-by-pixel basis, the directional image sensors under study are ideally well suited to implement this general approach for incoherent image processing.

As a particularly simple illustration, we consider the configuration shown schematically in Fig. 5(a). Here, the low-pass-filter metasurface of device A is fabricated on every other pixel of an image sensor array in a checkerboard pattern, while all the other devices are left uncoated. The photocurrent measured by each plasmonic sensor is then subtracted from that of its adjacent uncoated pixel to produce a null at q = 0, and therefore high-pass filtering. The resulting image is related to the object according to ${\textrm{I}_{\textrm{meas}}}({\mathbf q} )= \mathrm{\Delta }{\textrm{T}_{\textrm{tot}}}({\mathbf q} ){\textrm{I}_{\textrm{obj}}}({\mathbf q} )= \mathrm{\Delta }{\textrm{T}_{\textrm{meas}}}({\mathbf q} ){\textrm{T}_{\textrm{lens}}}({\mathbf q} ){\textrm{I}_{\textrm{obj}}}({\mathbf q} )$, where ΔT_meas(q) is the difference between the measurement OTFs of the two neighboring pixels. In the uncoated reference pixels, the responsivity R(k) is essentially constant with k, and the frequency dependence of the measurement OTF is mostly determined by pixelation effects. At the same time, their photocurrent signal at normal incidence (q = 0) is larger than in the plasmonic devices due to the aforementioned transmission penalty of the metasurfaces (about 23% for unpolarized light). This difference can be normalized out in the digital data processing before the subtraction step. Alternatively, it could be handled in the optical domain by reducing the size of the reference pixels along the x direction by the same factor of 23%, e.g., using the checkerboard pattern with alternating square and rectangular pixels shown in Fig. 5(a). In fact, this approach provides several other important advantages. First, it reduces the size of each super-pixel (i.e., each block of adjacent coated and uncoated devices), which is favorable to increase the spatial resolution of the high-pass-filtered images. Second, it produces a flatter frequency response for the reference pixels across the full bandwidth of T_lens(q), and therefore increases the frequency range over which ΔT_tot(q) can be tailored through the plasmonic pixel design. Finally, it can also result in improved noise cancellation upon signal subtraction.

 

Fig. 5. Incoherent edge detection protocol. (a) Schematic illustration of the envisioned sensor array, which consists of pixels based on the plasmonic directional photodetectors of Fig. 1(a) (yellow squares) combined with bare pixels (blue rectangles) in a checkerboard pattern. The edge-enhanced image is obtained by subtracting the signals of neighboring pixels. (b) Differential OTF ΔT_tot(q) = ΔT_meas(q)T_lens(q) of the resulting imaging system with an NA = 0.13 lens, computed by subtracting the cumulative OTFs of the two types of pixels. Here ΔT_tot(q) is normalized to the cumulative OTF of the bare pixels at q = 0. The inset shows the q_y= 0 line cut of ΔT_tot(q) (solid line), together with a numerical fit of the low-q_x portion of this trace to a quadratic function of q_x (dashed line).

Download Full Size | PDF

For the pixel configuration of Fig. 5(a), this procedure leads to the differential OTF ΔT_tot(q) plotted in Fig. 5(b) (see Supplement 1, Section S9 for more details). As expected, this transfer function is zero at q = 0, and features two pronounced peaks at symmetric locations around the origin along the q_x direction. The imaging system of Fig. 5(a) can therefore be used to enhance rapidly varying features of the object (i.e., edges) along the x direction. In particular, the q_y = 0 line cut of ΔT_tot(q) [shown by the solid line in the inset of Fig. 5(b)] is well approximated by a quadratic function of q_x (dashed line) over a broad portion of the accessible spatial-frequency range. Since multiplication by q_x² in the frequency domain is equivalent to taking the second-order derivative with respect to x in the space domain, the pixel array of Fig. 5 can provide (directional) spatial differentiation, leading to edge enhancement. Importantly, the computational cost of this protocol (associated with the pixel subtraction steps) is significantly smaller than that of the standard digital-electronics approach for second-order differentiation (based on a Laplacian of Gaussian filter), by an estimated factor of about 5 or larger [27].

The expected filtering behavior is illustrated in Figs. 6(a) and 6(b), where we plot a simple amplitude object and the absolute value of its detected image, for an array of 171×210 super-pixels again combined with a lens of NA = 0.13. Clear edge enhancement is observed with maximum contrast for the edges oriented along the y direction, whereas x-oriented edges are not resolved. In Fig. 6(c), the red and blue traces show, respectively, the line cut of the filtered image along the dashed line of Fig. 6(b) and the corresponding object. The comparison between the two traces clearly demonstrates the second-order derivative nature of this optical spatial filter along the x direction, which produces the two peaks per edge observed in the image. The magnitude of the detected signals in Fig. 6(b) is limited by the differential nature of the underlying data acquisition, combined with pixelation effects. In any case, the resulting image can be fully resolved with the signal-to-noise ratio (SNR) levels accessible with near-infrared photodetectors of similar dimensions [41] (see Supplement 1, Section S10). These results therefore demonstrate the feasibility of incoherent high-pass filtering with the plasmonic image sensors under study. Additional filtering operations (e.g., edge enhancement along different orientations and/or with different transfer functions) could similarly be produced at the same time with the same sensor array, using different metasurface designs and pixel arrangements.

 

Fig. 6. Incoherent edge detection simulation results. (a), (b) Object (a) and absolute value of its edge-enhanced image (b) computed for an array of 171×210 super-pixels based on the configuration of Fig. 5 [with the plasmonic pixels modeled using their experimental data of Fig. 1(c)] combined with an NA = 0.13 lens. (c) Red trace: line cut of the image along the dashed line of (b). Blue trace: line cut of the object along the same line. In (b) and (c), the differential signal produced by each super-pixel is normalized to the signal of the uncoated device under maximum illumination from the object.

Download Full Size | PDF

5. Conclusion

We have introduced a new approach for optical spatial filtering based on pixel arrays of plasmonic directional image sensors with tailored angular response. To establish the image processing capabilities of these devices, we have developed a rigorous theoretical model of their filter transfer function under both coherent and incoherent illumination. The effectiveness of this approach for phase imaging and incoherent edge detection has also been demonstrated through imaging simulations based on the measured angle-resolved responsivity of prototype samples. High-quality filtered images were correspondingly obtained, showing that these experimental samples can provide the required angular selectivity and contrast for the envisioned image processing functionalities.

These results are promising for a wide range of application areas including microscopy (e.g., for the visualization of transparent biological cells) and computer vision (e.g., for object recognition). Compared to more traditional optical spatial filters based on Fourier optics, our approach does not require any external spatial filtering elements, and therefore can provide extreme size miniaturization and improved ease of alignment, which are beneficial for embedded and mobile applications. Furthermore, this approach allows controlling the filter transfer function on a pixel-by-pixel basis, so that multiple filtered images of a same object can be produced simultaneously, similar to the output of the first layer of a CNN. These images could then be fed into the subsequent CNN layers to perform various visual recognition tasks. Compared to fully electronic solutions, this ability to synthesize multiple filtered images in the optical domain can provide significant savings in power consumption, estimated at about tenfold in prior studies of hybrid optoelectronic CNN configurations [19,42].

The devices developed in the present work rely on a diffractive plasmonic metasurface to produce the required angular sensitivity, and as a result are limited to narrow-band operation at infrared wavelengths. Specific applications of near-infrared light that would directly benefit from their unique capabilities include, for example, imaging through biological tissue, navigation and surveillance under low-visibility conditions, and facial/gesture recognition. Furthermore, more advanced metasurfaces, including dielectric systems and angle-sensitive meta-units [43], can be envisioned to extend the bandwidth and accessible wavelength range, as well as enable more complex filter transfer functions. We believe that the results presented in this work will provide a strong motivation, as well as theoretical guidance, for the further development of this new family of flat-optics devices.