1. Introduction

3D printing via femtosecond direct laser writing (fsDLW) has been well established since its first demonstration in the 1990s [1–6]. While the technology has advanced the development of micro-optics rapidly [7–10], freeform optical design has gained in significance [11–16]. The possibility to create real three dimensional structures with highly tilted or undercut features opens an entirely new field in optical design freedom. Therefore, conflicting specifications such as ultra-large fields of view combined with large lateral image sizes can be rethought. Conventional wide-angle cameras realized by fsDLW have been demonstrated [17]. Splitting the aperture in a biomimetic approach is one way of increasing the conveyable information [18–22]. Current limits contain the number of optical channels as well as the maximum FOV that can be imaged with an optical system of limited height. While a large number of optical channels can either raise the FOV by stitching or enable high resolution imaging by combination of the channels [23,24], the lateral extension of flat cameras generally rises with the number of channels. For instance, Gassner et al. [23] report a compact wide-angle camera with 110 $^{\circ }$ diagonal FOV with a height just below 1 mm with an array of 15 $\times$ 11 lenses. Another state-of-the-art example is Omnivision’s 120$^{\circ }$ diagonal FOV camera cube OVM6948-RALA [25] with a lateral extension below 1 mm and a height of 1.2 mm. None of these approaches reach a height well below 1 mm, neither was a 180$^{\circ }$ full FOV mapped with a limited number of lenses. We believe that freeform optical design of stacked lens systems in combination with fDLW as manufacturing technology can be the key to such compact wide-angle camera systems. Optical freeform designs utilizing strictly refractive surfaces to image FOVs up to 85$^{\circ }$ have been previously suggested for fDLW [26]. Here, we extend the concept following the Scheimpflug condition and add catadioptric freeform designs to reach extreme FOVs: We show that a set of four lenses can be sufficent for 180$^{\circ }$ panoramic imaging.

Apart from freeform optically active surfaces, also the lens fixtures can be equipped with functional elements due to fsDLW high resolution. It has been demonstrated that microfluidic channels can transport and contain absorbing fluids to shield stray light or create apertures [27], and that mechanical movement can be realized [28–30]. Here, we suggest to use the lens fixture for quality inspection, namely, by integrating reference structures to facilitate topology measurements. Taken together, arbitrarily oriented freeform surfaces can not only be fabricated but also measured and corrected. While often neglected, only the latter two can guarantee high-quality imaging.

In this work, we present freeform design concepts, correction methods and realizations of multi-aperture wide-angle cameras tailored for fabrication via fsDLW. Our findings can guide freeform optical design and shape correction not only for wide-angle cameras but for any off-axis imaging, illumination or sensor application, for example in endoscopy or consumer electronics.

2. Optical design

In the optical design, we examine three different concepts to realize the ultra-large FOVs ($180^{\circ }{\times }360^{\circ }$) with multiple apertures using Zemax OpticStudio (Fig. 1(a), left and Table 1). We utilize the Scheimpflug condition as the basic design principle, where a tilted lens with its principle plane intersects the crossing point between object and image plane (Fig. 1(a), middle). For all designs, the photoresist IP-S (Nanoscribe, refractive index Sellmeier coefficients from [31]) is used as a single material system. For simplicity, the lenses are optimized monochromatically at 550 nm (camera A) and 525 nm (cameras B and C). In principle, chromatic aberrations can be corrected by multi-material or single-material hybrid refractive-diffractive approaches [32]. The object is assumed to be at infinity for all cameras since the hyperfocal distance is relatively short due to the short effective focal lengths of our microlenses.

 

Fig. 1. Concepts for ultra-compact multi-aperture wide-angle cameras based on freeform optical design utilizing the Scheimpflug condition. a, Visualization of the imaging dome with an azimuthal angle of 360$^{\circ }$ and a polar angle of 180$^{\circ }$ (left), Scheimpflug condition with intersecting object plane, image plane and tiltet lens principal plane (middle, tilted lenses highlighted in A1 and B2), and camera concept A with a split central field of view (FOV, A1) and one additional side-viewing lens (A2) based on total internal reflection (TIR, right). b, Camera concept B with one center lens (B1) to cover the central FOV, one freeform refractive side-viewing lens (B2) and one TIR catadioptric side-viewing lens (B3). c, Camera concept C comparable to B with additional structures for the creation of defined apertures and stray light shielding. All concepts are vizualized as an overlay of optical ray tracing and 3D-printable mechanical design. The hovering B1 mechanical design indicates a first refocussing approach.

Download Full Size | PDF

Table 1. Designed wide-angle cameras overview. Rotationally symmetric: rot. symm., total internal reflection: TIR, field of view: FOV. Mechanical bounding box: BB.

View Table

First, we concentrate on an overall small form factor of the camera with design A. A wide-angle camera with a lateral extension of 1 mm and a vertical extension below ${300}\;\mathrm{\mu} \textrm {m}$ was realized (Fig. 1(a), right). It is also characterized by a split central field of view and thus consists of only two lens types to cover $0^{\circ }−40^{\circ }$ (A1) and $40^{\circ }−90^{\circ }$ (A2) FOVs. Surface types comprise Extended Polynomials (A1) and a mix of Extended Polynomials and Even Aspheres (A2). The full FOV is covered by reproducing these two lens types in different azimuthal rotation states as visualized for the $180^{\circ }$ rotation (mirrored setup).

Our camera design B (Fig. 1(b)) is characterized by an additional lens to cover the central field of view (B1). In general, imaging at the central FOV is still of highest relevance, just like our eyes have best resolution at the center. Furthermore, rotationally symmetric lenses are well explored in design and fabrication [9,22]. For ease of fabrication, these lenses avoid sags in negative $z$-direction as an additional design constraint. Layer-by-layer fabrication without support structures profits from such design as it can be operated without reversing printing direction and printing through already polymerized material. The surface types of the designs comprise Odd Aspheres in the center (B1: $−15^{\circ }$ to 15$^{\circ }$ FOV) and Extended Polynomials for the side-viewing lenses (B2: 15$^{\circ }$ to 45$^{\circ }$ and B3: $45^{\circ }−75^{\circ }$ FOV).

The camera design C (Fig. 1(c) utilizes apertures as additional design parameters. While the first lens surface defined the aperture in the previous designs without particularly differentiating between lens and mount, our last design makes use of an additional fabrication step for realization of defined apertures [27]. Ray aberrations are thus minimized by limiting the aperture diameter, and the lenses are equipped with some stray light shielding structures. In this concept, the central lens is further extended to a rotationally symmetric FOV of −45$^{\circ }$ to 45$^{\circ }$ (C1), C2 covers the range of $40^{\circ }−70^{\circ }$, and C3 has a FOV of $65^{\circ }−90^{\circ }$. To facilitate computational image stitching in subsequent steps, partly overlapping FOVs were specifically designed. C1 surface types are all Even Aspheres, while for the non-rotational C2 the surface type Polynomial and for C3 Biconic proved to be suitable. As visualized, C2 optimization was derived from a wide-angle design with the axis of symmetry at the lateral position of the aperture. In lens C2 the right half was not printed (indicated by black lines only).

The theoretical modulation transfer functions (MTF) of all designed objectives are displayed in Fig. 2. Both designs A and B suffer from aberrations that can be reduced by utilization of apertures as shown in design C. This way, a contrast above 0.2 at 300 cycles per mm can be reached with C1 and C3 and a contrast above 0.1 at 300 cycles per mm with C2.

 

Fig. 2. Theoretical modulation transfer function (MTF) of the designed objectives. The MTF was plotted for all design fields of the respective objectives. 300 cycles per mm and a contrast of 0.2 are highlighted in all graphs with vertical and horizontal dashed lines, respectively. Design concept C shows the best performance, generally a contrast better than 0.2 above 300 cycles per mm, due to utilization of defined apertures that minimize aberrations.

Download Full Size | PDF

All three camera designs are displayed as an overlay with the corresponding printable computer aided design (CAD) models in Fig. 1 (CAD: gray filling, Zemax: black outlines). Camera A has the smallest form factor and an optimum fill factor of the image surface, yet no optimum center imaging. Furthermore, to cover the full azimuthal 360$^{\circ }$, the lenses would need to be displaced somewhat to make room for rotated lenses. However, the current design is a promising layout for panoramic imaging. Camera B is optimized for ease of fabrication, has first stray light shielding structures (B3), yet suffers from ray aberrations. The floating B1 CAD design indicates a simple measure for refocussing to account for shrinkage in the fabrication process (see section 4.). Design C aims at optimizing imaging quality by introducing apertures and stray light shielding structures with a large central FOV. Furthermore, the vertical extension is restricted to ${300}\;\mathrm{\mu} \textrm {m}$ here, which is the high precision range of the 3D printer (piezo axis). In exchange, the lateral fill factor is reduced. All three cameras utilize a TIR catadioptric design for the extreme side-viewing lenses (A2, B3, and C3). By keeping all rays above the critical TIR angle of 41$^{\circ }$ at the refractive index of the photoresist $n_{IP-S, 525 nm}=1.515$, the reflective surface can be realized simply by the polymer-air interface. As a result, stray light shielding structures must account for a small gap between ink and TIR surface. While the use of a TIR-surface is the key for a high-quality optical design, it puts more focus on fabrication modalities. In comparison to a refractive surface, deviations in a reflective surface deform the wavefront roughly six times as much at the given difference of refractive indices. Therefore, a highly accurate shape fidelity is required after fabrication. A measurement process and correction methods for freeform and highly tilted surfaces are presented in the following sections.

3. Fabrication

The CAD-models are sliced and hatched with distances of $0.1-{0.2}\;\mathrm{\mu} \textrm {m}$ and $0.25-{0.5}\;\mathrm{\mu} \textrm {m}$, respectively. Structures that contain negative surface sags are divided into several parts with alternating printing $z$-direction to avoid free floating parts during fabrication. The lenses are fabricated by fsDLW from the photopolymer IP-S using Nanoscribe PPGT and PPGT2 3D printers in dip-in mode. While the designs aim at fabrication on image sensors or imaging fiber bundles, glass substrates are used for characterization and improvement studies. Polymerized structures are developed in propylene glycol methyl ether acetate, rinsed with isopropanol, and dried with a nitrogen blower. Prints on image sensors are plasma activated before production. Apertures and straylight-shielding structures are added after development by applying the superfine inkjet process described in [27].

4. Freeform measurement and iterative improvement

In general, using fsDLW for the fabrication of lenses provides almost unlimited design freedom. However, polymerization comes with the caveat of shrinking or expanding-processes in the material. These occur due to changes in the mechanical properties between polymerized and unpolymerized material. These processes reproduce the same deviations from print to print in the range of about 30 nm to 70 nm [14] when using the same target structures and printing parameters. Therefore, we used an iterative method to measure and mitigate deviations from the designed shape which has been employed in 2D for rotationally symmetric designs in previous works [33]. This method is divided into three steps: measurement, parametrization, and compensation, over which one or two iterations lead to overall better shape fidelity.

4.1 Measurement

3D-printed optics include optical surfaces as well as mechanical support structures which are fabricated in the same step of production. This integral character can complicate measurement and characterization of printed surfaces due to limited access. For this reason, special samples particularly for the measurement are prepared (Figs. 3(a) and (b)). These samples follow the regular fabrication process but are designed in such way that supportive structures can be detached at specific contact points. Consequently, they can be oriented such that the surface to be measured is easily accessible with a confocal microscope (NanoFocus $\mathrm{\mu}$-surf). This requires the introduction of reference geometries within the measurement sample (Fig. 3(a)). Errors due to misalignment, tilt or orientation can thus be reduced. Any structures which would largely obstruct the optical path of the high $NA = 0.95$ confocal microscope objective must be removed as well. For surfaces which are printed at a high angle to the substrate, as expected in side-looking objectives or large FOVs, an additional structure has to be provided. This structure can be used to detach parts of the lenses with a piece of adhesive tape and rotate them by a known angle, so the measurement can be performed in a flipped orientation (see Figs. 3(b) and (c)).

 

Fig. 3. Freeform measurement reference structures. a, The integration of planes (green) and markers (red) into the lens mounts facilitates detilt and rotation correction after the measurement. Predetermined breaking points (blue) enable measurement of inaccessible surfaces by removal with tape. b, Large surface tilts can be reduced by printing of plane assemblies that serve as contact surfaces for flipped measurement on tape. c, Tilt and rotation correction of an exemplary confocal measurement utilizing reference structures from a and b.

Download Full Size | PDF

4.2 Parametrization and improvement

The obtained measurement data describes the actually manufactured surface to the accuracy which is permitted by the confocal microscope, as shown in Fig. 4. The goal of the parametrization step is to translate these measurements into a functional description. This in turn can be used to determine necessary changes to the CAD-model (target geometry). The new target matches the design after the process-related shrinking and distortion effects more closely.

 

Fig. 4. Description of the measurement and modification process. Manufactured surfaces are measured and fitted using the same base function as the surface description. a, Shrinking and warping lead to deviations between design and measurement fit. b, Change of target structure for the next iteration of printing using the difference in function coefficients $\Delta C_i$, or half of them. c, Line profile cuts at $x=0$: measured profile fit (green), deviation from design (amplified 10x, black), new printing target profile (red). Axes in $\mathrm{\mu}$m.

Download Full Size | PDF

For this purpose, an appropriate method of data fitting has to be selected, depending on the surface description of the printed surfaces. The original surface definitions from Zemax OpticStudio are represented by equations, that provide an appropriately complex model for the fitting process. As an example all surfaces in the C3 lens from Fig. 1(c) are defined as biconic surfaces , which are represented by Eq. (1):

The data presented in this publication is obtained using a nonlinear least square fitting algorithm with up to 1000 iterations and a confidence level of 95 % from the cftool, which is using a weighted matrix in order to reduce the impact of the outer regions of the measured surfaces. This is done because the influence on optical quality from these areas is low or negligible, and the measurement data for these areas often is not sufficiently accurate due to obstructions or too steep surface angles for accurate measurement using the confocal microscope. Fit results returned a root mean squared (RMS) error for the quality of the functional fit description in the range of ${0.07}\;\mathrm{\mu} \textrm {m}$ to around ${0.5}\;\mathrm{\mu} \textrm {m}$. The deviation between the resulting fitted measured surface and the design can be represented by a surface plot as depicted in Fig. 4(a), or as a 2D deviation plot $\Delta z(x,y)$ which can be further analyzed using statistical methods like the RMS value, as shown in Fig. 5. In order to compensate for the differences between manufactured surfaces and design, a comparison of the designed surface definition coefficients and the measured fit coefficients is performed. In one approach the $\Delta C_i$ of the coefficients for each parameter is added to the original design coefficients, resulting in a modified target surface which will be closer to the desired shape after the printing process related shape deviations. This, however, can lead to overshooting the target shape. Therefore tests have been performed using $\Delta C_i / 2$ as change in parameters, which proved beneficial in some, but not all cases.

 

Fig. 5. Deviations of the measured surfaces to the designed surfaces per iteration for all three surfaces of the C3 lens. Comparing a-c shows the improvement in image clarity and resolution for each iteration. Positions for profile extraction for experimental MTF evaluation are highlighted in red. For surface 2 no second iteration was necessary, since the resulting deviations after the first iteration showed to be $< \lambda /10$ for the visible range. d, CAD model of the C3 lens highlighting the surfaces per column . Deviation height values in nm, lateral scales in $\mathrm{\mu}$ m.

Download Full Size | PDF

This process of definition, printing, measuring, fitting, and comparing is performed multiple times, until the shape changes converge. In this work, we regard RMS surface deviations below approximately $\lambda /10$ for reflective surfaces and $\lambda /2$ for refractive surfaces to be within an acceptable level of tolerance close to the designed surface shape. This is especially remarkable for surfaces almost perpendicular to the printing substrate, like surface 1 on the C3 objective. An example of the resulting deviations between process and design is depicted in Fig. 5. The imaging examples of the C3 lens (Figs. 5(a)-(c)) for the original system and one or two times iterative changes, respectively, show a significant improvement in image quality. The initial surface shape provides lower contrast and resolution in the center mainly due to a defocus of about ${20}\;\mathrm{\mu} \textrm {m}$ resulting from the contraction of the printed surfaces. In this example the main source of error can be found in surface 3 (closest to the image plane) of the system, where an RMS deviation in the order of ${1}\;\mathrm{\mu} \textrm {m}$ can be observed. Due to the iterative process the RMS deviation for surface 3 is reduced to about 125 nm resulting in a significant improvement of imaging quality.

In order to evaluate imaging performance, the MTF is derived from images of a spoke target for three iterations of the C3 lens (Fig. 6). The edge profiles indicated in Fig. 5 are extracted and averaged over 30 lines each. The modulus of the Fourier transform of the profile gradient is interpreted as the MTF. Extracted profiles were interpolated along the edge and extrapolated with their constant start and end values in order to reach frequency resolutions appropriate for display, which results in slight exaggeration of the experimental MTFs as an artifact in low frequencies. The theoretical range was drawn from Fig. 2. A significant improvement between iterations for the horizontal MTF can be observed, while the already good starting performance of the vertical MTF could be conserved at the same time.

 

Fig. 6. Experimental characterization of the iterative MTF improvement of the C3 lens. a, The horizontal profile evaluation shows improving characteristics with each iteration and b, the vertical profile evaluations confirm good imaging quality from the first print, which is conserved over the iterations. The MTF values are derived from extra- and interpolated profiles of the spoke targets indicated in Fig. 5. Inserts depict the extracted intensity profiles which served as the base for the evaluation.

Download Full Size | PDF

While the presented method is able to produce surface deviations down to $\lambda /10$ for the visible range within just one or two iterations, it is basically limited by the quality of the measurement fit and its effect on the lens parameters of the subsequent iteration. A fit using more degrees of freedom might prove to be beneficial, but would require additional computing in order to translate determined factors into printable lens descriptions. One other option would be to change the lens types from their initial surface shapes into sometimes available extended forms, which allow for more parameters to fine-tune the surface sags. While it is possible to quickly achieve good results using the difference-of-coefficients approach, more detailed analysis taking the characteristic behavior of different surface description functions into consideration might prove beneficial in the future.

5. Digital image correction

While aberrations can be corrected to a certain extend by shape fidelity improvement, the distortion of an imaging system can only be as good as the optical design. In our challenging freeform optical designs, distortion was partially sacrificed since it can easily be digitally corrected. We display this correction in Fig. 7 for the B2 and B3 lenses. The distortion is characterized by imaging of a checkerboard pattern and quantifying the edge deviation from a perfect 1:1 grid. The measurement of lenses on a glass substrate yields a maximum distortion of 17 % for the B2 lens. For the B3 lens mainly a different magnification in $x$ and $y$-direction can be observed which has a relative difference of 35 %. For imaging on a camera sensor and subsequent stitching, a MatLab-code was developed that automatically corrects for a distortion that is calculated from manually detected edges in the image of the checkerboard pattern (Fig. 7(b)).

For this purpose, the lenses were fabricated on a RaspberryPi V2.1 camera module after the proprietary objective was removed. The camera is initialized via MatLab with its properties ’Sharpness’ and ’Contrast’ set to a maximum. In the images on this sensor, a lighter part in the center of the B3 image can be observed. This is presumably due to residual undeveloped photoresist between the last lens surface and the image sensor. This unfavorable short distance was resolved in the final evolution step of the optical designs (camera C).

 

Fig. 7. Distortion quantification and digital correction. A backside-illuminated checkerboard pattern is imaged and the distortion is quantified from the offset of the edges from a grid with a 1:1 aspect ratio, indicated with arrows. a, Evaluation of lenses printed onto a glass substrate with a microscope and b, Evaluation and distortion correction for the respective lenses from a, printed directly onto an image sensor chip.

Download Full Size | PDF

6. Wide-angle cameras

Finally, two different freeform multi-aperture wide-angle cameras are realized. The camera shown in Fig. 8 (type C) covers the full $180^{\circ }\times 360^{\circ }$ dome FOV. This camera profits from the iterative surface correction applied for the C2 and C3 lenses. Apertures were created from ink for all lenses in the annular camera layout. Here, the C2 lens (middle ring) and the C3 lens (outer ring) arrange around a single center lens C1 (Figs. 8(a) and (b)). Imaging experiments are conducted with a hollow transparent cube laminated with a checkerboard pattern. The wide-angle camera was fabricated on a glass substrate and the image plane is thus recorded with a microscope (Keyence VHX500F, Fig. 8(c)). Here, the glass surface serves as a substitute for the image sensor plane of a fully integrated camera system. Direct fabrication on an image sensor is generally feasible [9,22] and demonstrated in the following paragraph with camera design B. All lenses image a different part of the checker cube in a multi-aperture fashion (Fig. 8(d)). Three single apertures, one for each lens type, are magnified. Both the checkerboard pattern and the edges of the cube are clearly visible in the single images.

 

Fig. 8. Realization of a $180^{\circ }\times 360^{\circ }$ dome imaging system on a glass substrate, based on camera design C. a, Microscope image of the realized lens arrangement with apertures fabricated by an inkjet process shown in b. c, A checker board cube is used as half-space object placed around the camera. Single images of the lenses can be extracted from the microscope overview image (d) where both the checker pattern and edges of the cube are visible.

Download Full Size | PDF

As a second example, the panoramic camera type B covering a FOV of 170$^{\circ }$ is fabricated directly on a RaspberryPi V2.1 image sensor (Figs. 9(a) and (b)). This camera design B has substantially lower theoretical resolution capacity than design C and did not profit from the surface correction methods presented in the previous sections. Therefore, imaging quality is intrinsically lower. In exchange, we demonstrate direct fabrication on an image sensor, distortion correction and image stitching here. The camera consists of five lenses: one B1, two B2 and two B3 lenses which were arranged in an X-shaped pattern for a small form factor. Here, we also observe some residual undeveloped photoresist that manifests itself in the lighter part of the leftmost and rightmost subimages (see also sec. 5.). This camera utilizes the automatic distortion correction method presented above which was calibrated with the checkerboard pattern. Using these static correction values, a set of five rectangular subaperture gray scale images is retrieved (Fig. 9(d)). The lateral sizes of the images of the B1 and B2 lenses are furthermore compressed linearly to resemble the magnification of the B3 lens. These images are finally stitched as depicted schematically with partly overlapping areas in the center and with a direct joint on the sides. The correlation of vertical image lines is simultaneously calculated to support finding the correct stitching positions. Due to limited image contrast, however, static stitching resulted in best panoramic image quality tested with different scenes. An example of a stitched image is presented for a modified arc object with an asymmetric pattern that can be recognized from the stitched image.

 

Fig. 9. Realization of a wide-angle panoramic camera on an image sensor with live image stitching, based on camera design B. a, Micrographs of the single lenses and b, all five lenses required for a full panorama printed directly onto a RaspberryPi V2.1 image sensor, using a dandelion seed for size comparison. c, 3D-printed arc that serves as object for imaging and d, a panoramic image reconstructed from overlapping areas of the distortion corrected single images.

Download Full Size | PDF

7. Conclusion

In conclusion, we presented a set of optical designs for panoramic and $180^{\circ }\times 360^{\circ }$ dome multi-aperture cameras based on refractive, catadioptric, and highly tilted non-rotational freeform surface designs. With manufacturability only as good as measurements, we furthermore developed a set of methods to fabricate, measure, and correct 3D-printed freeform surfaces in arbitrary orientation. Our iterative approach is straightforward and advances producibility of lenses with shape fidelities up to $\lambda /10$ without the need for complex models of time-dependent polymerization processes. Finally, two freeform multi-aperture cameras are realized that profit from our proposed correction methods. While imaging quality can still be improved by combining all learnings from our study, we present a methodology to design, fabricate and realize 3D-printed freeform lenses as a broadly applicable toolbox.