1. Introduction

Optical Tweezers (OT) are a powerful tool for manipulation and force characterization of micron-sized objects [1,2], with a wide variety of applications in living biological systems such as cells, bacteria and molecular motors [3]. Quadrant Photodiodes (QPD) have been commonly employed for indirectly retrieving the Brownian motion of a trapped object by measuring the scattered laser light, thus allowing a measure of the trap stiffness characterized mainly by the bandwidth of the motion spectrum. Lately, a novel approach for characterizing OT using high speed cameras has been presented in [4,5], showing competitive results compared to the use of a QPD. In fact, even inexpensive CMOS cameras allow the simultaneous force characterization and imaging of a trapped object by using a simple setup, avoiding the additional optical equipment related to the QPD, at the expense of a trade-off between the acquisition rate and imaging size, and an extra computational effort. In addition, the use of high speed cameras for OT characterization has shown to be less sensitive to optical alignment than QPD.

In contrast to the use of QPD, the characterization using high speed cameras rely on the proper estimation of the trapped particle displacement directly from the image sequence. Although it is not fully explained nor detailed in [4,5], we assume that the particle motion can indeed be retrieved by using image registration techniques, being the Phase-Correlation (PC) of the Fourier Transform method [6] the most likely choice for providing real-time estimations, knowing also that it can be extended to achieve sub-pixel motion resolution [7], which turns to be a requirement in order to be competitive with the use of a QPD.

Nevertheless, when using a CMOS camera for simultaneously retrieving the motion and also imaging the object, we face the constraint of using only a small part of the available field of view by setting a small region of interest (ROI) in order to achieve faster image acquisition rates than with QPD, thus reducing the available detector resolution. Therefore, in this work we propose to use the retrieved motion information to tackle the lack of resolution by applying multiframe Superresolution Reconstruction (SR) [8] techniques to the low resolution (LR) image sequences, obtaining a higher resolution image sequence for real-time visualization purposes. As a simple definition, multiframe SR is the process of fusing several aliased shifted versions of the same LR scene into a high resolution (HR) grid. A mandatory requirement for successful SR is that there should be enough motion diversity which, in this case, is provided by the Brownian nature of the trapped particle motion. On the other hand, the main practical assumptions for both subpixel motion registration and SR techniques to be properly performed is that the trapped object may remain unchanged during the length of the sequence used for zooming, and that it only suffers from rigid translational motion. Although these conditions hold for unanimated calibration particles, such as spherical polystyrene beads, it is not usually the case when trapping living objects, such as bacteria, where rotational motion patterns are also present. Fortunately, the PC method can also be extended to account for rigid rotations [9], surpassing the translation-only motion constraints, while also leaving the SR process as simple as possible, without having to deal with a more complex motion modeling.

Although a high-speed CMOS camera can perform as a good replacement for the QPD, the available light budget for typical microscope illumination makes unfeasible the split of light towards another imaging device for visualization-only purposes. Therefore, and in order to get the most of the light budget, in this work we demonstrate that with the proposed approach the OT is able to perform the online characterization of the trapped object motion, and forces as well, while still providing high resolution imaging by using a simplified optical setup and a CMOS camera. The rest of this paper is organized as follows. First a description of the experimental setup is presented, including details of the sample preparation and the image acquisition process. Then, we show a characterization of the OT using calibration beads by estimating its translational motion, and a SR example is explained and displayed. Next, the motion registration is extended to deal with living bacteria rotation and translation, which yields to the next SR example. Finally, the results are discussed and summarized in the last section.

2. Experimental setup

We have used a standard OT system as displayed in Fig. 1 , where a collimated laser light is strongly focused through a high Numerical Aperture (N.A.) microscope objective, while the sample plate is white illuminated and real-time monitored by a camera. As usual, a dichroic mirror reflects the laser to enter into the objective lens, while allowing the reflected white light to go into the opposite direction towards the camera. The trapping beam is a highly stable ND:YAG (TEM00) Laser working at 976 nm (ALPHALAS), with a maximum output power of 500 mW, and a power fluctuation of 0.5% in the full operation range. The collimated laser beam is focused with an Olympus UPLSAPO 60XW and 1.20 N.A. water-immersion microscope objective. Video acquisition is performed with an EO-1312M CMOS monochrome USB Camera (Edmund Optics) connected to a personal computer. The OT setup is mounted on an air-suspended optical table (Thorlabs) for achieving maximal isolation, while all possible noise sources such as power supplies and ventilators are placed away from the table. For optical trapping purposes, laser powers are between 1 to 10 mW, but in our case only one miliwatt has been noticed to be fairly enough. The choice of the near infrared laser light is due to the special care needed for handling trapped bacteria with increased bacterial viability, as precisely described in [10,11]. Moreover, to prevent the alignment of the trapped bacterium with the laser beam along the propagation direction, we enforce the use of the smallest trapping power as possible.

 

Fig. 1 Experimental setup: trap imaging and calibration is performed by using the CMOS camera. An infrared trapping laser is used for handling living objects, such as the zoomed Bacillus Subtilis bacteria shown. Lower inset: real system snapshot.

Download Full Size | PDF

The experiments were carried out by trapping particles of two kinds: artificial calibration beads and living bacteria. The unanimated particles consisted in a series of 2 micron polystyrene beads (Kisker-Biotech), and they were deluded at a proportion of 10 ul of concentration in 1.5 ml of distilled water, approximately the volume of an Eppendorf tube. In order to prevent stucking between polystyrene beads, a small quantity of Sodium Dodecyl Sulfate detergent (SDS) was added. For bacterial procedure we selected gram-positive bacteria, whose outer thick peptidoglycan layer gives a better definition and easier appreciation of focusing and borders of the cell wall, in comparison to gram-negative bacteria. A small amount of bacterial culture of Bacillus Subtilis was diluted into an Eppendorf tube using diluted water with approximately 5 ul of Methylene blue and Malachite green to dye the cell wall and inner capsule polysaccharide, respectively. Both dyes are necessary to increase imaging quality, improving overall definition and sharpness. At the moment to carry on the experiments, we prepared a single sample consisting of a tiny drop of solution sealed with two N°1 micro cover glasses at room temperature.

Image sequences were captured setting a ROI at different resolutions depending on the particle type. For the polystyrene beads, we set a ROI of 64x64 at 680 frames per second (fps). On the other hand, we set a ROI of 128x128 at 480 fps for the bacteria. Both resolutions fit without problems over the displacement range of the corresponding particles when they are trapped at different power levels. The frame rates were set at the maximum allowed given the corresponding ROI, yielding a reasonable high frame rate as required for a proper spectral analysis. The exposure time was set at the maximum allowed given the corresponding acquisition rate, while both gain and the incident illumination power were properly balanced for producing unnoticeable fixed-pattern noise levels.

For retrieving the motion data from the image sequences, an efficient and reliable implementation [12] of the subpixel registration PC was used. However, as the main constraint for using PC registration techniques is that the whole image is assumed to be under a global translational motion, we also implemented an extended PC method based on the Fourier-Mellin transform, as proposed in [9], in order to retrieve rotation as well. For both rotation and translation algorithms, images were previously preprocessed by applying spatial windowing masks in order to avoid undesired high frequency Fourier coefficients that lead to poor estimates. Finally, we implemented the Fast and Robust SR algorithm in [13] for obtaining the superresolved image estimates, which uses the motion previously retrieved from the registration stage. The algorithms were chosen in order to allow a feasible real-time implementation.

3. Optical tweezers characterization

As it is well-known, forces exerted on OT can be considered as equivalent to forces following Hooke's Law, where forces are proportional to the displacements with a stiffness constant κ, which is also proportional to the Laser power. The stiffness constant κ can be estimated from the cut-off frequency of the displacement Power Spectrum Density (PSD) of a trapped particle, or from its motion variance as well. Then, in order to validate the characteristics of an OT, we trapped a 2 um polystyrene bead and recorded several video sequences of 15000 frames (nearly 10 seconds at 680 fps) for different laser powers. Then, we retrieved the x and y-axis motion by using the subpixel PC algorithm [12] and estimated the respective PSD of each motion sequence. From the motion data the standard deviation can be computed, and from the PSD the cutoff frequency (bandwidth) of the motion is derived. Sample results for the y-axis motion are displayed in Fig. 2 for two different laser powers. Note that a smoother version (thick line) of the PSD is used for estimating the bandwidth.

 

Fig. 2 y-axis subpixel displacement estimates (up) and its related Power Spectral Density (bottom) for different laser power: (a) 33 mW, motion standard deviation of 0.65 pixels, bandwidth of 2.6 Hz; (b) 186 mW, motion standard deviation of 0.31 pixels, and bandwidth of 5.8 Hz.

Download Full Size | PDF

Results shown in Fig. 2 confirm the ability of the subpixel image registration method to retrieve the motion of the trapped object, as they present the typical behavior expected for the two different laser powers, similarly as when using a QPD. In particular, the motion standard deviation decreases and the bandwidth increases as the trapping laser power becomes higher, corroborating the results reported in [4,5] for the whole range of measured trapping laser powers. Hence, the presented subpixel motion registration allows the proper characterization of OT by using high speed video sequences. In this case we have obtained a stiffness constant κ = 1.14E-6 N/m for the weakly trapped bead in Fig. 2(a), and κ = 1.02E-5 N/m for the strongly trapped bead in Fig. 2(b). Note that the subpixel accuracy of the registration algorithm in [12] can be set according to the computational resources available, but we found that a 0.02 pixel resolution to be enough, particularly when the maximum trapping power is in use and the motion variance is small. Thus, as we are using a 60X microscope and a CMOS camera with a 5.2 um pixel size, we have a resolution of approximately 100 nm per pixel, leading to a final motion registration accuracy equivalent to near two nanometers of resolution.

4. Superresolution

The reduced field of view (FOV) and reduced available resolution offered by the CMOS camera when using it at high capture rates - for allowing the motion characterization of a trapped particle in OT - are clear drawbacks of the used setup. However, the reduced FOV may not be a serious disadvantage, because we may only want to focus the visualization on the trapped particle and not on what is going on its surroundings. On the other hand, the lack of resolution due to the small ROI selected is a real issue for visualization purposes, where also most of the frames are drop in order to match normal visualization rates. Nonetheless, we foresee and propose a solution for tackling the lack of resolution by applying SR techniques to the LR high speed video sequences, in order to reconstruct a magnified HR video sequence, but now at normal visualization rates.

SR in its simplest form is about smartly fusing several subpixel shifted LR versions of the same scene onto a HR grid. The resulting HR image quality is mostly related to the availability and richness of motion diversity, the accomplishment of the global translational motion model assumption, and the motion registration accuracy. Thus, the first and more important step for properly performing SR is the subpixel motion registration, in despite of the other two remaining steps regarding interpolation and deconvolution. Favorably, the tentative application of SR in OT image sequences is triggered by the availability of the required motion diversity, mainly due to the intrinsic presence of the Brownian motion of the trapped particles. Moreover, we assume that the same subpixel PC motion retrieval procedure previously employed for the characterization of the trapped particle motion can also be used to assist the SR reconstruction process.

Here, we apply the SR reconstruction to OT high-speed image sequences of two kinds of trapped particles. In the first case, we use trapped calibration polystyrene beads for illustrating the method. In the second case, as a more realistic and practical example, we use trapped living bacteria. Nonetheless, note that in practical situations other than calibration particles, such as in the second example, an enhanced motion registration method is needed to account at least for the rotation of the trapped particles. Once the zooming factor is chosen, and the number of frames needed to fill the tentative HR grid is selected, both registered translational displacement data, and rotation if required, are used as inputs to the SR method in [13], together with the corresponding image sequence for producing the HR image.

4.1 Polystyrene beads

In this case, we selected a sequence of the first 24 frames from the trapped polystyrene bead 64x64 LR video sequence, originally captured at 680 fps, for achieving a 4X SR factor that would lead to a 256x256 HR image. Even though only 16 frames may be enough for the chosen SR factor to succeed, an extra amount of frames assures the needed motion diversity, yielding also to a tentative display rate of nearly 30fps for the HR video if we repeat the process to the consecutive blocks of 24 frames. Thus, the SR process is performed as follows: First, motion registration is performed by computing the relative motion of the trapped particle at every frame in the sequence as compared to the first frame. Then, the SR algorithm is the final responsible of fusing the data onto the desired HR image estimate by using the registered motion data along the x and y directions, also setting a 4x4 uniform blur kernel as the Point Spread Function (PSF) for the deconvolution stage, given the requirements for the chosen 4x SR factor as widely discussed in the literature [8,13].

Results are summarized in Fig. 3 , where a sample of the first LR frame in the sequence is shown at the leftmost place. For comparison purposes, two 4X digital zoom versions of the same LR frame are shown in Fig. 3(b), 3(c) using nearest neighbor and bicubic interpolation respectively. Finally, the HR image estimated by applying SR over the whole image sequence is displayed in Fig. 3(d). Note that even though the bicubic interpolated version is able to apparently improve the appearance of the zoomed LR image in comparison to the nearest neighbor (or pixel replicated) version, it does not effectively improve the resolution. On the other hand, the SR estimate is not only able to improve the resolution showing a sharper version of the trapped particle, but also to noticeably diminish the noise, which can be seen in both line sections exemplified in the plots of Fig. 3(e) and Fig. 3(f).

 

Fig. 3 Images of a 2 um polystyrene calibration bead. (a) Sample 64x64 low resolution image; (b) 4X nearest neighbor interpolation; (c) 4X bicubic interpolation; (d) 4X multiframe superresolution; (e) Comparison plot of vertical line 1; (f) Comparison plot of vertical line 120.

Download Full Size | PDF

Although the employed SR algorithm has been designed to be robust against registration errors, there are still some minor image artifacts noticeable in the estimated HR image that could be tentatively triggered, for instance, by slight movements over the longitudinal direction, z-axis, due to laser power fluctuations, for example. If that happens, one of the main SR requirements regarding planar motion is not properly fulfilled. Note also that even if there is a rotation of the particle, it is not noticeable due to the symmetry of the polystyrene beads. Therefore, it should not affect the translational only motion assumption; however it can still be responsible of poor motion estimates and image artifacts. Anyway, in the worst case scenario, if the registration is not that accurate, or if the motion hypotheses are not properly accomplished, the robustness of the SR algorithm will still assures an interpolated but denoised HR version of one of the original LR frames. Furthermore, apart to proof the feasibility of applying SR to an OT image sequence, we are able to justify that it is a valuable effort over simple interpolation methods often used for digital zooming, even when this example is the simplest one.

4.2 Living bacteria

Since infrared regime in OT has been successfully applied for manipulation of living biological systems, such as bacteria, both rotational and translational motion have been widely studied in the context of photo-damage characterization, such as flagellum twisting mobility in E. Coli [10], and torque measurements [14]. A renewed interest in studying rotational motion in OT comes both from the study of thermo dynamical properties of small size systems which are far from equilibrium [15], and from the study of molecular motors [16], for example. In this context, we provide a framework to fully characterize the translation and rotation of a trapped bacterium, which is also needed to perform the SR process.

The tri-dimensional motion of the trapped test beads can be well approximated by the planar bi-dimensional rigid translational motion model, due to their spherical symmetry. However, when dealing with non-symmetric trapped living objects, such as in biology, the pure translational model may not hold any longer. For instance, by inspecting the dynamics of trapped bacteria, a good approximation for its real motion has to account both for the rotation over the xy plane and for the x and y-axis translations.

To overcome the additional requirements of an extended motion model in order to incorporate rotation, apart from translation, the registration procedure is now performed in three steps. The whole process is exemplified in Fig. 4 , where we are using a sequence of 5000 frames of a 128x128 LR image sequence of a trapped Bacillus Subtilis captured at 436 fps. The first step is the retrieval of the rotation, which can be clearly noticed in the image samples inside Fig. 4(a), now done by an extended PC method based on the Fourier-Mellin transform [9]. As a second step, once the rotation information is obtained as it is shown in the plot of Fig. 4(a), the image sequence is now compensated for rotation on the opposite direction using a simple matrix rotation transform jointly with bilinear interpolation. The compensated image sequence, now steady in the rotation direction and whose sample frames are displayed inside Fig. 4(b), is the input to the next step. The final step is similar to the one formerly performed to the pure translational motion case previously studied for both the OT characterization and polystyrene bead SR, but now the input image sequence to be registered is the one just compensated for the rotation. Results for the registered translation along the x and y directions are given in Fig. 4(b). Lastly, an additional step can be added, where the image sequence is also compensated for the translation as well as the former rotation, generating a fully motion compensated video, useful not only to visualize the effectiveness of the whole registration process, but also to perform specific analysis that may require a forced virtual steadiness of the trapped object.

 

Fig. 4 Motion estimation of a trapped Bacillus Subtilis. (a) Rotation estimation from the original video sequence (see the inset sample frames) by using the Fourier-Mellin transform; (b) Subpixel translation estimation from the rotation compensated image sequence (see the inset sample frames).

Download Full Size | PDF

The above described procedure seems to produce reasonable results. However, it does not work out accurately for the whole video because the trapped bacterium suffer from elastic deformations throughout the duration of the image sequence, and therefore the requirements of rigid motion imposed by the PC based registration methods are not accomplished. Nevertheless, we are still able to perform the motion retrieval in a block of frames, but the registration would only be valid if the speed of the video sequence is fast enough compared to the rate of change for the intrinsic and morphological dynamics of the bacterium itself, other than the already expected rigid rotational and translational motion. If this is the case, then it is exactly what we need for the successful application of SR.

Similarly to the calibration beads case, we choose to select the first 32 frames of the trapped Bacillus Subtilis video for performing a 4X SR experiment. Note that in this second example of SR, the LR image sequence is of higher resolution, and the frame rate is slower, compared to the video of the first example, and now the expected rate for the reconstructed HR video is about 15 fps, which is still suitable enough for real-time visualization. After performing the extended motion registration procedure, both rotation and translation information, relative to the first frame, is used by the SR algorithm to produce the final superresolved HR image estimate. Again, a uniform 4x4 detector PSF kernel is chosen for the deconvolution stage.

Results for the second SR example are displayed in Fig. 5 , showing a sample of a LR frame at the left, a 4X digital zoom version by using nearest neighbor interpolation at the center, and finally the 4X SR version at the rightmost. Both HR images were cropped in order to not show the rotation and displacement effects on the proximities of the borders of the SR estimate, thus keeping the same aspect ratio and size for comparison purposes. After inspecting Fig. 5(b) against Fig. 5(c), we can realize that the SR version presents better definition with less noise. However, in this case SR does not seem to enhance effectively the resolution up to the requested factor, looking more as a nicely interpolated and upsampled version of the original LR image, but now denoised. The latter can be explained by possible inaccuracies on the motion registration, lack of motion richness to fill the required HR grid, or undesired elastic deformation of the trapping bacterium, violating the requested extended motion model. Nonetheless, again the robustness of the utilized SR method allows incorporating as much information of the consecutive frames as can really help in the resolution enhancement process, yielding a HR image that will fluctuate between a single frame interpolated version, in the worst case, and a really resolution enhanced SR image. That is one of the reasons for using an extra amount of frames than theoretically required for such resolution factor improvement, thus assuring the needed motion variability to properly fill the desired HR grid.

 

Fig. 5 Images of a trapped Bacillus Subtilis. (a) Sample 128x128 low resolution image; (b) 4X nearest neighbor interpolation; (c) 4X multiframe superresolution.

Download Full Size | PDF

Despite the fact of the added complexity given by the extended motion modeling, together with the extra computational resources needed to perform the registration process, we can show that applying SR in OT with a CMOS camera is still a procedure to be considered whenever both visualizing and characterizing motion of living trapped particles is required. Moreover, the motion registration results may still be useful to perform the trap characterization through analysis of the variances or frequency response, and it also opens new opportunities to study and understand the rotation behavior of living bacteria.

5. Discussion and conclusions

One of the main contributions of this work is that it provides a real application of SR, where all theoretical hypotheses are put to test with a practical problem of visualization of high speed OT imaging, originally suited for their characterization. An interesting feature of this application, which is a requirement for SR, is that trapped objects undergo a rich motion structure in a natural form, due to their Brownian nature. The challenges arise from the ways of modeling the motion, to the ways of retrieving or registering such motion. As a first and novel approach in this matter, we proposed a practical solution for pursuing SR, choosing algorithms that allow a feasible real time implementation, providing reasonable and promising results. Another contribution can be summarized as the understanding of the algorithm for the motion retrieval directly from the image sequences, and the conditions and constraints related to its successful application for replacing the formerly used QPD by a single high speed CMOS camera, aimed now for both the visualization and the motion characterization of trapped objects, that also may lead to the force characterization of the OT.

From the above obtained results, we conclude that, apart from retrieving the motion information needed for characterizing the OT, using the whole available data from the high-speed camera to form a higher resolution image sequence at typical display rates is a worthwhile effort instead of just displaying an interpolated and temporally undersampled version of the high speed video frames. Thus, SR allows the formation of a sharper, denoised, and enhanced resolution imaging, only limited by the accuracy of the registration process or possible violations of the assumed motion model.

In this paper, we provided a framework for registering the motion in high speed OT imagery of trapped objects, showing results using calibration polystyrene beads and living bacteria. We shown that the retrieved motion is not only useful for performing force characterization of the trap, but also to allow the reconstruction of superresolved HR imaging from the high speed but low resolution image sequence. Specifically from the results obtained on living bacteria, further research may include the use of the rotation information for studying the viability of trapped bacteria [10,11]. In addition, the SR process may also be helpful in novel research related to the study of inner bacterial synthesis and for improving cell wall imaging [17]. Moreover, it can be tentatively integrated for enhanced results with other microscopy techniques such as phase contrast microscopy, and blurring phase imaging in holographic video microscopy, lately applied to micro-colloidal suspensions [18].

A general overview concerning a real application of superresolution reconstruction in optical tweezers imaging has been presented in this letter. A future goal is the development of the proposed setup and methods in a friendly and agile way for enhancing the experience on the optical table, allowing the simultaneous characterization of the optical trap while the SR images are displayed in the monitor in real-time.