We present a real-time target-locking confocal microscope that follows an object moving along an arbitrary path, even as it simultaneously changes its shape, size and orientation. This Target-locking Acquisition with Realtime Confocal (TARC) microscopy system integrates fast image processing and rapid image acquisition using a Nipkow spinning-disk confocal microscope. The system acquires a 3D stack of images, performs a full structural analysis to locate a feature of interest, moves the sample in response, and then collects the next 3D image stack. In this way, data collection is dynamically adjusted to keep a moving object centered in the field of view. We demonstrate the system’s capabilities by target-locking freely-diffusing clusters of attractive colloidal particles, and actively-transported quantum dots (QDs) endocytosed into live cells free to move in three dimensions, for several hours. During this time, both the colloidal clusters and live cells move distances several times the length of the imaging volume.
©2007 Optical Society of America
The advent of high-speed confocal microscope systems has allowed the rapid, three-dimensional imaging of a number of dynamic processes in physics, materials science and biology [1, 2, 3]. Typically, a fixed three-dimensional volume within the sample is imaged periodically over time, which is normally adequate for samples and systems where objects remain within the field of view for the duration of the experiment. However, many investigations attempt to image dynamic phenomena, where the object of interest can move out of the three-dimensional imaging volume. In soft-condensed matter physics, for instance, free clusters of attractive colloidal particles will typically diffuse out of view on timescales comparable to their growth or internal rearrangement . In biology, many in vivo investigations require the observation of processes involving freely-moving live cells, such as studies of motility or parasitic invasion [4, 5, 6], for even a basic qualitative understanding. For these experiments, immobilizing the cells can interfere with the ability to answer the question of interest [7, 8]. In whole-membrane investigations, for instance, surface-adhered areas of a cell membrane encounter a local chemical environment vastly different from the areas exposed to the medium.
An alternative approach is to target-lock by actively moving the sample to keep a moving object, such as a cell, in the center of the field of view; this enables observation for far longer periods of time. Well-established techniques can target-lock a single object in 3D at high speeds, treating it as an isolated point with no internal structure [9, 10, 11, 12, 13, 14]. However, these single-point techniques are inherently less adept at following objects with prominent internal structure, or multiple objects moving independently.
In this paper, we describe the Target-locking Acquisition with Real-time Confocal (TARC) microscopy system, which can follow a collection of multiple objects as they move along arbitrary 3D paths, even with significant changes in shape, size and orientation. Instead of following a single bright spot, we image multiple fluorescent objects, determine their positions and structure in three dimensions, and target-lock by moving the sample in response to geometric analysis of these data. The system integrates rapid image analysis into the data acquisition process, so that the results of analyzing one 3D stack of images influence the collection of the next stack. A departure from many confocal experiments, the volume we image in the sample is not fixed in space, but instead is moved in response to dynamic changes within the sample.
We demonstrate the TARC system by target-locking two systems: freely-diffusing clusters of attractive colloids, which change their shape, position, orientation and size throughout the experiment; and actively-transported quantum dots endocytosed into live cells free to move in three dimensions. The complete proprietary source code, project files, executables, electronic circuit diagrams and detailed schematics have been made freely available online for academic research purposes only (see website).
2. Materials and methods
2.1. Apparatus overview
To target-lock a moving object, the TARC system first acquires a 3D stack of data, rapidly collecting a sequence of 2D confocal images from successive planes in the sample perpendicular to the optic axis. We use a Nipkow-disk confocal scanner (NCS) and CCD camera to collect these images, but any confocal, multi-photon or related technique could be equivalently used to acquire a 3D image stack. Next, the system processes the images and performs a full structural analysis to identify and characterize the object it is target-locking. In the examples here, the TARC system determines the exact position of the center of mass (COM) of the largest object in the sample, and moves the microscope stage to bring that point to the center of the 3D imaging volume. The next 3D image stack is then acquired. Image collection and image analysis alternate, so that the results of analyzing one stack determine the position where the next stack is acquired.
In order to target-lock moving objects as quickly as possible, we chose an NCS for high speed, and a piezo-based objective translator to allow rapid access to different sample planes perpendicular to the optic axis. Separately, to reposition the imaging volume between 3D stacks, where speed is not crucial, the microscope stage could be driven along three orthogonal axes with stepper motors.
A major challenge is coordinating the actions of all the hardware and software components quickly enough for effective target-locking. One particular issue with most NCS systems is that the disk spins freely at one rate, the camera acquires streaming images at a different rate, and there is no external synchronization between the two. This phase mismatch can significantly constrain the maximum frame rate: fringing moiré patterns, and eventually large overall intensity fluctuations, appear in the acquired images as frame rates increase. In addition, piezo-based microscope objective translation is usually controlled via software on the host PC in many commercial implementations, which does not allow the precise timing control needed to move the piezo only during the few milliseconds after each frame when the camera is not collecting data.
Sufficient sub-millisecond timing precision cannot be achieved in software (using a Windows PC) alone, and attempts with an internal control board were swamped by electrical noise generated inside the PC. Instead, we opted for hardware external to the host PC for timing control, designing and building a custom pulse generator which triggers and synchronizes camera exposure, spinning-disk rotation rate and piezo translation, with 10 μs temporal precision.
Before data collection begins, the host PC initializes and uploads control parameters to the camera, piezo controller and pulse generator. The PC then signals the pulse generator to begin data collection. From that point onward, the PC receives camera images and analyzes them, moving the automated stage once per 3D stack to implement target-locking, but otherwise performs no timing control. The rest of the hardware is synchronized by the pulse generator.
2.2. Hardware description
A block diagram of the TARC system’s main hardware components is shownin Fig. 1.The main optical components are attached to an upright microscope (Leica DMRXA). Laser excitation is provided by a 532-nm Nd:YVO4 diode-pumped solid-state laser (CrystaLaser CGL-050-L), with a shutter controlled by a TTL signal from the pulse generator. The laser beam is coupled into a single-mode (TEM00) fiber, which delivers a few milliwatts of light into a commercial NCS (Yokogawa CSU-10B).
The internal components of the NCS are depicted within the dotted grey rectangle in Fig. 1, briefly summarized here (see [15, 16] and references therein for discussion of the optical characteristics of this NCS). Two parallel disks, one with microlenses and one with pinholes, are rigidly fixed to a single shaft driven by a variable-speed motor. The motor controller accepts a TTL pulse for synchronization (e.g. to phase-match an NTSC video signal), which is supplied by the pulse generator. The beam exiting the fiber into the NCS hits the upper disk, which contains thousands of micro-lenses, and is split into numerous small mini-beams. These pass through a dichroic mirror fixed between the two spinning disks and are focused through a set of pinholes in the second disk. The mini-beams are then focused by the objective onto the sample, where they excite fluorescence in the focal plane. The objective then focuses the corresponding emission mini-beams back through the pinholes, which block light originating from other planes in the sample and thereby create confocal depth-sectioning. The Stokes-shifted emission mini-beams are reflected by the dichroic and imaged onto a cooled-CCD camera (QImaging Retiga 1394 EXi Fast). Rotating the disks, which have a spiral pattern of microlenses and pin-holes, moves the excitation mini-beams within the sample focal plane in such a way to ensure uniform sample coverage. The CCD camera is configured by, and transfers data to, the host computer via IEEE1394 firewire, but is triggered by separate electronically-independent TTL logic circuitry, which we access with signals from the pulse generator. To ensure smooth data collection at high rates, the host PC is equipped with a hardware-based RAID5 array of 10,000 rpm Ultra320 SCSI drives (Seagate).
Because of the confocal pinholes, only light from the focal plane of the objective reaches the detector, so the objective must be physically translated to access planes at different depths within the sample. This is accomplished using a piezo-based microscope objective translator (Physiks Instruments PiFOC) with a high-accuracy closed-loop controller (Physiks Instruments E662K001), configured via RS232 by the host PC, but triggered separately with TTL logic pulses from the pulse generator. The PC uploads a list of positions into a memory buffer on the controller, and each time a TTL pulse is received from the pulse generator (on a separate coaxial input, isolated from the RS232), the piezo moves to the next value on the list. In this way, the sequence of precise positions can be loaded and stored before the experiment begins, and accessed with great temporal precision via TTL triggering.
The pulse generator contains a microcontroller to manage RS232 communication with the host PC, and a number of counters and comparators implemented on several CPLDs, which generate repeated bursts of pulses of programmable number, period and delay, to several TTL outputs. An example pulse sequence, showing relative timings of the TTL signals sent by the pulse generator to the other parts of the TARC system, is shown in Fig. 2.
The microscope stage (Marzhauser) is controlled, independent of the piezo, by stepper motors along three axes. The microscope stand’s electronic focus control moves the stage up and down, along the z axis (the optic axis), while a separate controller (Leica DMSTC) controls the x-y motion. Because the stage is moved only once per 3D image stack, precise timing control is not needed; the stage is controlled by software via RS232, with no TTL triggering by the pulse generator.
2.3. Software overview
The main acquisition program performs several functions: it initializes and configures the pulse generator (with numbers and timings of the pulses), the piezo controller (list of positions to move through when triggered) and the camera (imaging parameters); and it subsequently manages the data acquisition by writing individual image files to disk as soon as each 2D image is delivered via firewire from the camera. Each image is stored as single compressed 8-bit grayscale TIF file, universally accessible from any image-editing program. This represents a significant departure from the operation of most commercial confocal implementations, which typically combine 2D images into 3D stacks in a temporary memory buffer before writing out huge, cumbersome aggregated data files to disk. The size of this temporary buffer, typically a few gigabytes, is comparable to the amount of system RAM or OS-dependent single-file maximum size, and represents the largest amount of data that can be collected without interruption. By contrast, writing each 2D frame to disk individually requires only small megabyte-size memory buffers, which are then cleared and recycled immediately. The main acquisition program therefore executes in just a few megabytes of RAM, with continuous real-time data-streaming to disk limited only by total disk capacity. Images have been acquired continuously for days without interruption, resulting in tens of gigabytes of uninterrupted image data.
After each 3D image stack has been collected, the main acquisition program launches a wrapper program that manages the target-locking system by calling several other programs to analyze the images and move the stage in response. All programs execute from the command-line to maximize speed and facilitate automated scripting, and were written in platform-independent C++. Using fully object-oriented classes and wrappers not only abstracts the hardware details from the programmer, but also facilitates a completely modular software architecture for the analysis. In particular, while the image analysis protocol in this example target-locks by moving the stage to keep the COM of the largest cluster of bright objects centered in the 3D imaging volume, any program that calculates a final stage displacement from analyzing 3D image data can be used in place of these routines, with only trivial changes to the wrapper program. Although this flexible modular approach implicitly prevents the main acquisition from knowing how quickly the independent image analysis and stage displacement will execute, the hardware delay between image stacks (= T7 -T 6 in Fig. 2) is controlled by a single parameter in a configuration text file, and can be very easily measured in a few trial runs.
Our hybrid approach for 3D particle location identifies centroids in 2D images (largely based on a well-known algorithm ), then links up the text-data positions afterward into full 3D positions. Processing only a single 2D image at a time gives a number of performance advantages over the alternative approach of loading an entire 3D data set into memory at once. First, only a single image (of at most a few megabytes) needs to reside in memory at any given time, instead of the hundreds of megabytes of a typical 3D stack. Second, the optimized image-processing libraries used to increase performance are explicitly designed to work with 2D images, loading image data into the processor cache and parallel registers in a particular way to accelerate filtering operations that require access to adjacent rows of pixels; there is no corresponding method to do so for 3D data. Combining multi-threaded libraries with a vectorizing compiler (Intel) to take advantage of special features in recent processors yielded significant speed increases of several orders of magnitude relative to the standard implementations in MatLab and IDL, even when compiled. This speed increase, resulting from the hybrid particle-location strategy and optimized code, ultimately enables the TARC system to target-lock fast enough to be useful experimentally, with very modest requirements for the host PC (here a 2 GHz Pentium 4 Xeon, 256 MB of RAM, Dell).
Finally, note that trivially changing a configuration text file can set the TARC system to acquire 3D image stacks with a fixed x-y-z displacement between stacks, without running any image analysis. This capability can be used for sampling a much larger area, for example to gain better statistics in a measurement; for tiling adjacent 3D image stacks to make large composite images; or to sample a predetermined pattern or matrix of 3D volumes in the sample. Thus, in addition to running with full target-locking, the TARC system can also easily operate as a general-purpose, high-speed automated confocal acquisition system.
2.4. Sample preparation
To demonstrate the ability of the TARC system to properly target-lock highly anisotropic groups of objects over long times, we imaged aggregating clusters of attractive colloidal spheres . Colloidal spheres (1.1 μm diameter) of polymethylmethacrylate (PMMA) with embedded DiIC18 fluorescent dye were suspended in a mixture of bromocyclohexane and decahydron-aphthalene (Aldrich) in a proportion (nearly 5:1 by mass) that precisely matches the density of the particles, and sufficiently closely matches their index of refraction to enable confocal microscopy. Tetrabutyl ammonium chloride (Fluka), an organic salt, was added to screen Coulombic charge repulsion. Attraction between colloids was induced by the addition of nonadsorbing 11.6 MDa linear polystyrene (Polymer Labs), causing the colloidal spheres to aggregate into clusters several microns across, which diffuse as they continuously grow.
To demonstrate the capability to image living systems, we imaged live human lung cancer cells that actively transport endocytosed quantum dots (QDs) . Human lung cancer cells (A549) were cultured in Dulbecco’s Modified Eagle Medium (DMEM, ATCC) supplemented with 10% fetal bovine serum (FBS) at 37°C and 5% CO2. For QD aggregate endocytosis, streptavidin-coated QDs (Invitrogen) with emission at 655 nm were combined with an equal volume of biotinylated poly-arginine (Invitrogen). The mixture was incubated at room temperature for 10 minutes, and the functionalized QDs were introduced to the cell culture at 200 pM. Following a one-hour incubation under normal culturing conditions, the medium was replaced and aggregate endocytosis was allowed to occur over 18 hours. In order to visualize the cell membrane, Alexa Fluor 532-labeled streptavidin (Invitrogen) was combined with the aforementioned biotinylated poly-arginine, and the resulting complex was introduced to the cell culture at about 1 nM one hour before imaging and incubated under the normal culturing conditions. Immediately prior to imaging, the cell culture was trypsinized, and the cells were introduced to the imaging chamber following trypsin inhibition.
To explore target-locking in faster-moving prokaryotic cells, we also imaged quantum dots inside E. coli. BL21(DE3)pLysS E. coli cells were grown to mid-log phase in standard LB medium in a 37°C shaker. The cells were then incubated for one hour at room temperature following the addition of 1 nM streptavidin-coated quantum dots conjugated to biotinylated polyarginine. The cells were pelleted by centrifugation at 1500 g for 10 minutes and re-suspended in fresh LB medium before imaging.
3. Results and discussion
We imaged the colloidal clusters with a 100 × 1.4NA oil-immersion objective (Leica), collecting and analyzing a 3D stack of 61 images, each 500×500 pixels, every 40 seconds. Image collection took 6 s, and analysis took ~2 s, for each stack. As shown in Fig. 3, the TARC system properly target-locked the freely-diffusing single central cluster under a variety of circumstances: when other, smaller clusters entered and left the imaging volume [Figs. 3(a)–3(c)]; when two smaller clusters merged to form a single cluster, dramatically changing shape and size [Figs. 3(c)–3(d)]; and when a highly non-spherical cluster changed orientation [Figs. 3(d)–3(e)]. Proper target-locking was observed for 36,000 seconds (10 hours; full movie online), as the central cluster diffused a distance many times its own length, and several times that of the 24 × 24 × 16 μm3 imaging volume [Figs. 3(f)–3(g)].
We imaged the live human lung cancer cells with a 63 × 1.2NA water-immersion objective (Leica) at 37°C, collecting and analyzing 3D stacks of 61 images, each 300×300 pixels, every 10 seconds. Image collection took 6 s, and analysis took <1 s, for each stack. As shown in Fig. 4, the TARC system properly target-locked the living lung-cancer cell for more than 5,000 seconds (1.4 hours; full movie online). During this entire time, we observed active transport of the vesicle-enclosed quantum dot aggregates, which moved significantly relative to the cell membrane [Figs. 4(a)–4(b)], while the cell itself moved 50 μm, many times its own length and that of the 14 × 14 × 13 μm3 imaging volume [Figs. 4(c)–4(d)]. We also target-locked QDs absorbed by faster-moving E. coli by running at higher speeds, collecting and analyzing 3D stacks of 40 images every 5 seconds for several hours (data not shown).
In all of these examples, we collected ten 2D images per second with the CCD camera, limited by exposure time and readout speed. Our host PC processed each 3D image stack in at most a few seconds, and often much faster, so our efficient image processing scheme probably did not limit the speed in these cases. Ultimately, it appears that the mechanical stability of the piezo objective translator limits 3D acquisition to around thirty 2D images per second (video rate). Maximizing for speed, we successfully acquired complete 3D stacks of 40 images, each 512 × 400 pixels, every 2 seconds with an EMCCD camera (QImaging), whose greater sensitivity permitted far lower exposure times than the standard cooled CCD. In all cases, the TARC system ran indefinitely, and we have target-locked colloid clusters continuously for more than a day, generating thousands of 3D stacks.
This long-time stability is only possible by performing a full 3D reconstruction and locking onto a specific geometric feature determined in a complete structural analysis. This is a significant advance over previous systems, where the image processing consists of finding the intensity maximum within the imaging volume and following it . When multiple objects enter the imaging volume, systems with this approach can lock onto a point (the effective center of intensity) that lies outside of all the fluorescent objects, and may subsequently lose the proper target. By contrast, as shown in Fig. 3, the TARC system gracefully handles multiple objects coming in and out of the imaging volume, while keeping the largest cluster stably centered. Moreover, target-locking onto any well-defined point within a cluster, selected by any number of other structural characteristics (e.g. radius of gyration, fractal dimension, or density) instead of the mass, requires trivial changes to the code and will incur no performance penalty.
Even more generally, while the image analysis described here specifically identifies clusters of fluorescent objects, it is an independent program that executes separately from the main image acquisition program. This independence allows substitution of any analysis program, in any language, that takes a set of images as input and outputs a stage displacement. In this way, pre-existing image analysis routines, currently used to analyze data after image collection has ended, could be redeployed for active target-locking using the TARC system, thereby controlling the data acquisition process itself.
Finally, the examples here highlight direct imaging, but the TARC system could also be used as a target-locking system orthogonal to primary data collection, operating through one microscope camera-port and periodically moving the stage to track a freely-moving object, while data is collected simultaneously with an entirely separate technique. And as previously mentioned, while an NCS was chosen for several practical reasons, primarily high time resolution, our target-locking scheme should also work with other types of confocal or multi-photon systems.
By making the TARC system’s designs and code freely available for academic research purposes only, we hope that its unique capabilities will enable new and unique contributions to understanding dynamic interactions in physics, materials science and biology.
This work was supported by NASA (NAG3-2284), the NSF (DMR-0243715), the Harvard MRSEC (DMR-0602684) and the Marie-Curie RTN for Dynamical Arrest (MRTN-CT-2003-504712). P. J. L. was supported by an NDSEG graduate fellowship and received additional funding from C. & F. Lu. P. A. S. was supported by an NSF Graduate Research Fellowship. 3D reconstructions were created with Pixar’s RenderMan. We thank Prof. X. Sunney Xie for providing quantum dots and cell culture facilities; A. Schofield for particle synthesis; D. Hitrys for camera assistance; J. Crocker and E. Dufresne for help with algorithms; W. Walker for circuit assembly; D. Maas and Pixar for RenderMan assistance; D. Cotter and M. Christiansen for help with video production; and C. Evans for critical reading of the manuscript.
References and links
1. P. J. Lu, “Confocal Scanning Optical Microscopy and Nanotechnology,” in Handbook of Microscopy for Nan-otechnology, N. Yao and Z. L. Wang, eds. (Kluwer, 2005), pp. 3–24.
4. M. E. Wickham, M. Rug, S. A. Ralph, N. Klonis, G. I. McFadden, L. Tilley, and A. F. Cowman, “Trafficking and assembly of the cytoadherence complex in Plasmodium falciparum-infected human erythrocytes,” EMBO J. 20, 5636–5649 (2001). [CrossRef] [PubMed]
5. B. Gligorijevic, R. McAllister, J. S. Urbach, and P. D. Roepe, “Spinning Disk Confocal Microscopy of Live, Intraerythrocytic Malarial Parasites. 1. Quantification of Hemozoin Development for Drug Sensitive versus Resistant Malaria,” Biochemistry 45, 12400–12410 (2006). [CrossRef] [PubMed]
6. B. Gligorijevic, R. McAllister, J. S. Urbach, and P. D. Roepe, “Spinning Disk Confocal Microscopy of Live, Intraerythrocytic Malarial Parasites. 2. Altered Vacuolar Volume Regulation in Drug Resistant Malaria,” Biochemistry 45, 12411–12423 (2006). [CrossRef] [PubMed]
7. T. A. Camesano, M. J. Natan, and B. E. Logan, “Observation of Changes in Bacterial Cell Morphology Using Tapping Mode Atomic Force Microscopy,” Langmuir 16, 4563–4572 (2000). [CrossRef]
8. N. Arhel, A. Genovesio, K.-A. Kim, S. Miko, E. Perret, J.-C. Olivo-Marin, S. Shorte, and P. Charneau, “Quantitative four-dimensional tracking of cytoplasmic and nuclear HIV-1 complexes,” Nat. Meth. 3, 817–823 (2006). [CrossRef]
10. I. M. Peters, B. G. de Grooth, J. M. Schins, C. G. Figdor, and J. Greve, “Three dimensional single-particle tracking with nanometer resolution,” Rev. Sci. Instrum. 69, 2762–2766 (1998). [CrossRef]
11. G. Rabut and J. Ellenberg, “Automatic real-time three-dimensional cell tracking by fluorescence microscopy,” J. Mi-crosc. 216, 131–137 (2005).
12. V. Levi, Q. Q. Ruan, and E. Gratton, “3-D Particle Tracking in a Two-Photon Microscope: Application to the Study of Molecular Dynamics in Cells,” Biophys. J. 88, 2919–2928 (2005). [CrossRef] [PubMed]
13. H. Cang, C. M. Wong, C. S. Xu, A. H. Rizvi, and H. Yang, “Confocal three dimensional tracking of a single nanoparticle with concurrent spectroscopic readouts,” Appl. Phys. Lett. 88, 223901 (2006). [CrossRef]
14. T. Ragan, H. Huang, P. So, and E. Gratton, “3D Particle Tracking on a Two-Photon Microscope,” J. Fluorescence 16, 325–336 (2006). [CrossRef]
15. A. Egner, V. Andresen, and S. W. Hell, “Comparison of the axial resolution of practical Nipkow-disk confocal fluorescence microscopy with that of multifocal multiphoton microscopy: theory and experiment,” J. Microsc. 206, 24–32 (2002). [CrossRef] [PubMed]
16. E. Wang, C. M. Babbey, and K. W. Dunn, “Performance comparison between the high-speed Yokogawa spinning disc confocal system and single-point scanning confocal systems,” J. Microsc. 218, 148–159 (2005). [CrossRef] [PubMed]
17. J. C. Crocker and D. G. Grier, “Methods of Digital Video Microscopy for Colloidal Studies,” J. Colloid Interface Sci. 179, 298–310 (1996). [CrossRef]
18. X. L. Nan, P. A. Sims, P. Chen, and X. S. Xie, “Observation of Individual Microtubule Motor Steps in Living Cells with Endocytosed Quantum Dots,” J. Phys. Chem. B. 109, 24220–24224 (2005). [CrossRef] [PubMed]