This paper presents a technique that reaches 3-D shape measurement speed beyond the digital-light-processing (DLP) projector’s projection speed. In particular, a “solid-state” binary structured pattern is generated with each micro-mirror pixel always being at one status (ON or OFF). By this means, any time segment of projection can represent the whole signal, thus the exposure time can be shorter than the projection time. A sinusoidal fringe pattern is generated by properly defocusing a binary one, and the Fourier fringe analysis means is used for 3-D shape recovery. We have successfully reached 4,000 Hz rate (80 µs exposure time) 3-D shape measurement speed with an off-the-shelf DLP projector.
©2010 Optical Society of America
With recent advances in computational technology and shape analysis, high-speed 3-D shape measurement has become unprecedentedly important. Over the years, a number of techniques have been developed. Among these techniques, fringe analysis stands out because of its numerous advantages . A Fourier method reaches the fastest 3-D shape measurement rate because it only requires a single fringe pattern .
Conventionally, the fringe patterns are either generated by a mechanical grating or by a laser interference. These techniques have been widely applied to measuring numerous extreme phenomena . However, it is typically not very flexible for them to adjust the fringe pitch (period) at a desired value.
Digital fringe projection techniques, recently emerged as a mainstream, have the advantage of generating and controlling the fringe pitch accurately and easily. In such a system, a digital video projector is used to project the computer generated sinusoidal fringe patterns onto the object, and the camera is used to capture the fringe patterns scattered by the object, 3-D information can then obtained from the phase map once the system is calibrated.
Over the years, a number of fringe projection techniques have been developed including some high-speed ones [4–6]. As noted in Ref. , because of its digital fringe generation nature, the 3-D shape measurement speed is ultimately determined by the fringe projection rate: typically 120 Hz for a digital-light-processing (DLP) projector. For high-speed applications, using the minimum exposure time is always desirable. However, because the DLP projector generates the grayscale fringe images by time modulation . This means that if a grayscale image is used, the camera must be precisely synchronized with the projector in order to correctly capture the projected image channel by channel. In other words, the camera must start its exposure when the projector starts channel projection, and must stop its exposure when the projector stops projecting that channel. A conventional digital fringe projection technique, unfortunately, uses all grayscale values, thus the synchronization must be very precise to achieve 120 Hz 3-D shape measurement rate. In this paper, we will experimentally demonstrate that needs for precise synchronization if a conventional sinusoidal fringe pattern is used.
Because of the aforementioned fringe image generation mechanism of the DLP projector, the camera exposure time cannot be shorter than the single channel projection time (1/360 sec) for a 120 Hz projector. This limits its application to measure very fast motion (e.g., vibration, rotating fan blade, etc) when a very short exposure time is required. Our experiments demonstrated that in order to capture the blade of a rotating fan at 1793 rotations per second (rps), tens of µs exposure time is required. Therefore, it is impossible for a conventional fringe projection system to achieve the 3-D shape measurement speed faster than the DLP projector’s projection speed (typically 120 Hz), and is impossible for them to use exposure time shorter each individual channel projection time (typically 1/360 sec).
To capture very fast motions, a solid-state fringe pattern is usually desirable and a Fourier method  is usually necessary. The solid-state fringe pattern can be generated by a mechanical grating, or by a laser interference. However, as addressed earlier, it is very difficult for a digital fringe projection technology to produce solid-state fringe pattern, because it typically refreshes at 120 Hz. On the other hand, because of its inherently digital fringe generation nature, the digital fringe generation technique has some advantageous features including the flexibility to generate fringe patterns.
Because the projector is inherently a digital device, using binary structured patterns for 3-D shape measurement is advantageous. This leads to the exploration of utilizing binary structured patterns to generate sinusoidal ones to potentially overcome the speed bottleneck. Structuredlight technologies based on binary structured patterns have been extensively studied and well established . Typically, for such a system, multiple structured patterns are needed to achieve high spatial resolution. To reach real-time, the structured patterns must be switched rapidly and captured within a short period of time. Rusinwiski et al.  developed a real-time 3-D model acquisition system based on stripe boundary code . Davis et al. has developed a realtime 3-D shape measurement system based on Spacetime stereo vision method . Recently, Narasimhan et al. developed a temporal dithering technique for 3-D shape measurement . However, unlike an aforementioned sinusoidal fringe analysis technique, it is difficult for any of binary structured pattern based methods to reach pixel-level spatial resolution because the stripe width must be larger than one projector pixel . In addition, because it is required to switch structured patterns, the speed is even lower than the projector’s projection speed.
This research is to combine the binary structured light method with sinusoidal fringe analysis technique to achieve both high spatial and high temporal resolution. It is to enable digital fringe projection technique to generate “solid-state” by employing our recently developed flexible 3-D shape measurement technology through defocusing . For this technique, instead of using 8-bit grayscale fringe images, the binary graylevel (0s or 255s) is used. This coincides with the fundamental image generation mechanism of the DLP technology that operates the digital micro mirrors in binary status (ON of OFF). Therefore, theoretically, if a micro mirror is set to be a value of 0 or 255, it should stays OFF or ON all the time. By this means, the micro mirror will act as solid-state (does not change), thus the solid-state light should be generated. These binary structured patterns can be converted to seemingly sinusoidal ones if the projector is properly defocused . Therefore, by this means, this technique has both advantages of the fringe analysis based technique (high spatial resolution) and the binary structured pattern technique (high temporal resolution).
To verify the performance of the proposed technology, an inexpensive off-the-shelf DLP projector (less than $400) is used to generate the sinusoidal fringe patterns, and a high-speed CMOS camera is used to capture the fringe images reflected by the object. Our prototype system has successfully reached 4,000 Hz rate (80 µs exposure time) 3-D shape measurement speed with an off-the-shelf DLP projector. In contrast, if a conventional fringe generation technique is used, once the capturing rate goes beyond 360 Hz, the waveform of the capture fringe pattern becomes nonsinusoidal in shape, and measurement error will be significantly increased. Because the fringe pattern is generated digitally, this proposed technique provides an alternative flexible approach for high-speed 3-D shape measurement that is traditionally utilizes a mechanical grating, or a laser interference.
Section 2 introduces the principle of the proposed technique. Section 3 shows some experimental results. Section 4 discusses the advantages and limitations of the proposed technology, and Sec. 5 summarizes this paper.
2.1. Revisit of digital-light-processing (DLP) technology
Digital light processing (DLPTM) concept originated from Texas Instruments in the later 1980’s. In 1996, Texas Instruments began its commercialized DLPTM technology. At the core of every DLPTM projection system there is an optical semiconductor called the digital micro-mirror device, or DMD, which functions as an extremely precise light switch. The DMD chip contains an array of hinged, microscopic mirrors, each of which corresponds to one pixel of light in a projected image.
Figure 1 shows the working principle of the micro mirror. Data in the cell controls electrostatic forces that can move the mirror +θL (ON) or −θL (OFF), thereby modulating light that is incident on the mirror. The rate of a mirror switching ON and OFF determines the brightness of the projected image pixel. An image is created by light reflected from the ON mirrors passing through a projection lens onto a screen. Grayscale values are created by controlling the proportion of ON and OFF times of the mirror during one frame period - black being 0% ON time and white being 100% ON time.
DLPTM projectors embraced the DMD technology to generate the color images. All DLPTM projectors include light source, a color filter system, at least one digital micro-mirror device (DMD), digital light processing electronics, and an optical projection lens. For a single-chip DLP projector, the color image is produced by placing a color wheel into the system. The color wheel, that contains red, green, and blue filters, spins at a very fast speed, thus red, green, and blue channel images will be projected sequentially onto the screen. However, because the refreshing rate is so high, human eyes can only perceive like a color image instead of three sequential ones.
A DLP projector produces a grayscale value by time integration . A simple test was performed for a very inexpensive DLP projector, Dell M109S. The output light was sensed by a photodiode (Thorlabs FDS100), and photocurrent is converted to voltage signal and monitored by an oscilloscope. The projector has an image resolution of 858 × 600, and 10,000 hours of life time. The brightness of the projector is 50 ANSI Lumens. The projection lens is a F/2.0, f = 16.67 mm fixed focal length one. The projection distance is approximately 559–2,000 mm. The DMD used in this projector is 0.45-inch Type-Y chip. The photodiode used has a response time of 10 ns, an active area of 3.6 mm × 3.6 mm, and a bandwidth of 35 MHz. The oscilloscope used to monitor the signal is Tektronix TDS2024B, the oscilloscope has a bandwidth of 200 MHz.
Figure 2 shows some typical results when it was fed with uniform images with different grayscale values. The projector synchronizes with the computer’s video signal through VSync. If the pure green, RGB = (0, 255, 0), is supplied, there are five periods of signal output for each VSync period, and the signal has the duty cycle of almost 100% ON. When the grayscale value is reduced to 128, approximately half of the channel is filled. If the input grayscale value is reduced to 64, a smaller portion of the channel is filled. These experiments show that if the supplied grayscale value is somewhere between 0 and 255, the output signal becomes irregular. Therefore, if a sinusoidal fringe pattern varying from 0 to 255 is supplied, the whole projection period must be captured to correctly capture the image projected from the projector. This is certainly not desirable for high-speed 3-D shape measurement where the exposure time must be very short.
2.2. Principle of generating fringe pattern by defocusing
In the previous section, we have discussed that if the micro mirror of the DLP projector is fed with 0 or 255, it will remain the state of OFF or ON 100% of time. Therefore, if only 0 or 255 is used for each pixel, the projected light will be solid-state. This provides the insight that it might be feasible to generate solid-state fringe patterns by the DLP technology. However, it also indicates that only 0s or 255s can be used in order to do so. This means that it is impossible to generate 255 gray level sinusoidal fringe patterns in a conventional fashion.
Defocusing has been used to get rid of pixel effects for a long time, but using it to make smooth irradiance profiles is new. It also has been used to 3-D shape measurement using Ronchi grating . Our recent study showed that by properly defocusing a binary structured pattern, an approximately sinusoidal one can be generated . Figure 3 shows some typical results when the projector is defocused to different degrees while the camera is in focus. It shows that if the projector has a different defocusing level, the binary structured pattern is distorted to different degree. Figure 3(a) shows the result when the projector is in focus: clear binary structures on the image.With the degree of defocusing increasing, the binary structures become less and less clear, and the sinusoidal ones become more and more obvious. However, if the projector is defocused too much, sinusoidal structures start diminishing, as indicated in Fig. 3(f). Figure 3(g)–3(l) illustrate one cross sections of the associated fringe patterns. This experiment indicates that a seemingly sinusoidal fringe pattern can indeed be generated by properly defocusing a binary structured pattern.
2.3. Fourier method for 3-D shape measurement
Fourier method for 3-D shape measurement was proposed by Takeda and Mutoh in 1983 , and has been widely applied to many applications . This technique has the advantage of 3-D shape measurement speed because only one single fringe image is required. Essentially, it takes one single fringe images to perform Fourier transform, a band-pass filter is applied to keep the carrier frequency component, and finally the phase is obtained by applying an inverse Fourier transform for phase calculations. Typically, a fringe pattern can be mathematically represented as
where a(x,y) is the DC component or average intensity, b(x,y) the intensity modulation or the amplitude of the carrier fringes, and ϕ(x,y) the phase to be solved for.
Equation (1) can be rewritten in complex form as
If a bandpass filter is applied in the Fourier domain so that only one of the complex frequency component is preserved, we will have
Then the phase can be calculated by
here Im(X) is to take the imaginary part of the complex number X, and Re(X) to get the real part of the complex value X. This equation provides phase values ranging from −π to +π. The continuous phase map can be obtained by applying a phase unwrapping algorithm . 3-D coordinates can be calculated once the system is calibrated . However, in practice, because the projector is defocused, a conventional projector calibration technique does not apply. Therefore, the whole system calibration is very challenging.
In this research, we use a conventional approximation approach to calibrate the system, as described in Ref. . This technique is essentially to measure a known step height object relative to a flat reference plane, and calibrate the linear coefficient (Kz) between the phase changes and the true height of the step. The x and y are also linearly scaled (Kx, Ky) to match the real dimension. All the measurement is performed relative to the reference plane.
To verify the performance of the proposed algorithm, we developed a 3-D shape measurement system as shown in Fig. 4. We used the same LED projector, Dell M109S whose cost is less than $400, and is very compact. The camera used in this system is a high-speed CMOS camera, Phantom V9.1 (Vision Research, NJ), it can capture 2-D images at 2,000 Hz rate with a image resolution of 480 × 480. The exposure time used for all experiments is 250 µs. Because the brightness of the projector is not enough if the camera has a very short exposure time, a converging lens is placed in front of the projector is focus the projected image onto an area of approximately 67 mm × 50 mm.
We first measured a static object using the system described above. Figure 5 shows the measurement result. Figure 5(a) shows the photograph of the sculpture to be measured. Figure 5(b) shows the captured fringe image that shows seemingly sinusoidal patterns. A 2-D Fourier transform is then applied the fringe image that will result in the map in frequency domain as shown in Fig. 5(c). Once a proper band-pass filter is applied, the wrapped phase can be obtained by applying Eq. (4). Figure 5(d) shows the wrapped phase map. A phase unwrapping algorithm  is then applied to unwrapped the phase obtained continuous phase map as shown in Fig. 5(e). The unwrapped phase map can be converted to 3-D coordinates using a phase-to-height conversion algorithm introduced in Ref. . Figure 6 shows the 3-D plot of the measurement. The result looks good, however, some residual stripe errors remains. This might be because the defocusing technique cannot generate ideal sinusoidal fringe patterns, and a phase error compensation algorithm needs to be developed to reduce this type of errors.
As a comparison, we used the same system set up and a conventional sinusoidal fringe generation method to capture the fringe images at 2,000 Hz rate and 250 µs exposure time. The image resolution for this experiment is again 480 × 480. Figure 7 shows some typical recorded fringe images that do not appear to be sinusoidal in shape. From this experiment, we can see that even if the exposure time is 250 µs and the capture speed is 2,000 Hz, the sinusoidal fringe patterns cannot be well captured. Therefore, high-quality 3-D shape measurement cannot be performed from them.
On contrast, we used exactly the same system settings to capture the fringe patterns with defocusing technique: 2,000 Hz sampling rate with 250 µs exposure time. Figure 8 shows some typical fringe images. As can be seen from this experiment, when the exposure time is short, the fringe patterns are still sinusoidal even though the intensity varies from frame to frame. The intensity variation was caused by the following three factors: (1) the projector projects red, green, and blue light in different timing; (2) red, green, and blue color may not be balanced because they came from different LED; and (3) the camera has different sensitivity to different light of color.
To further compare with the traditional sinusoidal fringe projection technique and the proposed technique, we used two different exposure time, 1/60 sec, and 1/4,000 sec. Figure 9 shows four images for the sinusoidal and the binary methods with these exposure time. The associated four videos shows show that if the camera is precisely synchronized to the projector and the exposure time is one projection cycle, the both methods can result in high-quality fringe patterns without large problems. On the contrast, if the exposure time is much shorter than the channel projection time, the captured fringe images generated by the binary method only vary intensity while keep its sinusoidal structure, whilst the capture fringe images generated by the conventional method vary both intensity and sinusoidal structure from time to time. It should be noted that in this experiment, we do not correct the nonlinear gamma of the projector, even the exposure time is right, the fringe pattern does not appear ideally sinusoidal. On the contrary, the binary one is not affected by the nonlinear gamma because only two intensity values are used. This is another advantage of the new method.
To test how fast the system can reach, we set the camera capture speed to be 4,000 Hz, exposure time to be 80 µs, and image resolution to be 480 × 480. Due to the projector output light intensity limitation, 80 µs the shortest exposure time we can use for our system to capture bright enough fringe patterns. A rotating fan blade was measured to verify the performance of the system. For this experiment, the fan is rotating at 1,793 rotations per minutes (rpm). Figure 10 shows the experimental result. Figure 10(a) shows the photograph of the fan blade. Figure 10(b) shows the fringe pattern. It clearly shows the high-quality fringes. Fourier method is then applied to find the frequency spectrum of the fringe pattern, a band-pass filter is used to get one carrier frequency component, and the phase can be extracted. Figure 10(c) shows the wrapped phase map. From the fringe data, the DC component (I′(x,y)) can also be extracted to generate the mask [Fig. 10(d)]. After removing the background, the phase can be unwrapped, as shown in Fig. 10(e). Both the wrapped phase map and the unwrapped phase map show that the motion is well captured.
Using a very short exposure time is very essential in order to capture fast motion, such as the rotating fan blade as shown in the previous example. To demonstrate this, more experiments were performed, where the camera captures the image at 200 Hz with varying exposure time. Figure 11 shows some of the fringe images and the associated wrapped phase map when the exposure time was chosen as 80, 160, 320, 640, 2,778µs, respectively. Again, the image resolution is 480 × 480 for these experiments, and the fan are rotating at a constant speed of 1,793 rpm during data capture. It can be seen from this series of results that when the exposure time is long enough, the motion blur causes too much problem, the fringe pattern cannot be correctly captured, and thus the 3-D imaging cannot be performed. For example, if an exposure time of 2,778 µs, the shortest exposure time possible for a conventional system, is used, the phase map is blended together for most part, and the measurement cannot be correctly performed. This experiment clearly shows that an off-the-shelf DLP projector cannot be used to capture very fast motion when a conventional fringe generation technique is utilized. On the contrast, this new technique allows the use of such a projector for extreme fast motion capture.
It should be noted that the measurement accuracy of this system is not high at current stage because we have not found a way to calibrate the defocused projector yet. In this research, we followed a standard simple calibration approach introduced in Ref. . This calibration technique is a linear approximation. This technique is essentially to measure a flat reference plane, find the phase difference point by point between the measured object phase the and the reference phase, and approximate the depth (z) by scaling the phase. The scaling factor is determined by measuring a known step height object. Because this is an approximation, the accuracy is not very high . We have not been able to implement a high-accuracy structured light system calibration technique, such as the one introduced in Ref. . This is because the existing techniques require the projector be in focus, which is not the case for our system.We are exploring a new method to accurately calibrate a defocused projector, and if successful, it will significantly improve the measurement accuracy.” Even with such a simple calibration technique, we found that for a measurement area of 2″ × 2″, the measurement error is approximately 0.19 mm rms.
By properly defocusing binary structured patterns to be sinusoidal, the DLP projector can essentially be converted into a digital solid-state fringe generation system. Because of its digital fringe generation nature, there are some advantageous features associated with it
- Superfast: Our experiment has used 80 µs exposure time for data capture, this means that the frame rate can go up to 12,500 Hz 3-D shape measurement rate with such an inexpensive off-the-shelf projector. An brighter projector or better camera should be able to reach much higher frame rate 3-D shape measurement by using the same technique.
- Flexible: Because the fringe patterns are generated digitally, it is easier than a mechanical grating to change the fringe patterns, e.g., fringe pitch.
- Adaptable: This system can be easily converted to a phase-shifting based 3-D shape measurement system because the phase shift can be easily generated by spatially moving the binary structured patterns. In fact, we have developed a superfast 3-D shape measurement system based a similar fringe generation approach employing a faster binary structured pattern switching system (DLP Discovery D4000) . We have successfully realized 3-D shape measurement speed of 667 fps using a three-step phase-shifting algorithm.
- Compact: The whole system including the illuminator are packaged into the DLP projector. The DLP projector, especially the LED-based projector becomes smaller and smaller, thus the 3-D shape measurement system can be miniaturized by taking advantage of the new hardware technology.
- Inexpensive: The DLP projector becomes cheaper and cheaper, there are some with a price below $200 (e.g., Optoma PK100 Pico Projector).
However, because the projector is defocused, the depth range is relatively smaller comparing with a traditional sinusoidal fringe generation technique. Another possible shortcoming is that it is theoretically not possible to generate ideal sinusoidal fringe pattern from this manner, therefore, some phase error compensation methods need to be developed to reduce the associated measurement errors.
This paper has presented a technique that achieves unprecedentedly 3-D shape measurement speed with an off-the-shelf DLP projector. It eliminates the speed bottleneck of a conventional sinusoidal fringe generation technique. Because only binary structure patterns are used, with each micromirror always being one stage (ON or OFF), the exposure time can be shorter than projection time. By this means, the system can measure faster motion with high quality. Experiments have been presented to demonstrate we could achieve 3-D shape measurement speed at 4000 Hz rate with an exposure time of 80 µs. The speed and exposure time limits are determined by the projector output light intensity and the camera sensitivity. Even with such a projector, the 3-D shape measurement speed can be as high as 12,500 Hz if the image resolution is reduced. This proposed methodology has the potential to replace a conventional mechanical grating method for 3-D shape measurement while maintains the merits of a digital fringe generation technique.
With an off-the-shelf inexpensive DLP projector, this proposed technology reached an unprecedentedly high speed. Of course, this technology is not trouble free. Comparing with the conventional digital fringe projection technique, there are two major limitations: (1) the current measurement accuracy is lower because the approximation calibration method used in this technique is inherently lower than those absolute calibration method; and (2) the measurement range is smaller. This is because ideal sinusoidal fringe patterns only happen with a range of the distance. In the future, we will work on developing methodologies to compensate for the residual phase error that are caused by the nonsinusoidality of the fringe patterns, and will find means to extend the measurement range. We will also explore hardware and software means to increase the measurement depth range.
References and links
1. S. Gorthi and P. Rastogi, “Fringe projection techniques: Whither we are?” Opt. Laser. Eng. 48, 133–140 (2010). [CrossRef]
3. M. Takeda, “Measurements of extreme physical phenomena by Fourier fringe analysis,” in AIP Conference Proc., vol. 1236, pp. 445–448 (2010).
4. P. S. Huang, C. Zhang, and F.-P. Chiang, “High-speed 3-D shape measurement based on digital fringe projection,” Opt. Eng. 42, 163–168 (2003). [CrossRef]
5. Z. Wang, H. Du, S. Park, and H. Xie, “Three-dimensional shape measurement with a fast and accurate approach,” Appl. Opt. 48, 1052–1061 (2009). [CrossRef]
6. S. Zhang and P. S. Huang, “High-resolution, real-time three-dimensional shape measurement,” Opt. Eng. 45, 123601 (2006). [CrossRef]
7. S. Zhang, “Recent progresses on real-time 3-D shape measurement using digital fringe projection techniques,” Opt. Laser Eng. 40, 149–158 (2010). [CrossRef]
8. L. J. Hornbeck, “Digital light processing for high-brightness, high-resolution applications,” in Proc. SPIE, vol. 3013, pp. 27–40 (1997).
9. J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, “state of the art in structured light patterns for surface profilometry,” Pattern Recogn. 43, 2666–2680 (2010). [CrossRef]
10. S. Rusinkiewicz, O. Hall-Holt, and M. Levoy, “Real-time 3D model acquisition,” ACM Trans. Graph. 21, 438–446 (2002). [CrossRef]
11. O. Hall-Holt and S. Rusinkiewicz, “Stripe boundary codes for real-time structured-light range scanning of moving objects,” in The 8th IEEE International Conference on Computer Vision, pp. II: 359–366 (2001).
12. J. Davis, R. Ramamoorthi, and S. Rusinkiewicz, “Spacetime stereo: A unifying framework for depth from triangulation,” IEEE Trans. Pattern Anal. Mach. Intell. 27, 1–7 (2005). [CrossRef]
13. S. G. Narasimhan, S. J. Koppal, and S. Yamazaki, “Temporal Dithering of Illumination for Fast Active Vision,” Lect. Notes Comput. Sci. 5305, 830–844 (2008). [CrossRef]
14. S. Zhang, Advances in measurement systems, chap. 2, pp. 29–50 (In-Tech, 2010).
16. X. Y. Su, W. S. Zhou, G. Von Bally, and D. Vukicevic, “Automated phase-measuring profilometry using defocused projection of a Ronchi grating,” Opt. Commun. 94, 561–573 (1992). [CrossRef]
17. X. Su and Q. Zhang, “Dynamic 3-D shape measurement method: A review,” Opt. Laser Eng. 48, 191–204 (2010). [CrossRef]
18. D. C. Ghiglia and M. D. Pritt, Two-dimensional phase unwrapping: theory, algorithms, and software (JohnWiley and Sons, Inc., 1998).
19. S. Zhang and P. S. Huang, “Novel method for structured light system calibration,” Opt. Eng. 45, 083601 (2006). [CrossRef]
21. S. Zhang, X. Li, and S.-T. Yau, “Multilevel quality-guided phase unwrapping algorithm for real-time three-dimensional shape reconstruction,” Appl. Opt. 46, 50–57 (2007). [CrossRef]