Abstract

We have developed a field-worthy, high-definition, real-time depth-mapping television camera called the HDTV Axi-Vision Camera. The camera can simultaneously capture both an ordinary HDTV color image and a depth image of objects on more than 1280×720 pixels at a frame rate of 29.97 Hz, or on 853×480 pixels at a frame rate of 59.94 Hz. The number of detectable pixels per unit time was increased by about 5 times that of the prototype camera by improving the sensitivity and resolution of the depth-mapping camera. Short video clips demonstrate how depth information from the camera can be used to create a virtual image in actual television program production.

© 2004 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |

  1. S. Kimura, H. Kano, T. Kanade, A. Yoshida, E. Kawamura, and K. Oda, �??CMU video-rate stereo machine,�?? in Proceedings of 1995 Mobile Mapping Symposium (American Society for Photogrammetry and Remote Sensing, Columbus, Ohio, 1995), pp. 9-18.
  2. K. Sato and S. Inokuchi, �??Range-imaging system utilizing nematic liquid crystal mask,�?? in 1st International Conference on Computer Vision ICCV (Institute of Electrical and Electronics Engineers, London, 1987), pp. 657-661.
  3. Y. Oike, M. Ikeda, and K. Asada, �??Design and implementation of real-time 3-D image sensor with 640 �? 480 pixel resolution,�?? IEEE J. Solid-State Circuits, 39, 622-628 (2004).
    [CrossRef]
  4. R. A. Jarvis, �??A laser time-of-flight range scanner for robotic vision,�?? IEEE Trans. on Pattern Analysis and Machine Intelligence, PAMI-5, 505-512 (1983).
    [CrossRef]
  5. R. Lange, P. Seitz, A. Biber, and S. Lauxtermann, �??Demodulation pixels in CCD and CMOS technologies for time-of-flight ranging,�?? in Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications, Morley M. Blouke, Nitin Sampat, George M. Williams and Thomas Yeh, eds. Proc. SPIE 3965, 177-188 (2000).
  6. . Kawakita, K. Iizuka, H. Kikuchi, H. Fujikake, J. Yonai, and T. Aida, �??A 3D camera system using a high-speed shutter and intensity modulated illuminator,�?? (in Japanese) Institute of Image Information and Television Engineers ITE Tech. Rep. 22, 19-24 (1998).
  7. M. Kawakita, K. Iizuka, T. Aida, H. Kikuchi, H. Fujikake, J. Yonai, and K. Takizawa, �??Axi-Vision Camera (real-time depth-mapping camera),�?? Appl. Opt. 39, 3931-3939 (2000).
    [CrossRef]
  8. M. Kawakita, K. Iizuka, T. Aida, H. Kikuchi, H. Fujikake, J. Yonai, and K. Takizawa, �??Axi-Vision Camera: a three-dimension camera,�?? in Three-Dimensional Image Capture and Applications III, Brian D. Corner and Joseph H. Nurre, eds., Proc. SPIE 3958, 61-70 (2000).
  9. R. J. Hertel, �??Signal and noise properties of proximity focused image tubes,�?? in Ultrahigh Speed and High Speed Photography, Photonics, and Videography '89: Seventh in a Series, Gary L. Stradling, ed., Proc. SPIE 1155, 332- 343 (1989).
  10. M. Kawakita, K. Iizuka, Y. Iino, H. Kikuchi, H. Fujikake, and T. Aida, �??Real-time depth-mapping threedimension TV camera (Axi-Vision Camera),�?? (in Japanese) IEICE Trans. on Information & Systems J87-D-II, No.6, (2004). (to be published).
  11. Hamamatsu Photonics K.K., <a href="http://www.hpk.co.jp/">http://www.hpk.co.jp/</a>.
  12. S. Shimoda, M. Hayashi, and Y. Kanatsugu, �??New chroma-key imaging technique with hi-vision background,�?? IEEE Trans. on Broadcasting 35, 357-361 (1989).
    [CrossRef]
  13. Y. Yamanouchi, H. Mitsumine, and S. Inoue, �??Image-based virtual studio using ultra high-definition omnidirectional images,�?? (in Japanese) J. Inst. Image Information and Television Eng. ITE 55, 159-166 (2001).
    [CrossRef]

Appl. Opt.

IEEE J. Solid-State Circuits

Y. Oike, M. Ikeda, and K. Asada, �??Design and implementation of real-time 3-D image sensor with 640 �? 480 pixel resolution,�?? IEEE J. Solid-State Circuits, 39, 622-628 (2004).
[CrossRef]

IEEE Trans. on Broadcasting

S. Shimoda, M. Hayashi, and Y. Kanatsugu, �??New chroma-key imaging technique with hi-vision background,�?? IEEE Trans. on Broadcasting 35, 357-361 (1989).
[CrossRef]

IEEE Trans. on Pattern Analysis Mach. I

R. A. Jarvis, �??A laser time-of-flight range scanner for robotic vision,�?? IEEE Trans. on Pattern Analysis and Machine Intelligence, PAMI-5, 505-512 (1983).
[CrossRef]

IEICE Trans. on Information & Systems

M. Kawakita, K. Iizuka, Y. Iino, H. Kikuchi, H. Fujikake, and T. Aida, �??Real-time depth-mapping threedimension TV camera (Axi-Vision Camera),�?? (in Japanese) IEICE Trans. on Information & Systems J87-D-II, No.6, (2004). (to be published).

ITE Tech. Rep.

. Kawakita, K. Iizuka, H. Kikuchi, H. Fujikake, J. Yonai, and T. Aida, �??A 3D camera system using a high-speed shutter and intensity modulated illuminator,�?? (in Japanese) Institute of Image Information and Television Engineers ITE Tech. Rep. 22, 19-24 (1998).

J. Inst. Image Information Television En

Y. Yamanouchi, H. Mitsumine, and S. Inoue, �??Image-based virtual studio using ultra high-definition omnidirectional images,�?? (in Japanese) J. Inst. Image Information and Television Eng. ITE 55, 159-166 (2001).
[CrossRef]

Proc. CLEO/Europe

S. Kimura, H. Kano, T. Kanade, A. Yoshida, E. Kawamura, and K. Oda, �??CMU video-rate stereo machine,�?? in Proceedings of 1995 Mobile Mapping Symposium (American Society for Photogrammetry and Remote Sensing, Columbus, Ohio, 1995), pp. 9-18.

Proc. SPIE

R. Lange, P. Seitz, A. Biber, and S. Lauxtermann, �??Demodulation pixels in CCD and CMOS technologies for time-of-flight ranging,�?? in Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications, Morley M. Blouke, Nitin Sampat, George M. Williams and Thomas Yeh, eds. Proc. SPIE 3965, 177-188 (2000).

M. Kawakita, K. Iizuka, T. Aida, H. Kikuchi, H. Fujikake, J. Yonai, and K. Takizawa, �??Axi-Vision Camera: a three-dimension camera,�?? in Three-Dimensional Image Capture and Applications III, Brian D. Corner and Joseph H. Nurre, eds., Proc. SPIE 3958, 61-70 (2000).

R. J. Hertel, �??Signal and noise properties of proximity focused image tubes,�?? in Ultrahigh Speed and High Speed Photography, Photonics, and Videography '89: Seventh in a Series, Gary L. Stradling, ed., Proc. SPIE 1155, 332- 343 (1989).

Other

Hamamatsu Photonics K.K., <a href="http://www.hpk.co.jp/">http://www.hpk.co.jp/</a>.

K. Sato and S. Inokuchi, �??Range-imaging system utilizing nematic liquid crystal mask,�?? in 1st International Conference on Computer Vision ICCV (Institute of Electrical and Electronics Engineers, London, 1987), pp. 657-661.

Supplementary Material (3)

» Media 1: MOV (2496 KB)     
» Media 2: MOV (2310 KB)     
» Media 3: MOV (2428 KB)     

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (12)

Fig. 1.
Fig. 1.

(2.4 MB) Video clip from the Japan Broadcasting Station (NHK) broadcast produced using the HDTV Axi-Vision Camera. The high-definition TV program is the “50th Anniversary: Today is the birthday of TV. Grand finale.” broadcast live from NHK. A moving computer graphic image was amalgamated in real time with an image of a singer using the depth information.

Fig. 2.
Fig. 2.

Principle of acquiring depth information by using intensity-modulated illumination and an ultra-fast camera shutter using an image intensifier.

Fig. 3.
Fig. 3.

Configuration of the HDTV Axi-Vision Camera.

Fig. 4.
Fig. 4.

Photograph of the HDTV Axi-Vision Camera.

Fig. 5.
Fig. 5.

Quantum efficiency of the photocathode of the image intensifier.

Fig. 6.
Fig. 6.

Transmittance of the dichroic prism and the optical filter.

Fig. 7.
Fig. 7.

LED array units: (a) geometry, (b) spatial distribution of optical power.

Fig. 8.
Fig. 8.

Output image signal as a function of distance between the object and camera.

Fig. 9.
Fig. 9.

Depth resolution as a function of object distance.

Fig. 10.
Fig. 10.

Relationship between object reflectivity and depth resolution.

Fig. 11.
Fig. 11.

(2.5 MB) Video clip of depth-keying examples: (a) color image, (b) depth image, (c) objects in the furthest range only, (d) objects in the middle range only, (e) objects in the nearest range only.

Fig. 12.
Fig. 12.

(22.4 MB) Video clip of a virtual studio synthesized by combining live images with prerecorded scenes.

Tables (1)

Tables Icon

Table 1. Specifications of the HDTV Axi-Vision Camera

Equations (10)

Equations on this page are rendered with MathJax. Learn more.

I + ( t s , d ) = σ ( 4 π d 2 ) 2 s ( t s 2 d ν ) .
I ( t s , d ) = σ ( 4 π d 2 ) 2 s [ T 2 ( t s 2 d ν ) ] ,
d = 1 2 ν [ t s T 2 ( R 1 + R ) ] ,
R = I + I .
d = λ 8 ( 1 R 1 + R ) ,
n pe ¯ = η E A p τ ε m 2 ,
( S N ) pe = n pe ¯ σ pe = n pe ¯ = η E A p τ ε m 2 .
( S N ) phosphor = ( S N ) pe 1 N f = η E A p τ ε m 2 N f .
E ρ T L I 4 F N 2 S I d 2 ,
( S N ) phosphor η T L I A p τ d .

Metrics