Abstract

Numerous depth extraction techniques have been proposed in the past. However, the utility of these techniques is limited as they typically require multiple imaging units, bulky platforms for computation, cannot achieve high speed and are computationally expensive. To counter the above challenges, a sensor with Offset Pixel Apertures (OPA) has been recently proposed. However, a working system for depth extraction with the OPA sensor has not been discussed. In this paper, we propose the first such system for depth extraction using the OPA sensor. We also propose a dedicated hardware implementation for the proposed system, named as the Depth Map Processor (DMP). The DMP can provide depth at 30 frames per second at 1920 × 1080 resolution with 31 disparity levels. Furthermore, the proposed DMP has low power consumption as for the aforementioned speed and resolution it only requires 290.76 mW. The proposed system makes it an ideal choice for depth extraction systems in constrained environments.

© 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

Full Article  |  PDF Article
OSA Recommended Articles
Depth map generation using a single image sensor with phase masks

Jinbeum Jang, Sangwoo Park, Jieun Jo, and Joonki Paik
Opt. Express 24(12) 12868-12878 (2016)

Intensity guided cost metric for fast stereo matching under radiometric variations

Asim Khan, Muhammad Umar Karim Khan, and Chong-Min Kyung
Opt. Express 26(4) 4096-4111 (2018)

Aggregation functions to combine RGB color channels in stereo matching

Mikel Galar, Aranzazu Jurio, Carlos Lopez-Molina, Daniel Paternain, Jose Sanz, and Humberto Bustince
Opt. Express 21(1) 1247-1257 (2013)

References

  • View by:
  • |
  • |
  • |

  1. D. Lanman and G. Taubin, “Build your own 3D scanner: 3D photography for beginners,” in ACM SIGGRAPH 2009 Courses (ACM, 2009), p. 8.
  2. S. B. Gokturk, H. Yalcin, and C. Bamji, “A time-of-flight depth sensor-system description, issues and solutions,” in Workshop on Computer Vision and Pattern Recogntion (IEEE, 2004). pp. 35.
  3. D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Computer Vis. 47(1), 7–42 (2002).
    [Crossref]
  4. M. Martinello, A. Wajs, S. Quan, H. Lee, C. Lim, T. Woo, W. Lee, S.S. Kim, and D. Lee, “Dual aperture photography: image and depth from a mobile camera,” in 2015 IEEE International Conference on Computational Photography (ICCP) (IEEE, 2015). pp. 1–10.
  5. W. Yun, Y. G. Kim, Y. Lee, J. Lim, W. Choi, M. U. K. Khan, A. Khan, S. Homidov, P. Kareem, H. S. Park, and C. M. Kyung, “Offset aperture based hardware architecture for real-time depth extraction,” in 2017 IEEE International Conference on Image Processing (ICIP) (IEEE, 2017), pp. 4392–4396.
    [Crossref]
  6. B. S. Choi, M. Bae, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor for extracting depth information using offset pixel aperture technique,” Proc. SPIE 10376, 103760Y (2017)
  7. B. S. Choi, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “3D CMOS image sensor based on white pixel with off-center rectangular apertures,” in 2018 IS&T International Symposium on Electronic Imaging (2018).
  8. Y. Egawa, N. Tanaka, N. Kawai, H. Seki, A. Nakao, H. Honda, Y. Iida, and M. Monoi, “A white-RGB CFA-patterned CMOS image sensor with wide dynamic range,” in IEEE International Solid-State Circuits Conference (IEEE, 2008), pp. 52–595.
  9. J. Rabaey, Low Power Design Essentials (Springer, 2009).
    [Crossref]
  10. A. Khan, M. U. K. Khan, and C. M. Kyung, “Intensity guided cost metric for fast stereo matching under radiometric variations,” Opt. Express 26(4), 4096–4111 (2018).
    [Crossref] [PubMed]
  11. H. Hirschmuller, “Accurate and efficient stereo processing by semiglobal matching and mutual information,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2005). pp. 807–814.
  12. S. Birchfield and C. Tomasi, “Depth discontinuities by pixel-to-pixel stereo,” Int. J. Computer Vision,  35(2), 269–293 (1999).
    [Crossref]
  13. S. J. Ko and Y. H. Lee, “Center weighted median filters and their applications to image enhancement,” IEEE Trans. Circuits Systems 38(9), 984–993 (1991).
    [Crossref]
  14. C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Sixth International Conference on Computer Vision (IEEE, 1998). pp. 839–846.
    [Crossref]
  15. K. He, J. Sun, and X. Tang, “Guided image filtering,” in European Conference on Computer Vision (Springer, 2010), pp. 1–14.
  16. B. S. Choi, S. H. Kim, J. Lee, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor using pixel aperture technique for single-chip 2D and 3D imaging,” in SENSORS (IEEE, 2017), pp. 1–3.
  17. V. N. Dvorchenko, “Bounds on (deterministic) correlation functions with applications to registration,” IEEE Trans. Pattern Anal. Machine Intell,  5(2), 206–213 (1983).
    [Crossref]

2018 (1)

2017 (1)

B. S. Choi, M. Bae, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor for extracting depth information using offset pixel aperture technique,” Proc. SPIE 10376, 103760Y (2017)

2002 (1)

D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Computer Vis. 47(1), 7–42 (2002).
[Crossref]

1999 (1)

S. Birchfield and C. Tomasi, “Depth discontinuities by pixel-to-pixel stereo,” Int. J. Computer Vision,  35(2), 269–293 (1999).
[Crossref]

1991 (1)

S. J. Ko and Y. H. Lee, “Center weighted median filters and their applications to image enhancement,” IEEE Trans. Circuits Systems 38(9), 984–993 (1991).
[Crossref]

1983 (1)

V. N. Dvorchenko, “Bounds on (deterministic) correlation functions with applications to registration,” IEEE Trans. Pattern Anal. Machine Intell,  5(2), 206–213 (1983).
[Crossref]

Bae, M.

B. S. Choi, M. Bae, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor for extracting depth information using offset pixel aperture technique,” Proc. SPIE 10376, 103760Y (2017)

Bamji, C.

S. B. Gokturk, H. Yalcin, and C. Bamji, “A time-of-flight depth sensor-system description, issues and solutions,” in Workshop on Computer Vision and Pattern Recogntion (IEEE, 2004). pp. 35.

Birchfield, S.

S. Birchfield and C. Tomasi, “Depth discontinuities by pixel-to-pixel stereo,” Int. J. Computer Vision,  35(2), 269–293 (1999).
[Crossref]

Chang, S.

B. S. Choi, M. Bae, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor for extracting depth information using offset pixel aperture technique,” Proc. SPIE 10376, 103760Y (2017)

B. S. Choi, S. H. Kim, J. Lee, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor using pixel aperture technique for single-chip 2D and 3D imaging,” in SENSORS (IEEE, 2017), pp. 1–3.

B. S. Choi, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “3D CMOS image sensor based on white pixel with off-center rectangular apertures,” in 2018 IS&T International Symposium on Electronic Imaging (2018).

Choi, B. S.

B. S. Choi, M. Bae, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor for extracting depth information using offset pixel aperture technique,” Proc. SPIE 10376, 103760Y (2017)

B. S. Choi, S. H. Kim, J. Lee, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor using pixel aperture technique for single-chip 2D and 3D imaging,” in SENSORS (IEEE, 2017), pp. 1–3.

B. S. Choi, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “3D CMOS image sensor based on white pixel with off-center rectangular apertures,” in 2018 IS&T International Symposium on Electronic Imaging (2018).

Choi, W.

W. Yun, Y. G. Kim, Y. Lee, J. Lim, W. Choi, M. U. K. Khan, A. Khan, S. Homidov, P. Kareem, H. S. Park, and C. M. Kyung, “Offset aperture based hardware architecture for real-time depth extraction,” in 2017 IEEE International Conference on Image Processing (ICIP) (IEEE, 2017), pp. 4392–4396.
[Crossref]

Dvorchenko, V. N.

V. N. Dvorchenko, “Bounds on (deterministic) correlation functions with applications to registration,” IEEE Trans. Pattern Anal. Machine Intell,  5(2), 206–213 (1983).
[Crossref]

Egawa, Y.

Y. Egawa, N. Tanaka, N. Kawai, H. Seki, A. Nakao, H. Honda, Y. Iida, and M. Monoi, “A white-RGB CFA-patterned CMOS image sensor with wide dynamic range,” in IEEE International Solid-State Circuits Conference (IEEE, 2008), pp. 52–595.

Gokturk, S. B.

S. B. Gokturk, H. Yalcin, and C. Bamji, “A time-of-flight depth sensor-system description, issues and solutions,” in Workshop on Computer Vision and Pattern Recogntion (IEEE, 2004). pp. 35.

He, K.

K. He, J. Sun, and X. Tang, “Guided image filtering,” in European Conference on Computer Vision (Springer, 2010), pp. 1–14.

Hirschmuller, H.

H. Hirschmuller, “Accurate and efficient stereo processing by semiglobal matching and mutual information,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2005). pp. 807–814.

Homidov, S.

W. Yun, Y. G. Kim, Y. Lee, J. Lim, W. Choi, M. U. K. Khan, A. Khan, S. Homidov, P. Kareem, H. S. Park, and C. M. Kyung, “Offset aperture based hardware architecture for real-time depth extraction,” in 2017 IEEE International Conference on Image Processing (ICIP) (IEEE, 2017), pp. 4392–4396.
[Crossref]

Honda, H.

Y. Egawa, N. Tanaka, N. Kawai, H. Seki, A. Nakao, H. Honda, Y. Iida, and M. Monoi, “A white-RGB CFA-patterned CMOS image sensor with wide dynamic range,” in IEEE International Solid-State Circuits Conference (IEEE, 2008), pp. 52–595.

Iida, Y.

Y. Egawa, N. Tanaka, N. Kawai, H. Seki, A. Nakao, H. Honda, Y. Iida, and M. Monoi, “A white-RGB CFA-patterned CMOS image sensor with wide dynamic range,” in IEEE International Solid-State Circuits Conference (IEEE, 2008), pp. 52–595.

Kareem, P.

W. Yun, Y. G. Kim, Y. Lee, J. Lim, W. Choi, M. U. K. Khan, A. Khan, S. Homidov, P. Kareem, H. S. Park, and C. M. Kyung, “Offset aperture based hardware architecture for real-time depth extraction,” in 2017 IEEE International Conference on Image Processing (ICIP) (IEEE, 2017), pp. 4392–4396.
[Crossref]

Kawai, N.

Y. Egawa, N. Tanaka, N. Kawai, H. Seki, A. Nakao, H. Honda, Y. Iida, and M. Monoi, “A white-RGB CFA-patterned CMOS image sensor with wide dynamic range,” in IEEE International Solid-State Circuits Conference (IEEE, 2008), pp. 52–595.

Khan, A.

A. Khan, M. U. K. Khan, and C. M. Kyung, “Intensity guided cost metric for fast stereo matching under radiometric variations,” Opt. Express 26(4), 4096–4111 (2018).
[Crossref] [PubMed]

W. Yun, Y. G. Kim, Y. Lee, J. Lim, W. Choi, M. U. K. Khan, A. Khan, S. Homidov, P. Kareem, H. S. Park, and C. M. Kyung, “Offset aperture based hardware architecture for real-time depth extraction,” in 2017 IEEE International Conference on Image Processing (ICIP) (IEEE, 2017), pp. 4392–4396.
[Crossref]

Khan, M. U. K.

A. Khan, M. U. K. Khan, and C. M. Kyung, “Intensity guided cost metric for fast stereo matching under radiometric variations,” Opt. Express 26(4), 4096–4111 (2018).
[Crossref] [PubMed]

W. Yun, Y. G. Kim, Y. Lee, J. Lim, W. Choi, M. U. K. Khan, A. Khan, S. Homidov, P. Kareem, H. S. Park, and C. M. Kyung, “Offset aperture based hardware architecture for real-time depth extraction,” in 2017 IEEE International Conference on Image Processing (ICIP) (IEEE, 2017), pp. 4392–4396.
[Crossref]

Kim, S. H.

B. S. Choi, M. Bae, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor for extracting depth information using offset pixel aperture technique,” Proc. SPIE 10376, 103760Y (2017)

B. S. Choi, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “3D CMOS image sensor based on white pixel with off-center rectangular apertures,” in 2018 IS&T International Symposium on Electronic Imaging (2018).

B. S. Choi, S. H. Kim, J. Lee, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor using pixel aperture technique for single-chip 2D and 3D imaging,” in SENSORS (IEEE, 2017), pp. 1–3.

Kim, S.S.

M. Martinello, A. Wajs, S. Quan, H. Lee, C. Lim, T. Woo, W. Lee, S.S. Kim, and D. Lee, “Dual aperture photography: image and depth from a mobile camera,” in 2015 IEEE International Conference on Computational Photography (ICCP) (IEEE, 2015). pp. 1–10.

Kim, Y. G.

W. Yun, Y. G. Kim, Y. Lee, J. Lim, W. Choi, M. U. K. Khan, A. Khan, S. Homidov, P. Kareem, H. S. Park, and C. M. Kyung, “Offset aperture based hardware architecture for real-time depth extraction,” in 2017 IEEE International Conference on Image Processing (ICIP) (IEEE, 2017), pp. 4392–4396.
[Crossref]

Ko, S. J.

S. J. Ko and Y. H. Lee, “Center weighted median filters and their applications to image enhancement,” IEEE Trans. Circuits Systems 38(9), 984–993 (1991).
[Crossref]

Kyung, C. M.

A. Khan, M. U. K. Khan, and C. M. Kyung, “Intensity guided cost metric for fast stereo matching under radiometric variations,” Opt. Express 26(4), 4096–4111 (2018).
[Crossref] [PubMed]

W. Yun, Y. G. Kim, Y. Lee, J. Lim, W. Choi, M. U. K. Khan, A. Khan, S. Homidov, P. Kareem, H. S. Park, and C. M. Kyung, “Offset aperture based hardware architecture for real-time depth extraction,” in 2017 IEEE International Conference on Image Processing (ICIP) (IEEE, 2017), pp. 4392–4396.
[Crossref]

Lanman, D.

D. Lanman and G. Taubin, “Build your own 3D scanner: 3D photography for beginners,” in ACM SIGGRAPH 2009 Courses (ACM, 2009), p. 8.

Lee, D.

M. Martinello, A. Wajs, S. Quan, H. Lee, C. Lim, T. Woo, W. Lee, S.S. Kim, and D. Lee, “Dual aperture photography: image and depth from a mobile camera,” in 2015 IEEE International Conference on Computational Photography (ICCP) (IEEE, 2015). pp. 1–10.

Lee, H.

M. Martinello, A. Wajs, S. Quan, H. Lee, C. Lim, T. Woo, W. Lee, S.S. Kim, and D. Lee, “Dual aperture photography: image and depth from a mobile camera,” in 2015 IEEE International Conference on Computational Photography (ICCP) (IEEE, 2015). pp. 1–10.

Lee, J.

B. S. Choi, M. Bae, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor for extracting depth information using offset pixel aperture technique,” Proc. SPIE 10376, 103760Y (2017)

B. S. Choi, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “3D CMOS image sensor based on white pixel with off-center rectangular apertures,” in 2018 IS&T International Symposium on Electronic Imaging (2018).

B. S. Choi, S. H. Kim, J. Lee, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor using pixel aperture technique for single-chip 2D and 3D imaging,” in SENSORS (IEEE, 2017), pp. 1–3.

Lee, S. J.

B. S. Choi, M. Bae, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor for extracting depth information using offset pixel aperture technique,” Proc. SPIE 10376, 103760Y (2017)

B. S. Choi, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “3D CMOS image sensor based on white pixel with off-center rectangular apertures,” in 2018 IS&T International Symposium on Electronic Imaging (2018).

B. S. Choi, S. H. Kim, J. Lee, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor using pixel aperture technique for single-chip 2D and 3D imaging,” in SENSORS (IEEE, 2017), pp. 1–3.

Lee, W.

M. Martinello, A. Wajs, S. Quan, H. Lee, C. Lim, T. Woo, W. Lee, S.S. Kim, and D. Lee, “Dual aperture photography: image and depth from a mobile camera,” in 2015 IEEE International Conference on Computational Photography (ICCP) (IEEE, 2015). pp. 1–10.

Lee, Y.

W. Yun, Y. G. Kim, Y. Lee, J. Lim, W. Choi, M. U. K. Khan, A. Khan, S. Homidov, P. Kareem, H. S. Park, and C. M. Kyung, “Offset aperture based hardware architecture for real-time depth extraction,” in 2017 IEEE International Conference on Image Processing (ICIP) (IEEE, 2017), pp. 4392–4396.
[Crossref]

Lee, Y. H.

S. J. Ko and Y. H. Lee, “Center weighted median filters and their applications to image enhancement,” IEEE Trans. Circuits Systems 38(9), 984–993 (1991).
[Crossref]

Lim, C.

M. Martinello, A. Wajs, S. Quan, H. Lee, C. Lim, T. Woo, W. Lee, S.S. Kim, and D. Lee, “Dual aperture photography: image and depth from a mobile camera,” in 2015 IEEE International Conference on Computational Photography (ICCP) (IEEE, 2015). pp. 1–10.

Lim, J.

W. Yun, Y. G. Kim, Y. Lee, J. Lim, W. Choi, M. U. K. Khan, A. Khan, S. Homidov, P. Kareem, H. S. Park, and C. M. Kyung, “Offset aperture based hardware architecture for real-time depth extraction,” in 2017 IEEE International Conference on Image Processing (ICIP) (IEEE, 2017), pp. 4392–4396.
[Crossref]

Manduchi, R.

C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Sixth International Conference on Computer Vision (IEEE, 1998). pp. 839–846.
[Crossref]

Martinello, M.

M. Martinello, A. Wajs, S. Quan, H. Lee, C. Lim, T. Woo, W. Lee, S.S. Kim, and D. Lee, “Dual aperture photography: image and depth from a mobile camera,” in 2015 IEEE International Conference on Computational Photography (ICCP) (IEEE, 2015). pp. 1–10.

Monoi, M.

Y. Egawa, N. Tanaka, N. Kawai, H. Seki, A. Nakao, H. Honda, Y. Iida, and M. Monoi, “A white-RGB CFA-patterned CMOS image sensor with wide dynamic range,” in IEEE International Solid-State Circuits Conference (IEEE, 2008), pp. 52–595.

Nakao, A.

Y. Egawa, N. Tanaka, N. Kawai, H. Seki, A. Nakao, H. Honda, Y. Iida, and M. Monoi, “A white-RGB CFA-patterned CMOS image sensor with wide dynamic range,” in IEEE International Solid-State Circuits Conference (IEEE, 2008), pp. 52–595.

Oh, C. W.

B. S. Choi, M. Bae, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor for extracting depth information using offset pixel aperture technique,” Proc. SPIE 10376, 103760Y (2017)

B. S. Choi, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “3D CMOS image sensor based on white pixel with off-center rectangular apertures,” in 2018 IS&T International Symposium on Electronic Imaging (2018).

Park, H. S.

W. Yun, Y. G. Kim, Y. Lee, J. Lim, W. Choi, M. U. K. Khan, A. Khan, S. Homidov, P. Kareem, H. S. Park, and C. M. Kyung, “Offset aperture based hardware architecture for real-time depth extraction,” in 2017 IEEE International Conference on Image Processing (ICIP) (IEEE, 2017), pp. 4392–4396.
[Crossref]

Park, J.

B. S. Choi, M. Bae, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor for extracting depth information using offset pixel aperture technique,” Proc. SPIE 10376, 103760Y (2017)

B. S. Choi, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “3D CMOS image sensor based on white pixel with off-center rectangular apertures,” in 2018 IS&T International Symposium on Electronic Imaging (2018).

B. S. Choi, S. H. Kim, J. Lee, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor using pixel aperture technique for single-chip 2D and 3D imaging,” in SENSORS (IEEE, 2017), pp. 1–3.

Quan, S.

M. Martinello, A. Wajs, S. Quan, H. Lee, C. Lim, T. Woo, W. Lee, S.S. Kim, and D. Lee, “Dual aperture photography: image and depth from a mobile camera,” in 2015 IEEE International Conference on Computational Photography (ICCP) (IEEE, 2015). pp. 1–10.

Rabaey, J.

J. Rabaey, Low Power Design Essentials (Springer, 2009).
[Crossref]

Scharstein, D.

D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Computer Vis. 47(1), 7–42 (2002).
[Crossref]

Seki, H.

Y. Egawa, N. Tanaka, N. Kawai, H. Seki, A. Nakao, H. Honda, Y. Iida, and M. Monoi, “A white-RGB CFA-patterned CMOS image sensor with wide dynamic range,” in IEEE International Solid-State Circuits Conference (IEEE, 2008), pp. 52–595.

Shin, J. K.

B. S. Choi, M. Bae, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor for extracting depth information using offset pixel aperture technique,” Proc. SPIE 10376, 103760Y (2017)

B. S. Choi, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “3D CMOS image sensor based on white pixel with off-center rectangular apertures,” in 2018 IS&T International Symposium on Electronic Imaging (2018).

B. S. Choi, S. H. Kim, J. Lee, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor using pixel aperture technique for single-chip 2D and 3D imaging,” in SENSORS (IEEE, 2017), pp. 1–3.

Sun, J.

K. He, J. Sun, and X. Tang, “Guided image filtering,” in European Conference on Computer Vision (Springer, 2010), pp. 1–14.

Szeliski, R.

D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Computer Vis. 47(1), 7–42 (2002).
[Crossref]

Tanaka, N.

Y. Egawa, N. Tanaka, N. Kawai, H. Seki, A. Nakao, H. Honda, Y. Iida, and M. Monoi, “A white-RGB CFA-patterned CMOS image sensor with wide dynamic range,” in IEEE International Solid-State Circuits Conference (IEEE, 2008), pp. 52–595.

Tang, X.

K. He, J. Sun, and X. Tang, “Guided image filtering,” in European Conference on Computer Vision (Springer, 2010), pp. 1–14.

Taubin, G.

D. Lanman and G. Taubin, “Build your own 3D scanner: 3D photography for beginners,” in ACM SIGGRAPH 2009 Courses (ACM, 2009), p. 8.

Tomasi, C.

S. Birchfield and C. Tomasi, “Depth discontinuities by pixel-to-pixel stereo,” Int. J. Computer Vision,  35(2), 269–293 (1999).
[Crossref]

C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Sixth International Conference on Computer Vision (IEEE, 1998). pp. 839–846.
[Crossref]

Wajs, A.

M. Martinello, A. Wajs, S. Quan, H. Lee, C. Lim, T. Woo, W. Lee, S.S. Kim, and D. Lee, “Dual aperture photography: image and depth from a mobile camera,” in 2015 IEEE International Conference on Computational Photography (ICCP) (IEEE, 2015). pp. 1–10.

Woo, T.

M. Martinello, A. Wajs, S. Quan, H. Lee, C. Lim, T. Woo, W. Lee, S.S. Kim, and D. Lee, “Dual aperture photography: image and depth from a mobile camera,” in 2015 IEEE International Conference on Computational Photography (ICCP) (IEEE, 2015). pp. 1–10.

Yalcin, H.

S. B. Gokturk, H. Yalcin, and C. Bamji, “A time-of-flight depth sensor-system description, issues and solutions,” in Workshop on Computer Vision and Pattern Recogntion (IEEE, 2004). pp. 35.

Yun, W.

W. Yun, Y. G. Kim, Y. Lee, J. Lim, W. Choi, M. U. K. Khan, A. Khan, S. Homidov, P. Kareem, H. S. Park, and C. M. Kyung, “Offset aperture based hardware architecture for real-time depth extraction,” in 2017 IEEE International Conference on Image Processing (ICIP) (IEEE, 2017), pp. 4392–4396.
[Crossref]

IEEE Trans. Circuits Systems (1)

S. J. Ko and Y. H. Lee, “Center weighted median filters and their applications to image enhancement,” IEEE Trans. Circuits Systems 38(9), 984–993 (1991).
[Crossref]

IEEE Trans. Pattern Anal. Machine Intell (1)

V. N. Dvorchenko, “Bounds on (deterministic) correlation functions with applications to registration,” IEEE Trans. Pattern Anal. Machine Intell,  5(2), 206–213 (1983).
[Crossref]

Int. J. Computer Vis. (1)

D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Computer Vis. 47(1), 7–42 (2002).
[Crossref]

Int. J. Computer Vision (1)

S. Birchfield and C. Tomasi, “Depth discontinuities by pixel-to-pixel stereo,” Int. J. Computer Vision,  35(2), 269–293 (1999).
[Crossref]

Opt. Express (1)

Proc. SPIE (1)

B. S. Choi, M. Bae, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor for extracting depth information using offset pixel aperture technique,” Proc. SPIE 10376, 103760Y (2017)

Other (11)

B. S. Choi, S. H. Kim, J. Lee, C. W. Oh, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “3D CMOS image sensor based on white pixel with off-center rectangular apertures,” in 2018 IS&T International Symposium on Electronic Imaging (2018).

Y. Egawa, N. Tanaka, N. Kawai, H. Seki, A. Nakao, H. Honda, Y. Iida, and M. Monoi, “A white-RGB CFA-patterned CMOS image sensor with wide dynamic range,” in IEEE International Solid-State Circuits Conference (IEEE, 2008), pp. 52–595.

J. Rabaey, Low Power Design Essentials (Springer, 2009).
[Crossref]

M. Martinello, A. Wajs, S. Quan, H. Lee, C. Lim, T. Woo, W. Lee, S.S. Kim, and D. Lee, “Dual aperture photography: image and depth from a mobile camera,” in 2015 IEEE International Conference on Computational Photography (ICCP) (IEEE, 2015). pp. 1–10.

W. Yun, Y. G. Kim, Y. Lee, J. Lim, W. Choi, M. U. K. Khan, A. Khan, S. Homidov, P. Kareem, H. S. Park, and C. M. Kyung, “Offset aperture based hardware architecture for real-time depth extraction,” in 2017 IEEE International Conference on Image Processing (ICIP) (IEEE, 2017), pp. 4392–4396.
[Crossref]

D. Lanman and G. Taubin, “Build your own 3D scanner: 3D photography for beginners,” in ACM SIGGRAPH 2009 Courses (ACM, 2009), p. 8.

S. B. Gokturk, H. Yalcin, and C. Bamji, “A time-of-flight depth sensor-system description, issues and solutions,” in Workshop on Computer Vision and Pattern Recogntion (IEEE, 2004). pp. 35.

H. Hirschmuller, “Accurate and efficient stereo processing by semiglobal matching and mutual information,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2005). pp. 807–814.

C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Sixth International Conference on Computer Vision (IEEE, 1998). pp. 839–846.
[Crossref]

K. He, J. Sun, and X. Tang, “Guided image filtering,” in European Conference on Computer Vision (Springer, 2010), pp. 1–14.

B. S. Choi, S. H. Kim, J. Lee, S. Chang, J. Park, S. J. Lee, and J. K. Shin, “CMOS image sensor using pixel aperture technique for single-chip 2D and 3D imaging,” in SENSORS (IEEE, 2017), pp. 1–3.

Supplementary Material (2)

NameDescription
» Visualization 1       This video shows a demonstration of the proposed depth extraction system with OPA camera.
» Visualization 1       This video shows a demonstration of the proposed depth extraction system with OPA camera.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (18)

Fig. 1
Fig. 1 Graphical illustration of the design of OPA camera. The pixel apertures are at different locations relative to the pixel center.
Fig. 2
Fig. 2 The color filter array used in the OPA camera.
Fig. 3
Fig. 3 A graphical illustration of the depth-disparity relationship with the OPA camera.
Fig. 4
Fig. 4 Depth extraction process for OPA camera.
Fig. 5
Fig. 5 The paths for cost aggregation where (a) shows the typical eight-path approach and (b) the approach used in this work.
Fig. 6
Fig. 6 Structure of DMP.
Fig. 7
Fig. 7 Hardware block diagram of the local normalization, which is the implementation result of Eq. (2).
Fig. 8
Fig. 8 Hardware block diagram of the cost volume generation, which is the implementation result of Eq. (3).
Fig. 9
Fig. 9 Hardware block diagram of the cost aggregation, which is the implementation result of Eq. (4).
Fig. 10
Fig. 10 Hardware block diagram of the depth noise reduction, which is the implementation result of Eq. (5).
Fig. 11
Fig. 11 Plot of the average estimated depth of a flat surface against its distance with (a) the proposed, (b) the OA, and (c) the PA cameras. The bars show the variance in disparity.
Fig. 12
Fig. 12 Color quality evaluation of OPA camera where (a) shows a synthetic ground truth color chart, (b) is the captured color chart by OPA camera, reference image taken by a mobile camera is shown in (c), and (d) is captured by OPA camera for qualitative analysis.
Fig. 13
Fig. 13 Mico-photograph of the DMP chip and details of its implementation
Fig. 14
Fig. 14 Power consumption of the DMP with operating frequency.
Fig. 15
Fig. 15 The components of the implemented system. (a) DMP chip on ASIC board, (b) the OPA sensor, and (C) the OPA camera evaluation board.
Fig. 16
Fig. 16 Scenes observed by the OPA camera and their depths estimated by the DMP. Gray scale bar shows disparity. The operating distance of the scenes is from 20 cm to 120 cm.
Fig. 17
Fig. 17 3D reconstruction of a human face using the proposed OPA system.
Fig. 18
Fig. 18 (a) The observed scene with a normal camera under IR illumination, (b) the same scene captured by OPA camera under IR illumination, and (c) its depth by the proposed system.

Tables (3)

Tables Icon

Table 1 Comparison of camera based depth extraction schemes

Tables Icon

Table 2 Size of scanline buffer memories

Tables Icon

Table 3 Gate Count of DMP Sub-Blocks

Equations (8)

Equations on this page are rendered with MathJax. Learn more.

I ( x , y ) = I ( x + 1 , y ) I ( x 1 , y ) ,
I N ( x , y ) = I ( x , y ) μ Ω ( x , y ) σ Ω ( x , y ) ,
C ( x , y , d ) = x , y Π | I L ( x , y ) I R ( x + d , y ) | ,
A r ( x , y , d ) = C ( x , y , d ) + min { A r ( x 1 , y , d ) A r ( x 1 , y , d 1 ) + P 1 A r ( x 1 , y , d + 1 ) + P 1 min i A r ( x 1 , y , i ) + P 2 } min k A r ( x 1 , y , k ) ,
D ( x , y ) = i , j ψ D i ( i , j ) 1 { | I ( i , j ) I ( x , y ) | < S I ( x , y ) } i , j ψ 1 { | I ( i , j ) I ( x , y ) | < S I ( x , y ) } ,
Memory = ( M 1 ) W B .
Q ( x ) = ( x 1.96 × σ ( x ) Δ d Δ x , x + 1.96 × σ ( x ) Δ d Δ x ) ,
g ( x , y ) = c 1 w ( x , y ) + c 2 r ( x , y ) + c 3 b ( x , y ) + c 4

Metrics