Abstract

Recent research has demonstrated that the fusion of complementary information captured by multi-modal sensors (visible and infrared cameras) enables robust pedestrian detection under various surveillance situations (e.g., daytime and nighttime). In this paper, we investigate a number of fusion architectures in an attempt to identify the optimal way of incorporating multispectral information for joint semantic segmentation and pedestrian detection. We made two important findings: (1) the sum fusion strategy, which computes the sum of two feature maps at the same spatial locations, delivers the best performance of multispectral detection, while the most commonly used concatenation fusion surprisingly performs the worst; and (2) two-stream semantic segmentation without multispectral fusion is the most effective scheme to infuse semantic information as supervision for learning human-related features. Based on these studies, we present a unified multispectral fusion framework for joint training of semantic segmentation and target detection that outperforms state-of-the-art multispectral pedestrian detectors by a large margin on the KAIST benchmark dataset.

© 2018 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Image processing pipeline for segmentation and material classification based on multispectral high dynamic range polarimetric images

Miguel Ángel Martínez-Domingo, Eva M. Valero, Javier Hernández-Andrés, Shoji Tominaga, Takahiko Horiuchi, and Keita Hirai
Opt. Express 25(24) 30073-30090 (2017)

Collaborative multicue fusion using the cross-diffusion process for salient object detection

Jin-Gang Yu, Changxin Gao, and Jinwen Tian
J. Opt. Soc. Am. A 33(3) 404-415 (2016)

ReLayNet: retinal layer and fluid segmentation of macular optical coherence tomography using fully convolutional networks

Abhijit Guha Roy, Sailesh Conjeti, Sri Phani Krishna Karri, Debdoot Sheet, Amin Katouzian, Christian Wachinger, and Nassir Navab
Biomed. Opt. Express 8(8) 3627-3642 (2017)

References

You do not have subscription access to this journal. Citation lists with outbound citation links are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access OSA Member Subscription

Cited By

You do not have subscription access to this journal. Cited by links are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access OSA Member Subscription

Figures (8)

You do not have subscription access to this journal. Figure files are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access OSA Member Subscription

Tables (3)

You do not have subscription access to this journal. Article tables are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access OSA Member Subscription

Equations (9)

You do not have subscription access to this journal. Equations are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access OSA Member Subscription

Metrics

You do not have subscription access to this journal. Article level metrics are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access OSA Member Subscription