Expand this Topic clickable element to expand a topic
Skip to content
Optica Publishing Group

Evaluation Study of a Proposed Hadoop for Data Center Networks Incorporating Optical Circuit Switches

Abstract

Hybrid data center network architectures, which combine electrical packet switches (EPSs) with optical circuit switches (OCSs), have been proposed to handle the inter-rack link oversubscription problem in a cost- and power-efficient manner. However, the high reconfiguration delay of optical switching technologies makes it challenging to effectively leverage OCSs in these hybrid networks. In our prior work, we proposed a modified version of Hadoop, called Hadoop for hybrid networks (HHN), to create network traffic patterns that match the characteristics of OCSs. This paper presents a comprehensive comparative evaluation of HHN on hybrid networks and the original Hadoop on conventional EPS-only networks. Numerical results validate our hypothesis that it is feasible to achieve similar system-level and user-level performance with HHN while simultaneously achieving power and cost savings with the hybrid network. When the percentage of shuffle-heavy (SH) jobs is small, e.g., 5%, HHN performance is the same as that of the original Hadoop on an EPS-only network. When the percentage of SH jobs is large, e.g., 20%, the HHN performance is almost the same even at high loads, and even with a smaller number of input-block replicas when we placed an upper bound on the per-job input-data size. For large SH jobs with input-data sizes larger than the upper bound, the performance of HHN could be improved significantly when storing a larger number of replicas for each input block.

© 2018 Optical Society of America

Full Article  |  PDF Article
More Like This
PODCA: A Passive Optical Data Center Network Architecture

Maotong Xu, Chong Liu, and Suresh Subramaniam
J. Opt. Commun. Netw. 10(4) 409-420 (2018)

Application-Triggered Automatic Distributed Cloud/Network Resource Coordination by Optically Networked Inter/Intra Data Center [Invited]

Naoaki Yamanaka, Satoru Okamoto, Masayuki Hirono, Yukihiro Imakiire, Wataru Muro, Takehiro Sato, Eiji Oki, Andrea Fumagalli, and Malathi Veeraraghavan
J. Opt. Commun. Netw. 10(7) B15-B24 (2018)

Flexspander: augmenting expander networks in high-performance systems with optical bandwidth steering

Min Yee Teh, Zhenguo Wu, and Keren Bergman
J. Opt. Commun. Netw. 12(4) B44-B54 (2020)

Cited By

You do not have subscription access to this journal. Cited by links are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Figures (13)

You do not have subscription access to this journal. Figure files are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Tables (7)

You do not have subscription access to this journal. Article tables are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Equations (2)

You do not have subscription access to this journal. Equations are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Select as filters


Select Topics Cancel
© Copyright 2024 | Optica Publishing Group. All Rights Reserved