Abstract
Hybrid data center network architectures, which combine electrical packet switches (EPSs) with optical circuit switches (OCSs), have been proposed to handle the inter-rack link oversubscription problem in a cost- and power-efficient manner. However, the high reconfiguration delay of optical switching technologies makes it challenging to effectively leverage OCSs in these hybrid networks. In our prior work, we proposed a modified version of Hadoop, called Hadoop for hybrid networks (HHN), to create network traffic patterns that match the characteristics of OCSs. This paper presents a comprehensive comparative evaluation of HHN on hybrid networks and the original Hadoop on conventional EPS-only networks. Numerical results validate our hypothesis that it is feasible to achieve similar system-level and user-level performance with HHN while simultaneously achieving power and cost savings with the hybrid network. When the percentage of shuffle-heavy (SH) jobs is small, e.g., 5%, HHN performance is the same as that of the original Hadoop on an EPS-only network. When the percentage of SH jobs is large, e.g., 20%, the HHN performance is almost the same even at high loads, and even with a smaller number of input-block replicas when we placed an upper bound on the per-job input-data size. For large SH jobs with input-data sizes larger than the upper bound, the performance of HHN could be improved significantly when storing a larger number of replicas for each input block.
© 2018 Optical Society of America
Full Article | PDF ArticleMore Like This
Maotong Xu, Chong Liu, and Suresh Subramaniam
J. Opt. Commun. Netw. 10(4) 409-420 (2018)
Naoaki Yamanaka, Satoru Okamoto, Masayuki Hirono, Yukihiro Imakiire, Wataru Muro, Takehiro Sato, Eiji Oki, Andrea Fumagalli, and Malathi Veeraraghavan
J. Opt. Commun. Netw. 10(7) B15-B24 (2018)
Min Yee Teh, Zhenguo Wu, and Keren Bergman
J. Opt. Commun. Netw. 12(4) B44-B54 (2020)