MARIANE [electronic resource] : MApReduce Implementation Adapted for HPC Environments
- Published:
- Berkeley, Calif. : Lawrence Berkeley National Laboratory, 2011.
Oak Ridge, Tenn. : Distributed by the Office of Scientific and Technical Information, U.S. Dept. of Energy. - Physical Description:
- 8 : digital, PDF file
- Additional Creators:
- Lawrence Berkeley National Laboratory
United States. Department of Energy. Office of Scientific and Technical Information - Access Online:
- www.osti.gov
- Summary:
- MapReduce is increasingly becoming a popular framework, and a potent programming model. The most popular open source implementation of MapReduce, Hadoop, is based on the Hadoop Distributed File System (HDFS). However, as HDFS is not POSIX compliant, it cannot be fully leveraged by applications running on a majority of existing HPC environments such as Teragrid and NERSC. These HPC environments typicallysupport globally shared file systems such as NFS and GPFS. On such resourceful HPC infrastructures, the use of Hadoop not only creates compatibility issues, but also affects overall performance due to the added overhead of the HDFS. This paper not only presents a MapReduce implementation directly suitable for HPC environments, but also exposes the design choices for better performance gains in those settings. By leveraging inherent distributed file systems' functions, and abstracting them away from its MapReduce framework, MARIANE (MApReduce Implementation Adapted for HPC Environments) not only allows for the use of the model in an expanding number of HPCenvironments, but also allows for better performance in such settings. This paper shows the applicability and high performance of the MapReduce paradigm through MARIANE, an implementation designed for clustered and shared-disk file systems and as such not dedicated to a specific MapReduce solution. The paper identifies the components and trade-offs necessary for this model, and quantifies the performance gains exhibited by our approach in distributed environments over Apache Hadoop in a data intensive setting, on the Magellan testbed at the National Energy Research Scientific Computing Center (NERSC).
- Subject(s):
- Note:
- Published through SciTech Connect.
07/06/2011.
"lbnl-5427e"
Grid Computing (GRID), 2011 12th IEEE/ACM International conference, Lyon.
Govindaraju, Madhusudhan; Ramakrishnan, Lavanya; Fadika, Zacharia; Dede, Elif.
Computing Sciences Directorate - Funding Information:
- DE-AC02-05CH11231
View MARC record | catkey: 14653371