Accelerating Big Data Processing with Hadoop, Spark and Memcached on Datacenters with Modern Architectures
When: April 8, 2017
(1:30pm-5:00pm)
Where: Xi'an, China
Abstract
Apache Hadoop and Spark are gaining prominence in handling Big Data and
analytics. Similarly, Memcached in Web 2.0 environment is becoming important
for large-scale query processing. These middleware are traditionally written
with sockets and do not deliver best performance on datacenters with modern
high performance networks. In this tutorial, we will provide an in-depth
overview of the architecture of Hadoop components (HDFS, MapReduce, RPC, HBase,
etc.), Spark and Memcached. We will examine the challenges in re-designing the
networking and I/O components of these middleware with modern interconnects,
protocols (such as InfiniBand, iWARP, RoCE, and RSocket) with RDMA and storage
architecture. Using the publicly available software packages in the
High-Performance Big Data (HiBD, http://hibd.cse.ohio-state.edu) project,
we will provide case studies of the new designs for several
Hadoop/Spark/Memcached components and their associated benefits. Through these
case studies, we will also examine the interplay between high performance
interconnects, storage systems (NVM, SSD, HDD, and Parallel Filesystem), and multi-core platforms to
achieve the best solutions for these components.
Targeted Audience and Scope
The tutorial content is planned for half-a-day. This tutorial
is targeted for various categories of people working in the areas of Big Data
including high-performance Hadoop/Spark/Memcached, high performance communication and I/O
architecture, storage, networking, middleware, cloud computing and applications.
Specific audience this tutorial is aimed at include: - Scientists,
engineers, researchers, and students engaged in designing next-generation Big
Data systems and applications
- Designers and developers of Big Data,
Hadoop, Spark and Memcached middleware
- Newcomers to the field of Big Data who
are interested in familiarizing themselves with Hadoop, Spark, Memcached, RDMA, and
high-performance networking
- Managers and administrators responsible
for setting-up next generation Big Data environment and high-end systems/facilities
in their organizations/laboratories
The content level will be as follows: 30% beginner, 40%
intermediate, and 30% advanced. There is no fixed pre-requisite. As long as the
attendee has a general knowledge in Big Data, Hadoop, Spark, Memcached, high performance
computing, networking and storage architecture, and related issues, he/she will
be able to understand and appreciate it. The tutorial is designed in such a way
that an attendee gets exposed to the topics in a smooth and progressive manner.
This tutorial is organized as a coherent talk to cover multiple topics.
Outline of the Tutorial
- Introduction to Big Data Applications and Analytics
- Overview of MapReduce and Resilient Distributed Datasets (RDD) Programming Models
- Architecture Overview of Apache Hadoop, Spark and Memcached
- MapReduce and YARN
- HDFS
- Spark
- RPC
- HBase
- Memcached
- Overview of High-Performance Interconnects, Protocols, and Storage Architectures for Modern Datacenters
- InfiniBand and RDMA
- 10/40 GigE, iWARP and RoCE technologies
- RSocket and SDP protocols
- SSD-based storage
- Challenges in Accelerating Hadoop, Spark and Memcached on Modern Datacenters
- Overview of Benchmarks and Applications using Hadoop, Spark and Memcached
- Acceleration Case Studies and In-Depth Performance Evaluation
- MapReduce over InfiniBand with RDMA, NVM, SSD, and Lustre
- HDFS over InfiniBand with RDMA and Heterogeneous Storage (RAMDisk, NVM, SSD, HDD, and Lustre)
- Spark over InfiniBand with RDMA, NVM, SSD, and Lustre
- RPC over InfiniBand with RDMA
- HBase over InfiniBand with RDMA, NVM, and SSD
- Memcached over InfiniBand with RDMA, NVM, SSD, and Lustre
- The High-Performance Big Data (HiBD) Project and Associated Releases
- Ongoing and Future Activities for High-Performance Big Data
- Conclusion and Q&A
Brief Biography of Speakers
Dr. Dhabaleswar
K. (DK) Panda is a Professor of Computer Science at the Ohio State
University. He obtained his Ph.D. in computer engineering from the University
of Southern California. His research interests include parallel computer
architecture, high performance computing, communication protocols, files
systems, network-based computing, and Quality of Service. He has published over
400 papers in major journals and international conferences related to these
research areas. Dr. Panda and his research group members have been doing
extensive research on modern networking technologies including InfiniBand, HSE
and RDMA over Converged Enhanced Ethernet (RoCE). His research group is
currently collaborating with National Laboratories and leading InfiniBand and
10GigE/iWARP companies on designing various subsystems of next generation
high-end systems. The
MVAPICH2 (High Performance MPI over InfiniBand, iWARP and RoCE)
open-source software
package, developed by his research group, are currently being used by more than
2,700 organizations worldwide (in 83 countries). This software has enabled
several InfiniBand clusters (including the 1st one) to get into the latest
TOP500 ranking. More than 405,000 downloads of
these libraries have taken place from the project's site.
These software packages are also available with the
stacks for network vendors (InfiniBand and iWARP), server vendors and
Linux distributors. The RDMA-enabled Apache Hadoop, Spark
and Memcached packages, consisting of
acceleration for HDFS, MapReduce, RPC, Spark and Memcached, are publicly available from High-Performance Big Data (HiBD) project site:
http://hibd.cse.ohio-state.edu. These packages are currently being
used by more than 200 organizations in 27 countries. More than 18,900
downloads have taken place from the project's site.
Dr. Panda's research is supported
by funding from US National Science Foundation, US Department of Energy, and
several industry including Intel, Cisco, SUN, Mellanox, QLogic, NVIDIA and
NetApp. He is an IEEE Fellow and a member of ACM. More details about Dr.
Panda, including a comprehensive CV and publications are available here.
Dr. Xiaoyi Lu
is a Research Scientist
of the Department of Computer Science and Engineering at the Ohio
State University, USA. His current research interests include high
performance interconnects and protocols, Big Data, Hadoop/Spark/Memcached
Ecosystem, Parallel Computing Models (MPI/PGAS), Virtualization and
Cloud Computing. He has published over 70 papers in international
journals and conferences related to these research areas. He has been
actively involved in various professional activities (PC Co-Chair, PC
Member, Reviewer, Session Chair) in academic journals and
conferences. Recently, Dr. Lu is leading the research
and development of
RDMA-based accelerations for Apache Hadoop, Spark, HBase, and Memcached, and OSU HiBD
micro-benchmarks, which are publicly available from
(http://hibd.cse.ohio-state.edu). These libraries are currently being
used by more than 200 organizations from 27 countries. More than
18,900 downloads of these libraries have taken place from the project
site. He is a core member of the
MVAPICH2 (High Performance MPI over InfiniBand, iWARP and RoCE)
project and he is leading the
research and development of MVAPICH2-Virt (high-performance and scalable MPI for hypervisor and container based HPC cloud).
He is a member of IEEE and ACM. More
details about Dr. Lu are available at here.
Last Updated: January 12, 2017