TR-16-2.pdf

"Spark-GPU: an accelerated in-memory data processing engine on clusters" 

Yuan Yuan, Meisam Fathi Salmi, Yin Huai, Kaibo Wang, Rubao Lee, 
and Xiaodong Zhang 

Proceedings of 2016 IEEE International Conference on Big Data 
(IEEE BigData 2016), Washington DC, USA, December 5-8, 2016.


Abstract

Apache Spark is an in-memory data processing system that
supports both SQL queries and advanced analytics over large
data sets. In this paper, we present our design and implementation
of Spark-GPU that enables Spark to utilize GPU’s massively
parallel processing ability to achieve both high performance and
high throughput. Spark-GPU transforms a general-purpose data
processing system into a GPU-supported system by addressing
several real-world technical challenges including minimizing
internal and external data transfers, preparing a suitable data
format and a batching mode for efficient GPU execution, and
determining the suitability of workloads for GPU with a task
scheduling capability between CPU and GPU. We have comprehensively
evaluated Spark-GPU with a set of representative
analytical workloads to show its effectiveness. Our results show
that Spark-GPU improves the performance of machine learning
workloads by up to 16.13x and the performance of SQL queries
by up to 4.83x.