XSEDE16 has ended
Back To Schedule
Tuesday, July 19 • 4:30pm - 5:00pm
SW: Introducing a New Client/Server Framework for Big Data Analytics with the R Language

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Historically, large scale computing and interactivity have been at odds. This is a particularly sore spot for data analytics applications, which are typically interactive in nature. To help address this problem, we introduce a new client/server framework for the R language. This framework allows the R programmer to remotely control anywhere from one to thousands of batch servers running as cooperating instances of R. And all of this is done from the user's local R session. Additionally, no specialized software environment is needed; the framework is a series of R packages, available from CRAN. The communication between client and server(s) is handled by the well-known ZeroMQ library. To handle computations, we use the established pbdR packages for large scale distributed computing. These packages utilize HPC standards like MPI and ScaLAPACK to handle complex, tightly-coupled computations on large datasets. In this paper, we outline the client/server architecture, discuss the pros and cons to this approach, and provide several example workflows which bring

interactivity to terabyte size computations.

Tuesday July 19, 2016 4:30pm - 5:00pm EDT