XSEDE16 has ended
Tuesday, July 19 • 4:30pm - 5:00pm
SW: Introducing a New Client/Server Framework for Big Data Analytics with the R Language

Sign up or log in to save this to your schedule and see who's attending!

Historically, large scale computing and interactivity have been at odds. This is a particularly sore spot for data analytics applications, which are typically interactive in nature. To help address this problem, we introduce a new client/server framework for the R language. This framework allows the R programmer to remotely control anywhere from one to thousands of batch servers running as cooperating instances of R. And all of this is done from the user's local R session. Additionally, no specialized software environment is needed; the framework is a series of R packages, available from CRAN. The communication between client and server(s) is handled by the well-known ZeroMQ library. To handle computations, we use the established pbdR packages for large scale distributed computing. These packages utilize HPC standards like MPI and ScaLAPACK to handle complex, tightly-coupled computations on large datasets. In this paper, we outline the client/server architecture, discuss the pros and cons to this approach, and provide several example workflows which bring

interactivity to terabyte size computations.

Tuesday July 19, 2016 4:30pm - 5:00pm

Attendees (19)