Globus offers a broad suite of research data management cappabilities to the research community as web-accessible services. The initial service, launched in 2010, focused on reliable, high-performance, secure data transfer; since thattime, Globus capabilities have been progressively enhancedin response to user demand. In 2015, secure data sharing and publication services were introduced. Other recent enhancements include support for secure HTTP data access, new storage system types (e.g., Amazon S3, HDFS, Ceph), endpoint search, and administrator management. A powerful new authentication and authorization platform service,Globus Auth, addresses identity, credential, and delegation management needs encountered in research environments.New REST APIs allow external and third-party services to leverage Globus data management, authentication, and authorization capabilities as a platform, for example when building research data portals. We describe these and other recent enhancements to Globus, review adoption trends (to date, 38,000 registered users have operated on more than 150PB and 25B les), and present future plans.
The Developing Applications with Networking Capabilities via End-to-End SDN (DANCES) project was a collaboration between The University of Tennessee’s National Institute for Computational Sciences (UT-NICS), Pittsburgh Supercomputing Center (PSC), Pennsylvania State University (Penn State), the National Center for Supercomputing Applications (NCSA), Texas Advanced Computing Center (TACC), Georgia Institute of Technology (Georgia Tech), the Extreme Science and Engineering Discovery Environment (XSEDE), and Internet2 to investigate and develop a capability to add network bandwidth scheduling capability via software-defined networking (SDN) programmability to selected cyberinfrastructure services and applications. DANCES, funded by the National Science Foundation’s Campus Cyberinfrastructure – Network Infrastructure and Engineering (CC-NIE) program award numbers 1341005, 1340953, and1340981, has investigated three vendor network devices in order to determine which implements the OpenFlow 1.3 standard with DANCES requirements of meters and per-port queueing in order to provide a network reservation and rate-limiting capability desired to implement the goals of DANCES. Of the devices tested the DANCES project determined that the Corsa DP6410 met the requirements of OpenFlow 1.3 especially the implementation of features of metering and per port queueing which allow complex quality of service configuration for network flows. After selection of the network device a test environment was setup between the University of Tennessee and PSC to simulate a supercomputer center compute and data transfer resource environment. This paper described the DANCES project, the DANCES OpenFlow 1.3 specification requirements, the determination and acquiring of a sufficient OpenFlow 1.3 network device, the provisioning of a test environment, the UT-NICS test plan and the exciting and successful results of the tests.
In recent years, big data analysis has been widely applied to many research fields including biology, physics, transportation, and material science. Even though the demands for big data migration and big data analysis are dramatically increasing in campus IT infrastructures, there are several technical challenges that need to be addressed. First of all, frequent big data transmission between storage systems in different research groups imposes heavy burdens on regular campus network. Second, the current campus IT infrastructure is not designed to fully utilize the hardware capacity for big data migration and analysis. Last but not the least, running big data applications on top of large-scale high-performance computing facilities is not straightforward, especially for researchers and engineers in non-IT disciplines. We develop a campus IT infrastructure for big data migration and analysis, called BIC-LSU, which consists of a task-aware Clos OpenFlow network, high-performance cache storage servers, customized high-performance transfer applications, a light-weight control framework to manipulate existing big data storage systems and job scheduling systems, and a comprehensive social networking-enabled web portal. BIC-LSU achieves 40Gb/s disk-to-disk big data transmission, maintains short average transmission task completion time, enables the convergence of control on commonly deployed storage and job scheduling systems, and enhances easiness of big data analysis with a universal user-friendly interface. BIC-LSU software requires minimum dependencies and has high extensibility. Other research institutes can easily customize and deploy BIC-LSU as an augmented service on their existing IT infrastructures.
