Dynamic Network-Centric Multi-cloud Platform for Real-Time and Data-Intensive Science Workflows

Abstract

Data-driven application systems often depend on complex, data-intensive programs operating on distributed datasets that originate from a variety of scientific instruments and repositories to provide time-critical responses for observed phenomena in different areas of science, e.g., weather warning systems, seismology, and ocean sciences, among others. A major challenge for these observational science application systems is the integration of data into the scientist’s workflow and how these workflows could leverage advanced networking and distributed computational capabilities to analyze real-time data streams. In particular and, moreover, in the case of dynamic data-driven applications systems (DDDAS), such capabilities become even more imperative. In this chapter, we present the DyNamo network-centric platform that addresses some of the critical challenges faced by dynamic data-driven workflows. DyNamo enables high-performance, adaptive, performance-isolated data flows across distributed cloud computing resources and community data repositories for analyzing data for observational science applications. DyNamo is capable of dynamically provisioning appropriate computing, networking and storage resources from diverse, national-scale cyberinfrastructures (CI). Through easy-to-use interfaces and integration with the Pegasus Workflow Management System, DyNamo is able to automate the orchestration of data-driven science workflows on the provisioned infrastructures, thereby offering capabilities that are crucial for support of DDDAS environments.