Physical Sciences and Mathematics Commons^™

An Extensible And Scalable Pilot-Mapreduce Framework For Data Intensive Applications On Distributed Cyberinfrastructure, Pradeep Kumar Mantha

LSU Master's Theses

The volume and complexity of data that must be analyzed in scientific applications is increasing exponentially. Often, this data is distributed; thus, the ability to analyze data by localizing it will yield limited returns. Therefore, an efficient processing of large distributed datasets is required, whilst ideally not introducing fundamentally new programming models or methods. For example, extending MapReduce - a proven effective programming model for processing large datasets, to work more effectively on distributed data and on different infrastructure (such as non-Hadoop, general-purpose clusters) is desirable. We posit that this can be achieved with an effective and efficient runtime environment …