Physical Sciences and Mathematics Commons^™

Significantly Reducing The Processing Times Of High-Speed Photometry Data Sets Using A Distributed Computing Model, Paul Doyle, Fredrick Mtenzi, Niall Smith, Adrian Collins, Brendan O'Shea

Other

The scientific community is in the midst of a data analysis crisis. The increasing capacity of scientific CCD instrumentation and their falling costs is contributing to an explosive generation of raw photometric data. This data must go through a process of cleaning and reduction before it can be used for high precision photometric analysis. Many existing data processing pipelines either assume a relatively small dataset or are batch processed by a High Performance Computing centre. A radical overhaul of these processing pipelines is required to allow reduction and cleaning rates to process terabyte sized datasets at near capture rates using …