Physical Sciences and Mathematics Commons^™

High-Speed Distributed Data Process Of Photometric Astronomical Data, Paul Doyle

Other

Since the 1970s the CCD has been the principle method of measuring flux to calculate the apparent magnitude of celestial objects within astronomical photometry. Each CCD image must be digitally cleaned and calibrated prior to its use. As data archives increase in size to Petabytes, the data processing challenge requires image processing techniques to continue to exceed the rate of data capture.

This paper describes NIMBUS, a rapidly scalable, failure resilient distributed network architecture capable of processing CCD image data at a rate of hundreds of Terabytes per day. NIMBUS is implemented using a decentralized web queue to control the …