Databases and Information Systems Commons^™

Scraping Bepress: Downloading Dissertations For Preservation, Stephen Zweibel

Copyright, Fair Use, Scholarly Communication, etc.

This article will describe our process developing a script to automate downloading of documents and secondary materials from our library’s BePress repository. Our objective was to collect the full archive of dissertations and associated files from our repository into a local disk for potential future applications and to build out a preservation system.

Unlike at some institutions, our students submit directly into BePress, so we did not have a separate repository of the files; and the backup of BePress content that we had access to was not in an ideal format (for example, it included “withdrawn” items and did not …