Computer Engineering Commons^™

Observed Web Robot Behavior On Decaying Web Subsites, Joan A. Smith, Frank Mccown, Michael L. Nelson

Computer Science Faculty Publications

We describe the observed crawling patterns of various search engines (including Google, Yahoo and MSN) as they traverse a series of web subsites whose contents decay at predetermined rates. We plot the progress of the crawlers through the subsites, and their behaviors regarding the various file types included in the web subsites. We chose decaying subsites because we were originally interested in tracking the implication of using search engine caches for digital preservation. However, some of the crawling behaviors themselves proved to be interesting and have implications on using a search engine as an interface to a digital library.