rcrawl

= Rcrawl web crawler for Ruby Rcrawl is a web crawler written entirely in ruby. It's limited right now by the fact that it will stay on the original domain provided.

Rcrawl uses a modular approach to processing the HTML it receives. The link exraction portion of Rcrawl depends on the scrAPI toolkit by Assaf Arkin (http://labnotes.org). gem install scrapi

The structure of the crawling process was inspired by the specs of the Mercator crawler (http://www.cindoc.csic.es/cybermetrics/pdf/68.pdf). A (somewhat) brief overview of the design philosophy follows.

== The Rcrawl process

1. Remove an absolute URL from the URL Server.
2. Download corresponding document from the internet, grabbing and processing robots.txt first, if available.
3. Feed the document into a rewind input stream(ris) to be read/re-read as needed. Based on MIME type, invoke the process method of the
4. processing module associated with that MIME type. For example, a link extractor or tag counter module for text/html MIME types, or a gif stats module for image/gif. By default, all text/html MIME types will pass through the link extractor. Each link will be converted to an absolute URL and tested against a (ideally user-supplied) URL filter to determine if it should be downloaded.
5. If the URL passes the filter (currently hard coded as Same Domain?), then call the URL-seen? test.
6. Has the URL been seen before? Namely, is it in the URL Server or has it been downloaded already? If the URL is new, it is added to the URL Server.
7. Back to step 1, repeat until the URL Server is empty.

== Examples

Instantiate a new Rcrawl object

crawler = Rcrawl::Crawler.new(url)

Begin the crawl process

crawler.crawl

== After the crawler is done crawling

Returns an array of visited links

crawler.visited_links

Returns a hash where the key is a url and the value is

the raw html from that url

crawler.raw_html

Returns a hash where the key is a URL and the value is

the error message from stderr

crawler.errors

Returns an array of external links

crawler.external_links

Set user agent

crawler.user_agent = "Your fancy crawler name here"

== License Copyright © 2006 Digital Duckies, LLC, under MIT License

Developed for http://digitalduckies.net

News, code, and documentation at http://blog.digitalduckies.net