downcache

downcache

Version 0.0.6

Downcache is a Node.js module for downloading and caching webpages for fast future retrieval. It is modeled on the download function we use at the UnitedStates project.

Any sort of scraping project often ends up hitting pages far more often than is reasonably necessary. This module functions like @mikeal's request -- in fact, it uses it as a dependency -- but stores a copy of the HTML on your local machine. The next time you make a request to that page using downcache, it checks for that local copy before making another call to the live page.

#Installation

npm install downcache

#Usage

var downcache = require("downcache");

downcache("http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=Jimmy%20Rollins&rvprop=content&format=json", function(err, resp, body) {
	// do something with the HTML body response
});

If you request this page sometime later, you will see that the response returns MUCH fast. That's because the response is loading from your hard drive, not the Internet.

#Callbacks

The only required input to downcache is a url. Most of the time, you'll want to pass a callback function as well. This receives three variables: An error (hopefully null), a response object that is either the response provided by request or an object indicating that the page was loaded from cache, and the body of the page requested. Do with them what you will (or not).

downcache("http://time.com/7612/americas-mood-map-an-interactive-guide-to-the-united-states-of-attitude/", function(err, resp, body) {
	if (err) throw err;
	if (resp.socket) {
		console.log(resp.socket.bytesRead);
	} else {
		console.log(resp);
	}
});

Any error from the request, file retrieval or file writing is elevated to the callback.

#Caching

By default, this module creates a cache directory in the current directory. The path to the cached file is created from the url so that the local file structure resembles the website being crawled.

#Options

There are a variety of options for you to specify and two ways to specify them.

-`dir`: The cache directory. Default is "cache"
-`path`: The filepath to write the url response to. Default is the url itself as a file structure
-`force`: Don't bother looking for the file in cache and call it live
-`nocache`: Don't write the response to cache. Then question why you are using this module.
-`json`: Run `JSON.parse` on the response
-`log`: Log level for module, using [npmlog](https://www.npmjs.com/package/npmlog) values: `verbose`, `info`, `warn` (default), `error`. 
-`limit`: How long to wait in milliseconds between each http call. Default is 100ms.

To specify options for a single URL call, you can pass a third argument to downcache between the url and the callback, like so:

downcache("http://example.com/article.html", { 
	dir: '/Users/jsmith/Desktop/cache/', 
	log: 'verbose'
}, function(err, resp, body) {
	//callback
});

But if you want to store the cache somewhere else, like in the above example, it would be a hassle to specify this every time. So you can also set pseudo-global options that apply to all future downcache calls using the set method:

downcache.set("dir", "/Users/jsmith/Desktop/cache/");
// or 
downcache.set({
	log: "info",
	json: true
});

If you specify these universal options, you can still override them with options passed to the main downcache() function call, but doing so with not overwrite the options specified by .set() for future calls.

#Note on downcache and wget

The default behavior for "path" is similar to the structure created by wget, in which the directory structure of the website is replicated on disk. At some future point, I may make this identical so that one can "precache" a site by mirroring it and then use this module to make requests to it, falling back on the live site.

The most common case for specifying your own path is if the site that you're requesting attaches session keys to the links in the source code, as many government database search results have the awful habit of doing. If you don't specify a path without these keys, they will fool the module into requesting a new URL each time.

#Rate limiting If you invoke this module many times in a row, there is built-in rate limiting to prevent bad behavior. The default is not more than one call per 100 milliseconds, which you can adjust with the limit option. (Feel free to call downcache() as you would normally, and the rate limiter will store the calls until their turn comes up.) I recommend you set the rate limiting interval using .set() at the beginning of the program and leave it at that value for the duration of the execution. Changing it midstream should not cause an interruption, but it can create some confusing race conditions.

#To Do -Allow for optional "should I cache?" logic so as to ignore certain types of responses you don't want (say, those that are under 1KB, indicating an error) -Allow for cache expiration -Return a better response when called from cache

#Changes v0.0.6 Rate limiting implemented. Can now set "global" options for all downcache calls in a session.

v0.0.5b Try out rate limiting

v0.0.5a Does not cache responses that have a status code other than 200.

v0.0.4 Checks to see if cached version is empty, and calls live if so.

v0.0.3 +Changed order of arguments passed to callback from (err, body, resp) to (err, resp, body) to match the request module.

#Compatibility

Downcache is tested on Node v0.10 and v0.11. It breaks on v0.8 due to a problem with an npmlog dependency.

#License MIT