async-delay-queue

async-delay-queue

Minimal ES7 async queue with dynamic delay between functions.

Usage

queue.delay(fn, delay, timer, add)

Returns a promise resolving the return value of the given function.

Argument	Description	Default
fn	Function to delay. Supports promise/async functions.	None
delay	Delay between this function and the last one in ms. Required.	null
timer	Max time until the active promise is forcefully resolved to prevent the queue getting stuck.	null
add	Method with which the function is added to the queue stack. "unshift" to add to start, "push" to add to end of queue.	"push"

Example

Crawl Google.com and retry when hitting rate limits.

const queue = require("async-delay-queue")
const request = require("request-promise") // for example purposes only

async crawl(url) {
  let res = await queue.delay(() => request(url), 100)

  // Hit rate limits? Put the same request at the start of the queue.
  // "unshift" to push at start, and increase delay to 10s.
  if (res.statusCode === 429) {
    res = await queue.delay(() => request(url), 10000, null, "unshift")
  }

  return res
}

// Crawls google.com 20 times with a 100ms delay between each request.
for (let i = 0; i < 20; i++) {
  crawl("http://google.com")
}