Scarlet Task
Scarlet Task is an advanced task queue module for Node.js, featuring the ability to configure multiple child queues for
a single task queue.
Why named Scarlet? ๛ก(ー̀ωー́ก)
The story behind this module's name is quite serendipitous. One day, while browsing 萌否收音機, I
stumbled upon a captivating song. Though its title slipped my mind, I distinctly remembered its considerable length.
Driven by a desire to rediscover this elusive track, I developed this queue module. It allowed me to perform an
exhaustive search across the entire Moe FM platform. After a thorough exploration, I finally unearthed the song that had
been haunting my memory: <the Embodiment of Scarlet Devil>.
To commemorate this musical treasure hunt and pay homage to my favorite character,
Flandre Scarlet, I christened this module 'Scarlet Task'.
It's a nod to both the journey of rediscovery and the Scarlet-themed song that sparked it all.
Use Case
Scarlet Task is particularly useful for scenarios requiring controlled asynchronous operations, such as web crawling. By
allowing you to set up multiple child queues, it enables concurrent processing while maintaining control over
parallelism, preventing issues like unintentional DDoS-like behavior on target websites.
Installation
$ npm install --save scarlet-task
If you're using it in browser, you may need to install events
as well.
$ npm install --save events
Tutorials
First, require the module and instantiate a Scarlet object:
const { Scarlet } = require("scarlet-task");
const scarlet = new Scarlet(10);
The parameter for the constructor
represents the number of child queues. If no parameter is passed, it defaults to 1 child queue.
Define a processor
function to handle tasks. You can also pass an anonymous function:
function processor(taskObject) {
const task = taskObject.task;
taskObject.done();
console.log(scarlet.numberOfProcessed());
}
Note: In the processor
function, you should call taskObject.done()
or scarlet.taskDone(taskObject)
when you consider the task complete. Then scarlet
will process the next task. The taskObject
parameter is passed to you by the scarlet
instance.
You can push tasks into the queue at any time. The task object can be of any type - string, number, JSON, etc.:
const task = "This could be a URL, or an object that the processor can handle";
scarlet.push(task, processor);
For more reference, see test/hackernews.test.ts
.
You can reset the number of processed tasks:
scarlet.resetNumberOfProcessed();
You can also set an after-finish callback function that Scarlet will call after a specified number of tasks are completed:
scarlet.afterFinish(20, done, false);
scarlet.clearAfterFinish();
For more reference, see test/hackernews.test.ts
.
APIs
Scarlet
Constructor
const scarlet = new Scarlet(queueCount);
queueCount
(optional): Number of child queues. Defaults to 1 if not specified.
Methods
-
push(task, processor)
: Adds a new task to the queue.
task
: The task to be processed (can be any type).processor
: Function to process the task.
-
numberOfProcessed()
: Returns the number of processed tasks.
-
resetNumberOfProcessed()
: Resets the count of processed tasks to 0.
-
taskDone(taskObject)
: Marks a task as completed.
taskObject
: The task object to be marked as done.
-
afterFinish(count, processor, loop)
: Sets a callback to be executed after a certain number of tasks are completed.
count
: Number of tasks to complete before calling the processor.processor
: Function to be called after count
tasks are completed.loop
(optional): If true, the processor will be called repeatedly every count
tasks. Defaults to false.
-
clearAfterFinish()
: Clears the after-finish processor.
TaskObject
Properties
task
: The task data.queueId
: The ID of the queue processing this task.
Methods
done()
: Marks the task as completed.
Contribute
You're welcome to make pull requests!
「雖然我覺得不怎麼可能有人會關注我」