bee-queue - npm Package Compare versions

Comparing version 1.2.2 to 1.2.3

HISTORY.md

@@ -0,1 +1,10 @@
+1.2.3 / 2020-01-28
+==================
+
+  * Allow arbitrary values for progress (#109).
+  * Fix cleanup of event redis client (#178).
+  * Fix bluebird warning spam (#85).
+  * Update Typescript definition for `Queue#close` method (#180).
+  * Remove Node 4, 6 from CI (#156, 223151c).
+
 1.2.2 / 2018-01-25
@@ -2,0 +11,0 @@ ==================

index.d.ts

@@ -44,4 +44,5 @@ import { ClientOpts } from 'redis';
 
-  close(): Promise<void>;
   close(cb: () => void): void;
+  close(timeout?: number | null): Promise<void>;
+  close(timeout: number | undefined | null, cb: () => void): void;
 
@@ -48,0 +49,0 @@ removeJob(jobId: string): Promise<void>;

lib/job.js

@@ -138,8 +138,4 @@ 'use strict';
   reportProgress(progress, cb) {
-    // right now we just send the pubsub event
-    // might consider also updating the job hash for persistence
-    progress = parseInt(progress, 10);
-
     let promise;
-    if (progress >= 0 && progress <= 100) {
+    if (progress != null) {
       this.progress = progress;
@@ -158,3 +154,3 @@ promise = this.queue._commandable().then((client) => {
     } else {
-      promise = Promise.reject(new Error('Progress must be between 0 and 100'));
+      promise = Promise.reject(new Error('Progress cannot be empty'));
     }
@@ -161,0 +157,0 @@

lib/queue.js

@@ -198,3 +198,3 @@ 'use strict';
       }
-      if (this.settings.getEvents) {
+      if (this.eclient) {
         clients.push(this.eclient);
@@ -690,3 +690,3 @@ }
           /* istanbul ignore next: these are only redis and connection errors */
-          postStalled.catch(cb ? (err) => this.emit('error', err) : null);
+          postStalled.catch(cb ? (err) => this.emit('error', err) : () => {});
         }, interval);
@@ -693,0 +693,0 @@ });

package.json

 {
   "name": "bee-queue",
-  "version": "1.2.2",
+  "version": "1.2.3",
   "description": "A simple, fast, robust job/task queue, backed by Redis.",
@@ -5,0 +5,0 @@ "main": "index.js",

README.md

@@ -15,5 +15,6 @@ <a name="top"></a>
 
-const job = queue.createJob({x: 2, y: 3}).save();
+const job = queue.createJob({x: 2, y: 3})
+job.save();
 job.on('succeeded', (result) => {
-  console.log(`Received result for job ${job.id}: result`);
+  console.log(`Received result for job ${job.id}: ${result}`);
 });
@@ -74,2 +75,3 @@
   - [Stalling Jobs](#stalling-jobs)
+  - [Optimizing Redis Connections](#optimizing-redis-connections)
 - [API Reference](#api-reference)
@@ -199,3 +201,3 @@ - [Under The Hood](#under-the-hood)
 job.on('progress', (progress) => {
-  console.log(`Job ${job.id} reported progress: ${progress}%`);
+  console.log(`Job ${job.id} reported progress: page ${progress.page} / ${progress.totalPages}`);
 });
@@ -205,5 +207,5 @@
   // do some work
-  job.reportProgress(30);
+  job.reportProgress({ page: 3, totalPages: 11 });
   // do more work
-  job.reportProgress(80);
+  job.reportProgress({ page: 9, totalPages: 11 });
   // do the rest
@@ -213,3 +215,3 @@ });
 
-Just like `.process`, these `progress` events work across multiple processes or servers; the job instance will receive the progress event no matter where processing happens. Note that this mechanism depends on Pub/Sub, and thus will incur additional overhead for each additional worker node.
+Just like `.process`, these `progress` events work across multiple processes or servers; the job instance will receive the progress event no matter where processing happens. The data passed through can be any JSON-serializable value. Note that this mechanism depends on Pub/Sub, and thus will incur additional overhead for each additional worker node.
 
@@ -234,2 +236,50 @@ ## Job and Queue Events
 
+## Optimizing Redis Connections
+
+By default, every time you create a queue instance with `new Queue()` a new redis connection will be created. If you have a small number of queues accross a large number of servers this will probably be fine. If you have a large number of queues with a small number of servers, this will probably be fine too. If your deployment gets a bit larger you will likely need to optimize the Redis connections.
+
+Let's say for example you have a web application with 30 producer queues and you run 10 webservers & 10 worker servers, each one with 4 processes/server. With the default settings this is going to add up to a lot of Redis connections. Each Redis connection consumes a fairly large chunk of memory, and it adds up quickly!
+
+The producer queues are the ones that run on the webserver and they push jobs into the queue. These queues do not need to receive events so they can all share one redis connection by passing in an instance of [node_redis `RedisClient`](https://github.com/NodeRedis/node_redis#rediscreateclient).
+
+Example:
+
+```js
+// producer queues running on the web server
+const Queue = require('bee-queue');
+const redis = require('redis');
+const sharedConfig = {
+  getEvents: false,
+  isWorker: false,
+  redis: redis.createClient(process.env.REDIS_URL)
+};
+
+const emailQueue = new Queue('EMAIL_DELIVERY', sharedConfig);
+const facebookUpdateQueue = new Queue('FACEBOOK_UPDATE', sharedConfig);
+
+emailQueue.createJob({});
+facebookUpdateQueue.createJob({});
+```
+
+Note that these "producer queues" above are only relevant for the processes that have to put jobs into the queue, not for the workers that need to actually process the jobs.
+
+In your worker process where you define how to process the job with `queue.process` you will have to run "worker queues" instead of "producer queues". In the example below, even though you are passing in the shared config with the same redis instance, because this is a worker queue Bee-Queue will `duplicate()` the client because it needs the blocking commands for PubSub subscriptions. This will result in a new connection for each queue.
+
+```js
+// worker queues running on the worker server
+const Queue = require('bee-queue');
+const redis = require('redis');
+const sharedConfig = {
+  redis: redis.createClient(process.env.REDIS_URL)
+};
+
+const emailQueue = new Queue('EMAIL_DELIVERY', sharedConfig);
+const facebookUpdateQueue = new Queue('FACEBOOK_UPDATE', sharedConfig);
+
+emailQueue.process((job) => {});
+facebookUpdateQueue.process((job) => {});
+```
+
+For a more detailed example and explanation see [#96](https://github.com/bee-queue/bee-queue/issues/96)
+
 # API Reference
@@ -236,0 +286,0 @@

New alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Fixed alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Improved metrics

Total package byte prevSize

89088

increased by3.54%

Number of lines in readme file

866

increased by6.13%

Worsened metrics

Lines of code

1178

decreased by-0.17%

No dependency changes