qlobber

What is qlobber?

The qlobber npm package is a fast and flexible topic-based publish/subscribe library for Node.js. It allows you to match topics to subscribers using a trie-based approach, which is efficient and scalable. This package is particularly useful for implementing message brokers, event emitters, and other pub/sub systems.

What are qlobber's main functionalities?

Basic Topic Matching

This feature allows you to add topics and match them against patterns. The code sample demonstrates how to add topics and retrieve matches for specific topics.

const Qlobber = require('qlobber').Qlobber;

let matcher = new Qlobber();
matcher.add('foo.*', 'it matches foo.bar');
matcher.add('foo.*', 'it also matches foo.baz');
matcher.add('foo.bar', 'it matches foo.bar exactly');

console.log(matcher.match('foo.bar')); // ['it matches foo.bar', 'it matches foo.bar exactly']
console.log(matcher.match('foo.baz')); // ['it also matches foo.baz']

Removing Topics

This feature allows you to remove specific topics from the matcher. The code sample demonstrates how to add and then remove a topic, showing the updated matches.

const Qlobber = require('qlobber').Qlobber;

let matcher = new Qlobber();
matcher.add('foo.*', 'it matches foo.bar');
matcher.add('foo.bar', 'it matches foo.bar exactly');

matcher.remove('foo.bar', 'it matches foo.bar exactly');

console.log(matcher.match('foo.bar')); // ['it matches foo.bar']

Wildcard Matching

This feature supports wildcard matching using '*' and '#' symbols. The code sample demonstrates how to use wildcards to match multiple topics.

const Qlobber = require('qlobber').Qlobber;

let matcher = new Qlobber();
matcher.add('foo.*', 'it matches foo.bar');
matcher.add('foo.#', 'it matches foo.bar and foo.baz');

console.log(matcher.match('foo.bar')); // ['it matches foo.bar', 'it matches foo.bar and foo.baz']
console.log(matcher.match('foo.baz.qux')); // ['it matches foo.bar and foo.baz']

Other packages similar to qlobber

mqtt

The mqtt package is a client library for the MQTT protocol, which is a lightweight messaging protocol for small sensors and mobile devices. It provides similar pub/sub functionality but is more focused on network communication and IoT applications. Unlike qlobber, mqtt is protocol-specific and includes features for connecting to MQTT brokers.

eventemitter2

The eventemitter2 package is an enhanced version of Node.js's EventEmitter class. It provides advanced features like wildcard event names, which are similar to qlobber's topic matching. However, eventemitter2 is more general-purpose and is used for event handling within applications rather than for message brokering.

postal

The postal package is a JavaScript library for pub/sub messaging. It offers similar topic-based subscription and publication features. Postal is more focused on in-app messaging and provides additional features like message channels and middleware, making it more versatile for complex applications compared to qlobber.

README

qlobber   

Node.js globbing for amqp-like topics.

Note: Version 5.0.0 adds async and worker thread support when used on Node 12+.

Example:

var assert = require('assert');
var Qlobber = require('qlobber').Qlobber;
var matcher = new Qlobber();
matcher.add('foo.*', 'it matched!');
assert.deepEqual(matcher.match('foo.bar'), ['it matched!']);
assert(matcher.test('foo.bar', 'it matched!'));

The API is described here.

qlobber is implemented using a trie, as described in the RabbitMQ blog posts here and here.

Installation

npm install qlobber

Another Example

A more advanced example using topics from the RabbitMQ topic tutorial:

var assert = require('assert');
var Qlobber = require('qlobber').Qlobber;
var matcher = new Qlobber();
matcher.add('*.orange.*', 'Q1');
matcher.add('*.*.rabbit', 'Q2');
matcher.add('lazy.#', 'Q2');
assert.deepEqual(['quick.orange.rabbit',
                  'lazy.orange.elephant',
                  'quick.orange.fox',
                  'lazy.brown.fox',
                  'lazy.pink.rabbit',
                  'quick.brown.fox',
                  'orange',
                  'quick.orange.male.rabbit',
                  'lazy.orange.male.rabbit'].map(function (topic)
                  {
                      return matcher.match(topic).sort();
                  }),
                 [['Q1', 'Q2'],
                  ['Q1', 'Q2'],
                  ['Q1'],
                  ['Q2'],
                  ['Q2', 'Q2'],
                  [],
                  [],
                  [],
                  ['Q2']]);

Async Example

Same as the first example but using await:

const assert = require('assert');
const { Qlobber } = require('qlobber').set_native(require('qlobber-native'));
const matcher = new Qlobber.nativeString();

(async () => {
    await matcher.addP('foo.*', 'it matched!');
    assert.deepEqual(await matcher.matchP('foo.bar'), ['it matched!']);
    assert(await matcher.testP('foo.bar', 'it matched!'));
})();

Worker Thread Example

Same again but the matching is done on a separate thread:

const { Qlobber } = require('qlobber').set_native(require('qlobber-native'));
const {
    Worker, isMainThread, parentPort, workerData
} = require('worker_threads');

if (isMainThread) {
    const matcher = new Qlobber.nativeString();
    matcher.add('foo.*', 'it matched!');
    const worker = new Worker(__filename, {
        workerData: matcher.state_address
    });
    worker.on('message', msg => {
        const assert = require('assert');
        assert.deepEqual(msg, [['it matched!'], true]);
    });
} else {
    const matcher = new Qlobber.nativeString(workerData);
    parentPort.postMessage([
        matcher.match('foo.bar'),
        matcher.test('foo.bar', 'it matched!')
    ]);
}

Licence

MIT

Tests

qlobber passes the RabbitMQ topic tests (I converted them from Erlang to Javascript).

To run the tests:

npm test

Lint

grunt lint

Code Coverage

npm run coverage

c8 results are available here.

Coveralls page is here.

Benchmarks

grunt bench

qlobber is also benchmarked in ascoltatori.

Native Qlobbers

The Javascript Qlobbers don't support asynchronous calls and worker threads because Javascript values can't be shared between threads.

In order to support asynchronous calls and worker threads, a native C++ implementation is available in the qlobber-native module.

Add qlobber-native as a dependency to your project and then add it to qlobber like this:

require('qlobber').set_native(require('qlobber-native'));

Note that set_native returns qlobber's exports so you can do something like this:

const { Qlobber } = require('qlobber').set_native(require('qlobber-native'));

Note that qlobber-native requires Gnu C++ version 9+ and Boost 1.70+, including the boost_context runtime library.

Once's you've added it to qlobber, the following classes will be available alongside the Javascript classes:

• Qlobber.nativeString
• Qlobber.nativeNumber
• QlobberDedup.nativeString
• QlobberDedup.nativeNumber
• QlobberTrue.native

They can only hold values of a single type (currently strings or numbers).

Asynchronous calls

The native classes support the same API as the Javascript classes but have the following additional methods:

• addP
• removeP
• matchP
• match_iterP
• testP
• clearP
• visitP
• get_restorerP

They correspond to their namesakes but return Promises. Note that match_iterP and visitP return async iterators.

API

Source: lib/qlobber.js

• Qlobber
• Qlobber.prototype.add
• Qlobber.prototype.remove
• Qlobber.prototype.match
• Qlobber.prototype.match_iter
• Qlobber.prototype.test
• Qlobber.prototype.test_values
• Qlobber.prototype.clear
• Qlobber.prototype.visit
• Qlobber.prototype.get_restorer
• QlobberDedup
• QlobberDedup.prototype.test_values
• QlobberDedup.prototype.match
• QlobberTrue
• QlobberTrue.prototype.test_values
• QlobberTrue.prototype.match
• VisitorStream
• RestorerStream
• set_native

Qlobber([options])

Creates a new qlobber.

Parameters:

• {Object} [options] Configures the qlobber. Use the following properties:

  • {String} separator The character to use for separating words in topics. Defaults to '.'. MQTT uses '/' as the separator, for example.

  • {String} wildcard_one The character to use for matching exactly one non-empty word in a topic. Defaults to '*'. MQTT uses '+', for example.

  • {String} wildcard_some The character to use for matching zero or more words in a topic. Defaults to '#'. MQTT uses '#' too.

  • {Boolean} match_empty_levels If true then wilcard_one also matches an empty word in a topic. Defaults to false.

  • {Boolean|Map} cache_adds Whether to cache topics when adding topic matchers. This will make adding multiple matchers for the same topic faster at the cost of extra memory usage. Defaults to false. If you supply a Map then it will be used to cache the topics (use this to enumerate all the topics in the qlobber).

  • {Integer} cache_splits How many topic.split results to cache. When you pass in a topic, it has to be split on the separator. Caching the results will make using the same topics multiple times faster at the cost of extra memory usage. Defaults to 0 (no caching). The number of split results cached is limited by the value you pass here.

  • {Integer} max_words Maximum number of words to allow in a topic. Defaults to 100.

  • {Integer} max_wildcard_somes Maximum number of wildcard_some words in a topic. Defaults to 3.

_{Go: TOC}

Qlobber.prototype.add(topic, val)

Add a topic matcher to the qlobber.

Note you can match more than one value against a topic by calling add multiple times with the same topic and different values.

Parameters:

• {String} topic The topic to match against.
• {Any} val The value to return if the topic is matched.

Return:

{Qlobber} The qlobber (for chaining).

_{Go: TOC | Qlobber.prototype}

Qlobber.prototype.remove(topic, [val])

Remove a topic matcher from the qlobber.

Parameters:

• {String} topic The topic that's being matched against.
• {Any} [val] The value that's being matched. If you don't specify val then all matchers for topic are removed.

Return:

{Qlobber} The qlobber (for chaining).

_{Go: TOC | Qlobber.prototype}

Qlobber.prototype.match(topic)

Match a topic.

Parameters:

• {String} topic The topic to match against.

Return:

{Array} List of values that matched the topic. This may contain duplicates. Use a QlobberDedup if you don't want duplicates.

_{Go: TOC | Qlobber.prototype}

Qlobber.prototype.match_iter()

Match a topic, returning the matches one at a time.

Return:

{Iterator} An iterator on the values that match the topic. There may be duplicate values, even if you use a QlobberDedup.

_{Go: TOC | Qlobber.prototype}

Qlobber.prototype.test(topic, val)

Test whether a topic match contains a value. Faster than calling match and searching the result for the value. Values are tested using test_values.

Parameters:

• {String} topic The topic to match against.
• {Any} val The value being tested for.

Return:

{Boolean} Whether matching against topic contains val.

_{Go: TOC | Qlobber.prototype}

Qlobber.prototype.test_values(vals, val)

Test whether values found in a match contain a value passed to test. You can override this to provide a custom implementation. Defaults to using indexOf.

Parameters:

• {Array} vals The values found while matching.
• {Any} val The value being tested for.

Return:

{Boolean} Whether vals contains val.

_{Go: TOC | Qlobber.prototype}

Qlobber.prototype.clear()

Reset the qlobber.

Removes all topic matchers from the qlobber.

Return:

{Qlobber} The qlobber (for chaining).

_{Go: TOC | Qlobber.prototype}

Qlobber.prototype.visit()

Visit each node in the qlobber's trie in turn.

Return:

{Iterator} An iterator on the trie. The iterator returns objects which, if fed (in the same order) to the function returned by get_restorer on a different qlobber, will build that qlobber's trie to the same state. The objects can be serialized using JSON.stringify, if the values you store in the qlobber are also serializable.

_{Go: TOC | Qlobber.prototype}

Qlobber.prototype.get_restorer([options])

Get a function which can restore the qlobber's trie to a state you retrieved by calling visit on this or another qlobber.

Parameters:

• {Object} [options] Options for restoring the trie.
  • {Boolean} cache_adds Whether to cache topics when rebuilding the trie. This only applies if you also passed cache_adds as true in the constructor.

Return:

{Function} Function to call in order to rebuild the qlobber's trie. You should call this repeatedly with the objects you received from a call to visit. If you serialized the objects, remember to deserialize them first (e.g. with JSON.parse)!

_{Go: TOC | Qlobber.prototype}

QlobberDedup([options])

Creates a new de-duplicating qlobber.

Inherits from Qlobber.

Parameters:

• {Object} [options] Same options as Qlobber.

_{Go: TOC}

QlobberDedup.prototype.test_values(vals, val)

Test whether values found in a match contain a value passed to test. You can override this to provide a custom implementation. Defaults to using has.

Parameters:

• {Set} vals The values found while matching (ES6 Set).
• {Any} val The value being tested for.

Return:

{Boolean} Whether vals contains val.

_{Go: TOC | QlobberDedup.prototype}

QlobberDedup.prototype.match(topic)

Match a topic.

Parameters:

• {String} topic The topic to match against.

Return:

{Set} ES6 Set of values that matched the topic.

_{Go: TOC | QlobberDedup.prototype}

QlobberTrue([options])

Creates a new qlobber which only stores the value true.

Whatever value you add to this qlobber (even undefined), a single, de-duplicated true will be stored. Use this qlobber if you only need to test whether topics match, not about the values they match to.

Inherits from Qlobber.

Parameters:

• {Object} [options] Same options as Qlobber.

_{Go: TOC}

QlobberTrue.prototype.test_values()

This override of test_values always returns true. When you call test on a QlobberTrue instance, the value you pass is ignored since it only cares whether a topic is matched.

Return:

{Boolean} Always true.

_{Go: TOC | QlobberTrue.prototype}

QlobberTrue.prototype.match(topic)

Match a topic.

Since QlobberTrue only cares whether a topic is matched and not about values it matches to, this override of match just calls test (with value undefined).

Parameters:

• {String} topic The topic to match against.

Return:

{Boolean} Whether the QlobberTrue instance matches the topic.

_{Go: TOC | QlobberTrue.prototype}

VisitorStream(qlobber)

Creates a new Readable stream, in object mode, which calls visit on a qlobber to generate its data.

You could pipe this to a JSONStream.stringify stream, for instance, to serialize the qlobber to JSON. See this test for an example.

Inherits from Readable.

Parameters:

• {Qlobber} qlobber The qlobber to call visit on.

_{Go: TOC}

RestorerStream(qlobber)

Creates a new Writable stream, in object mode, which passes data written to it into the function returned by calling get_restorer on a qlobber.

You could pipe a JSONStream.parse stream to this, for instance, to deserialize the qlobber from JSON. See this test for an example.

Inherits from Writable.

Parameters:

• {Qlobber} qlobber The qlobber to call get_restorer on.

_{Go: TOC}

set_native(qlobber_native)

Add qlobber-native to qlobber.

Parameters:

• {Object} qlobber_native The qlobber-native module, obtained using require('qlobber-native').

Return:

{Object} The qlobber exports with the following native classes added:

• Qlobber.nativeString
• Qlobber.nativeNumber
• QlobberDedup.nativeString
• QlobberDedup.nativeNumber
• QlobberTrue.native

_{Go: TOC}

—generated by apidox—