Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More →

bulksearch

Package Overview

Dependencies

Advanced tools

Install Socket

Detect and block malicious and high-risk dependencies

Install

bulksearch

Superfast, lightweight and read-write optimized full text search library.

0.1.22
Source
npm

Version published: 7 years ago

Weekly downloads: 13

Maintainers: 1

Weekly downloads

Created: 7 years ago

Source

BulkSearch

Superfast, lightweight and read-write optimized full text search library.

When it comes to the overall speed, BulkSearch outperforms every searching library out there and also provides flexible search capabilities like multi-word, phonetics or partial matching. Adding, updating or removing items are also fast as searching them. When your index don't needs to be updated continuously then FlexSearch may be a better choice. BulkSearch also provides you a asynchronous processing model to perform any updates on the index in background.

Benchmark: https://jsperf.com/bulksearch

All Features:

Partial Words
Multiple Words
Flexible Word Order
Phonetic Search
Limit Results
Caching
Asynchronous Mode
Custom Matchers
Custom Encoders

Installation

Node.js

npm install bulksearch

In your code include as follows:

var BulkSearch = require("BulkSearch");

HTML / Javascript

<html>
<head>
    <script src="https://cdn.rawgit.com/nextapps-de/bulksearch/master/bulksearch.min.js"></script>
</head>
...

AMD

var BulkSearch = require("BulkSearch");

Usage (API)

Create a new index

var index = new BulkSearch();

alternatively you can also use:

var index = BulkSearch.create();

Create a new index with custom options

BulkSearch.create(options)

var index = new BulkSearch({

    // default values:

    type: "integer",
    encode: "icase",
    boolean: "and",
    size: 4000,
    depth: 3,
    multi: false,
    strict: false,
    ordered: false,
    async: false,
    cache: false
});

Add items to an index

Index.add_(id, string)

index.add(10025, "John Doe");

Search items

Index.search(string, limit, callback)

index.search("John");

Limit the result

index.search("John", 10);

Perform queries asynchronously

index.search("John", function(result){
    
    // array of results
});

Update item to the index

Index.update(id, string)

index.update(10025, "Road Runner");

Remove item to the index

Index.remove(id)

index.remove(10025);

Destroy the index

index.destroy();

Initialize the index

Index.init(options)

index.init();

Add custom matcher

Index.addMatcher(KEY_VALUE_PAIRS)

index.addMatcher({

    'ä': 'a', // replaces all 'ä' to 'a'
    'ö': 'o',
    'Ü': 'u'
});

Add custom encoder

var index = new BulkSearch({

    encode: function(str){
    
        // do something with str ...
        
        return str;
    }
});

Get info

index.info();

Returns information about the index, e.g.:

{
    "bytes": 3600356288,
    "chunks": 9,
    "fragmentation": 0,
    "fragments": 0,
    "id": 0,
    "length": 7798,
    "matchers": 0,
    "size": 10000,
    "status": false
}

Note: When the fragmentation value is about 50% or higher, your should consider using cleanup() to free all fragmented available memory.

Optimize / Cleanup index

index.cleanup();

Options

Option	Values	Description
type	"byte" "short" "integer" "float" "string"	The data type of passed IDs has to be specified on creation. It is recommended to uses to most lowest possible data range here, e.g. use "short" when IDs are not higher than 65,535.
encode	false "icase" "simple" "advanced" "extra" {function}	The encoding type. Choose one of the built-ins or pass a custom encoding function.
boolean	"and" "or"	The applied boolean model when comparing multiple words. Note: When using "or" the first word is also compared with "and". Example: a query with 3 words, results has either: matched word 1 & 2 and matched word 1 & 3.
size	2500 - 10000	The size of chunks. It depends on content length which value fits best. Short content length (e.g. User names) are faster with a chunk size of 2,500. Bigger text runs faster with a chunk size of 10,000. Note: It is recommended to use a minimum chunk size of the maximum content length which has to be indexed to prevent fragmentation.
depth	0 - 6	Set the depth of register. It is recommended to use a value in relation to the number of stored index and content length for an optimum performance-memory value. Note: Increase this options carefully!
multi	true false	Enable multi word processing.
ordered	true false	Multiple words has to be the same order as the matched entry.
strict	true false	Matches exactly needs to be started with the query.
cache	true false	Enable caching.

Phonetic Encoding

Option	Description	Example	False Positives	Compression Level
false	Turn off encoding	Reference: "Björn-Phillipp Mayer" Matches: "Phil"	no	no
icase	Case in-sensitive encoding	Reference: "Björn-Phillipp Mayer" Matches: "phil"	no	no
simple	Phonetic normalizations	Reference: "Björn-Phillipp Mayer" Matches: "bjoern fillip"	no	~ 3%
advanced	Phonetic normalizations + Literal transformations	Reference: "Björn-Phillipp Mayer" Matches: "filip meier"	no	~ 25%
extra	Phonetic normalizations + Soundex transformations	Reference: "Björn-Phillipp Mayer" Matches: "byorn mair"	yes	~ 50%

Compare Phonetic Search Results

Reference String: "Björn-Phillipp Mayer"

Query	ElasticSearch	BulkSearch (iCase)	BulkSearch (Simple)	BulkSearch (Adv.)	BulkSearch (Extra)
björn	yes	yes	yes	yes	yes
björ	no	yes	yes	yes	yes
bjorn	no	no	yes	yes	yes
bjoern	no	no	no	yes	yes
philipp	no	no	no	yes	yes
filip	no	no	no	yes	yes
björnphillip	no	no	yes	yes	yes
meier	no	no	no	yes	yes
björn meier	no	no	no	yes	yes
meier fhilip	no	no	no	yes	yes
byorn mair	no	no	no	no	yes
(false positives)	yes	no	no	no	yes

Compare Memory Usage

Note: The data type of passed IDs has to be specified on creation. It is recommended to uses to most lowest possible data range here, e.g. use "short" when IDs are not higher than 65,535.

ID Type	Range of Values	Memory usage of every ~ 100,000 indexed words (Index + Content)
Byte	0 - 255	683 kb + 2.4 Mb
Short	0 - 65,535	1.3 Mb + 2.4 Mb
Integer	0 - 4,294,967,295	2.7 Mb + 2.4 Mb
Float	0 - * (16 digits)	5.3 Mb + 2.4 Mb
String	* (unlimited)	1.3 Mb * char length of IDs + 2.4 Mb

Author BulkSearch: Thomas Wilkerling
License: Apache 2.0 License

FAQs

What is bulksearch?

Is bulksearch popular?

Is bulksearch well maintained?

Package last updated on 10 Mar 2018

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

bulksearch

BulkSearch

Superfast, lightweight and read-write optimized full text search library.

Installation

Node.js

HTML / Javascript

Usage (API)

Create a new index

Create a new index with custom options

Add items to an index

Search items

Limit the result

Perform queries asynchronously

Update item to the index

Remove item to the index

Destroy the index

Initialize the index

Add custom matcher

Add custom encoder

Get info

Optimize / Cleanup index

Options

Phonetic Encoding

Compare Phonetic Search Results

Compare Memory Usage

Related posts

The Cyber Security Council Podcast: Securing Modern Applications in a Decentralized Open Source World

Weaponizing OAST: How Malicious Packages Exploit npm, PyPI, and RubyGems for Data Exfiltration and Recon