flexsearch

What is flexsearch?

FlexSearch is a high-performance, full-text search library for JavaScript. It is designed to be fast and efficient, providing a variety of indexing and search capabilities. FlexSearch can be used in both Node.js and browser environments, making it versatile for different types of applications.

What are flexsearch's main functionalities?

Indexing

FlexSearch allows you to create an index and add documents to it. Each document is associated with a unique ID.

const FlexSearch = require('flexsearch');
const index = new FlexSearch();
index.add(1, 'Hello World');
index.add(2, 'Hi there');

Searching

You can perform searches on the indexed documents. The search results will return the IDs of the matching documents.

index.search('Hello', function(results) {
  console.log(results); // [1]
});

Asynchronous Search

FlexSearch supports asynchronous search operations, which can be useful for handling large datasets or integrating with other asynchronous workflows.

index.search('Hello').then(function(results) {
  console.log(results); // [1]
});

Custom Configuration

FlexSearch allows for extensive customization of the indexing and search process. You can configure encoding, tokenization, threshold, and resolution to suit your specific needs.

const customIndex = new FlexSearch({
  encode: 'icase',
  tokenize: 'forward',
  threshold: 0,
  resolution: 3
});
customIndex.add(1, 'Custom configuration example');

Other packages similar to flexsearch

lunr

Lunr.js is a small, full-text search library for use in the browser and Node.js. It provides a simple and easy-to-use API for indexing and searching documents. Compared to FlexSearch, Lunr.js is more lightweight but may not offer the same level of performance and customization.

elasticlunr

Elasticlunr.js is a lightweight full-text search library that is inspired by Lunr.js and Elasticsearch. It offers more features than Lunr.js, such as support for multiple languages and more advanced search capabilities. However, it may still not match the performance and flexibility of FlexSearch.

search-index

Search-Index is a powerful, full-text search library for Node.js and the browser. It provides a wide range of features, including real-time indexing, faceted search, and more. While it offers more advanced features than FlexSearch, it may be more complex to set up and use.

README

FlexSearch

Superfast, lightweight and memory efficient full text search library with zero dependencies.

When it comes to raw search speed FlexSearch outperforms every searching library out there and also provides flexible search capabilities like multi-word, phonetics or partial matching. Keep in mind that updating / removing items from the index has a significant cost. When your index needs to be updated continuously then BulkSearch may be a better choice. FlexSearch also provides you a asynchronous processing model as well as web workers to perform any updates on the index in background.

Benchmark: https://jsperf.com/flexsearch

Supported Platforms:

• Browser
• Node.js

Supported Module Definitions:

• AMD (RequireJS, Xone)
• CommonJS (Node.js, Xone)
• Global (window)

All Features:

• Web-Worker Support (not available in Node.js)
• Partial Matching
• Multiple Words
• Flexible Word Order
• Phonetic Search
• Limit Results
• Caching
• Asynchronous Mode
• Custom Matchers
• Custom Encoders

Web-Worker Support

Workers get its own dedicated memory. Especially for larger indexes, web worker improves speed and available memory a lot. FlexSearch index was tested with a 250 Mb text file including 10 Million words. The indexing was done silently in background by multiple parallel running workers in about 7 minutes and reserve ~ 8.2 Mb memory. The search result took ~ 0.25 ms!

Note: It is slightly faster to use no web worker when the index isn't too big (< 100,000 words).

Compare BulkSearch vs. FlexSearch

Description	BulkSearch	FlexSearch
Access	Read-Write optimized index	Read-Memory optimized index
Memory	Large (~ 1 kb per word)	Tiny (~ 16 bytes per word)
Usage	Limited content, Index updates continously	Fastest possible search, Rare updates on index Low memory capabilities

Installation

HTML / Javascript

<html>
<head>
    <script src="https://cdn.rawgit.com/nextapps-de/flexsearch/master/flexsearch.min.js"></script>
</head>
...

Node.js

npm install flexsearch

In your code include as follows:

var FlexSearch = require("flexsearch");

Or pass in options when requiring:

var index = require("flexsearch").create({/* options */});

AMD

var FlexSearch = require("./flexsearch.js");

Usage (API)

Create a new index

var index = new FlexSearch();

alternatively you can also use:

var index = FlexSearch.create();

Create a new index with custom options

FlexSearch.create(options)

var index = new FlexSearch({

    // default values:

    encode: "icase",
    mode: "forward",
    multi: false,
    ordered: false,
    async: false,
    cache: false
});

Read more: Phonetic Search, Phonetic Comparison, Improve Memory Usage

Add items to an index

Index.add_(id, string)

index.add(10025, "John Doe");

Search items

Index.search(string, limit, callback)

index.search("John");

Limit the result

index.search("John", 10);

Perform queries asynchronously

index.search("John", function(result){
    
    // array of results
});

Update item to the index

Index.update(id, string)

index.update(10025, "Road Runner");

Remove item to the index

Index.remove(id)

index.remove(10025);

Destroy the index

index.destroy();

Initialize the index

Index.init(options)

index.init();

Add custom matcher

Index.addMatcher(KEY_VALUE_PAIRS)

index.addMatcher({

    'ä': 'a', // replaces all 'ä' to 'a'
    'ö': 'o',
    'Ü': 'u'
});

Add custom encoder

var index = new FlexSearch({

    encode: function(str){
    
        // do something with str ...
        
        return str;
    }
});

Get info

index.info();

Returns information about the index, e.g.:

{
    "bytes": 3600356288,
    "id": 0,
    "matchers": 0,
    "size": 10000,
    "status": false
}

Note: When the fragmentation value is about 50% or higher, your should consider using cleanup() to free all fragmented available memory.

Optimize / Cleanup index

index.cleanup();

Options

Option	Values	Description
mode	"strict" "foward" "inverse" "full"	The indexing mode.
encode	false "icase" "simple" "advanced" "extra" {function}	The encoding type. Choose one of the built-ins or pass a custom encoding function.
boolean	"and" "or"	The applied boolean model when comparing multiple words. Note: When using "or" the first word is also compared with "and". Example: a query with 3 words, results has either: matched word 1 & 2 and matched word 1 & 3.
multi	true false	Enable multi word processing.
ordered	true false	Multiple words has to be the same order as the matched entry.
cache	true false	Enable/Disable caching.
async	true false	Enable/Disable asynchronous processing.

Phonetic Encoding

Option	Description	Example	False-Positives	Compression
false	Turn off encoding	Reference: "Björn-Phillipp Mayer" Matches: "Phil"	no	no
icase	Case in-sensitive encoding	Reference: "Björn-Phillipp Mayer" Matches: "phil"	no	no
simple	Phonetic normalizations	Reference: "Björn-Phillipp Mayer" Matches: "bjoern fillip"	no	~ 7%
advanced	Phonetic normalizations + Literal transformations	Reference: "Björn-Phillipp Mayer" Matches: "filip meier"	no	~ 35%
extra	Phonetic normalizations + Soundex transformations	Reference: "Björn-Phillipp Mayer" Matches: "byorn mair"	yes	~ 60%

Compare Phonetic Encoder

Reference String: "Björn-Phillipp Mayer"

Query	iCase	Simple	Advanced	Extra
björn	yes	yes	yes	yes
björ	yes	yes	yes	yes
bjorn	no	yes	yes	yes
bjoern	no	no	yes	yes
philipp	no	no	yes	yes
filip	no	no	yes	yes
björnphillip	no	yes	yes	yes
meier	no	no	yes	yes
björn meier	no	no	yes	yes
meier fhilip	no	no	yes	yes
byorn mair	no	no	no	yes
(false positives)	no	no	no	yes

Compare Memory Usage

Note: The required memory depends on several options.

Encoding	Memory usage of every ~ 100,000 indexed words
"icase" (default) / false	210 kb
"simple"	190 kb
"advanced"	150 kb
"extra"	90 kb
Mode	Multiplied with:
"strict"	x 1
"forward" (default)	x 1.5
"inverse"	x 2
"full"	x 2.3

---

Author FlexSearch: Thomas Wilkerling
License: Apache 2.0 License