jh-search-index
Advanced tools
A text search index module for Node.js. Search-index allows applications to add, delete and retrieve documents from a corpus. Retrieved documents are ordered by tf-idf relevance, filtering on metadata, and field weighting
Table of Contents generated with DocToc
Search-index is a text search module for Node.js. Think "node version of Lucene, but much simpler".
Search-index allows you to perform free-text search over structured or unstructured data, and return a resultset ordered by relevance.
Search-index is built with the soooperfast levelUP module, and the very useful Natural module.
The Point of Search-Index is to simplify set up and operation of an search engine. Search-index is essentially free from configuration- the index is dynamic and morphs into the structure that you require automatically, based on the documents that it is fed.
Search-index is in an alpha stage- meaning that it has been known to work quite well, but edge cases and portability may be challenging. Query-result is robust. See known issues and performance tips below.
Search-index is currently the index powering the Norch search engine.
Releases are listed here. Generally you will want the most recent one.
The easiest way to include search-index in your project is by using npm
npm install search-index
The module can then be initialised by including the line
si = require('search-index')();
at the top of your app.
To make a searchable index, you must first add documents with si.add.
Documents are then searchable with si.search.
#Initialization
search-index is called with require like so:
var si = require('search-index')();
search-index can be initialized with options like so:
var options = { indexPath: 'si2', logLevel: 'error' }
var si = require('search-index')(options)
Available options
siinfo, debug (lots of logs) or error (nearly silent). Default is warnInserts document into the index
si.add({'batchName': batchName, 'filters': filters}, data, function(err) {
if (!err) console.log('indexed!');
});
Where batchName is any name to tag the batch, and filters tells the index which fields can be filtered and agregated on, and data is an object containing one or more documents in a format similar to:
[
{
'id':'1',
'title':'A really interesting document',
'body':'This is a really interesting document',
'metadata':['red', 'potato']
},
{
'id':'2',
'title':'Another interesting document',
'body':'This is another really interesting document that is a bit different',
'metadata':['yellow', 'potato']
}
]
...and filters is an array of field names that may be contained
in the document that the index will use for building filters. A filter
field must always be an array of single String tokens, for example
['metadata','places']. 'search-index' wont accept strings, so
remember to wrap home-rolled JSON with 'JSON.parse' to turn it into an
object.
Example:
var batch = [
{
'id':'1',
'title':'A really interesting document',
'body':'This is a really interesting document',
'metadata':['red', 'potato']
},
{
'id':'2',
'title':'Another interesting document',
'body':'This is another really interesting document that is a bit different',
'metadata':['yellow', 'potato']
}
];
var batchName = 'twoDocs';
var filters = ['metadata'];
si.add({'batchName': batchName, 'filters': filters}, batch, function(err) {
if (!err) console.log('indexed!');
});
Note: if you dont specify an id field, search-index will specify one for you.
Delete the document and all associated index entries.
si.del(docID, function(err) {
if (!err) console.log('success!');
});
Empties the search index, can be used in conjunction with replication.
si.empty(function(err) {
if (!err) console.log("Emptied! Search-index now contains no documents- please refeed or replicate");
});
Get the document and all associated index entries.
si.get(docID, function(err, doc) {
if (!err) console.log(doc);
});
A matcher is a service that generates a dictionary of words based on
the contents of the index, and then returns appropriate sets based on
substrings. For example, once the matcher is generated, a beginsWith
of "lon" might return ['London', 'longing', 'longitude'] depending on
the contents of the index. Terms are ordered by total occurances in
index.
si.match(beginsWith, function(err, matches) {
if (!err) console.log(matches);
});
Queries the search index
si.search(query, function(err, results) {
//check for errors and do something with search results, for example this:
if (!err) console.log(results)
});
...where query is an object similar to (see Query Parameters for more info):
{
"query": {
"*": [
"africa", "bank"
]
},
"facets": {
"totalamt": {
"ranges":[
["000000000000000","000000050000000"],
["000000050000001","100000000000000"]
]
},
"mjtheme": {
"ranges": [
["A","J"],
["K","Z"]
]
}
},
"offset": 0,
"pageSize": 100
}
Allows faceted navigation.
"facets": {
"totalamt": {"sort":"keyDesc"},
"price": {}
}
Defines the amount of entries per facet category. Defaults to 10.
"facets": {"places":{"sort":"keyDesc","limit":20}}
Defines "buckets" or "ranges" of values. ranges comprises an array of tuplet arrays, where each tuplet consists of a start and end (inclusive) of the range.
"facets": {
"totalamt": {
"ranges": [
[
"000000000000000",
"000000006000000"
],
[
"000000006000001",
"010000000000000"
]
]},
"mjtheme": {
"ranges": [
[
"A",
"J"
],
[
"K",
"Z"
]
]}
Defines the sort order of facets. Facets can be sorted on keys or values in an ascending or descening order. Possible values for facetSort are keyAsc, keyDesc, valueAsc, and valueDesc. The default sort is equivalent to valueDesc.
"facets": {
"totalamt": {"sort":"keyDesc"},
"mjtheme": {
"sort": "keyAsc",
"ranges": [
[
"A",
"J"
],
[
"K",
"Z"
]
]
}
}
Used to return a resultset on a facet.
'filter': {
'user': [['eklem', 'eklem']]
}
A free text string containing one or many tokens. * is
equivalent to 'search all fields'
"query": {"*":["usa"]}
You can also specify named fields like so :
"query": {
"title":["usa", "reagan"]
}
and so:
"query": {
"title": ["reagan"],
"body": ["intelligence", "agency", "contra"]
}
The starting point in the resultset of the results returned. Useful for paging
"offset": 0
The maximum number of results in the resultset that can be
returned. Counted from offset
"pageSize": 20
Creates a field that shows where the search terms exist in the given
field. For example, a teaser field could be generated from the
document field body
"teaser": "body"
Sets a factor by which the score of a field should be weighted. Useful for building custom relevancy models
"weight": {
"body": [
"10"
],
"title": [
"5"
]
}
Replicates an index from a snapshot file generated by si.snapshot.
//assumes that backup is in a file called 'backup.gz'
si.replicate(fs.createReadStream('backup.gz'), function(msg){
that.completed = true;
});
Returns a readStream that can then be piped on, for instance to file.
//assumes that: var fs = require('fs')
si.snapShot(function(readStream) {
readStream.pipe(fs.createWriteStream('backup.gz'))
.on('close', function() {
//a snapshot of the search-index now exists in the file 'backup.gz'
});
});
Returns metadata about the state of the index.
si.tellMeAboutMySearchIndex(function(msg) {
console.log(msg);
});
Search-index is released under the MIT license:
Copyright (c) 2013 Fergus McDowall
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
FAQs
A text search index module for Node.js. Search-index allows applications to add, delete and retrieve documents from a corpus. Retrieved documents are ordered by tf-idf relevance, filtering on metadata, and field weighting
We found that jh-search-index demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
GitHub removed 27 malicious pull requests attempting to inject harmful code across multiple open source repositories, in another round of low-effort attacks.
Security News
RubyGems.org has added a new "maintainer" role that allows for publishing new versions of gems. This new permission type is aimed at improving security for gem owners and the service overall.
Security News
Node.js will be enforcing stricter semver-major PR policies a month before major releases to enhance stability and ensure reliable release candidates.