mongo-es

Mongo-ES

A MongoDB to Elasticsearch connector

Installation

npm i -g mongo-es

Usage

# normal mode
mongo-es ./config.json

# debug mode, with debug info printed
NODE_ENV=dev mongo-es ./config.json

Concepts

Scan phase

scan entire database (or select with query) for existed documents

Tail phase

tail the oplog for new documents, updates or deletions

Configuration

Structure:

{
  "controls": {},
  "mongodb": {},  
  "elasticsearch": {},
  "tasks": [
    {
      "extract": {},
      "transform": {},
      "load": {}
    }
  ]
}

Detail example

controls

• mongodbReadCapacity - Max docs read per second (default: 10000).
• elasticsearchBulkSize - Max bluk size per request (default: 5000).
• tailFromTime - If set, program start to tail oplog with query: { ts: { $gte: new Timestamp(new Date(tailFromTime).getTime(), 0) } }, without scan entire database. (optional)

mongodb

• url - The connection URI string, eg: mongodb://user:password@localhost:27017/db?replicaSet=rs0. notice: 'db' in url will be ignored. notice: must use a admin user to access oplog.
• options - Connection settings, see: MongoClient. (optional)

elasticsearch

• options - Elasticsearch Config Options, see: Configuration.
• index - If set, auto create index when program start, see: Indeces Create. (optional)

task.extract

• db - Database name.
• collection - Collection name in database.
• query - Query selector, see Query. notice: not work in Tail phase.
• projection - Projection selector, see Projection. notice: works in Tail phase.
• sort Sort order of the result set. recommend: { "$natural": -1 }, new documents first, see Sort.

task.transform

• mapping - The field mapping from mongodb's collection to elasticsearch's index.
• parent - The field in mongodb's collection to use as the _parent in elasticsearch's index. (optional)

task.load

• index - The name of the index.
• type - The name of the document type.
• body - The request body, see Put Mapping.

License

Mozilla Public License Version 2.0