@elastic.io/batching-library

batching-library

Library for batching in elastic.io. Batching allows users to combine several calls to API in one big call.

BatchClient

BatchClient represents client that works with batches. It contains logic for creating, updating statuses and storing batches.

Constructor

new BatchClient(store, config, emitter)

BatchClient constructor expects three parameters:

1. store - implementation of IStore interface. For example MongoStore. This is where batches will be stored.
2. config - instance of BatchConfig. It represent configuration and defines limitation for batches e.g: maxSize, maxItemsNumber.
3. emitter - implementation of Node.Js EventEmitter. Client use provided emmiter to communicate with upper layer.

Example

import { MongoStore, IStore } from '@elastic.io/batching-library';
import { BatchClient } from '@elastic.io/batching-library';
import { BatchConfig } from '@elastic.io/batching-library';
import { ConnectionOptions } from 'mongoose';

const maxSize = 1000;
const maxItemsNumber = 10;
const maxWaitTime = 1000;
const maxEmitRate = 1;
const maxRetry = 0;
const uri: string = 'url_to_mongo';
const collection = 'collection_name';
const connectionOpt: ConnectionOptions = {
  user: 'user',
  pass: 'password', 
};
const mgStore: IStore = new MongoStore(collection, uri, connectionOpt);
const config = new BatchConfig(maxSize, maxItemsNumber, maxWaitTime, maxEmitRate, maxRetry);
const emitter: any = new MyEmitter();
const client = new BatchClient(mgStore, config, emitter);

Methods

1. emitReadyBatches()

Using provided emitter emit batches with status 'READY'

Example

await client.emitReadyBatches();

2. saveItem(item)

Save provided item to batch, returns batch with saved item.

Example

batch: Batch = await client.saveItem({ id: 0, item: {}});

Implementing your own batch storage

For reference implementation example take a look at MongoStore

1. Extend AbstractStore and override each abstract method.
2. Implementation must ensure that each operation is transactional

Environment variables

Name	Mandatory	Description	Values
LOG_OUTPUT_MODE	Yes	Log record view format	short long simple json bunyan
LOG_LEVEL	Yes	Log Level	fatal error warn info debug trace
MONGO_URL	No	URL of mongo db used during integration tests	url to mongo db
MONGO_USER	No	Username of mongo db user, used during integration tests	username
MONGO_PASSWORD	No	Password for connecting to mongo db, used during integration tests	password
MONGO_DB	No	Database of mongo db used during integration tests	database name
MONGO_DROP_DB	No	If 1 drop database each test integration test if 0 dont	1 0

Environment template file

Limitations

version: 0.0.1

1. Currently only Mongo database storage for batches available. Supported Mongo database versions: 4.0 and higher.

2. Retries do not implemented.

3. Emit rate not implemented.