@elastic.io/batching-library

batching-library

Library for batching in elastic.io. Batching allows users to combine several calls to API in one big call.

BatchClient

BatchClient represents client that works with batches. It contains logic for creating, updating statuses and storing batches.

Constructor

new BatchClient(store, config)

BatchClient constructor expects three parameters:

1. store - implementation of IStore interface. For example MongoStore. This is where batches will be stored.
2. config - instance of BatchConfig. It represent configuration and defines limitation for batches e.g: maxSize, maxItemsNumber.

Example

import { MongoStore, IStore } from '@elastic.io/batching-library';
import { BatchClient } from '@elastic.io/batching-library';
import { BatchConfig } from '@elastic.io/batching-library';
import { ConnectionOptions } from 'mongoose';

const maxSize = 1000;
const maxItemsNumber = 10;
const maxWaitTime = 1000;
const maxEmitRate = 1;
const maxRetry = 0;
const uri: string = 'url_to_mongo';
const collection = 'collection_name';
const connectionOpt: ConnectionOptions = {
  user: 'user',
  pass: 'password', 
};
const mgStore: IStore = new MongoStore(collection, uri, connectionOpt);
const config = new BatchConfig(maxSize, maxItemsNumber, maxWaitTime, maxEmitRate, maxRetry);
const client = new BatchClient(mgStore, config);

Methods

1. saveItem(item)

Save provided item to batch, returns batch with saved item (without another Batch items).

BatchItem properties

Parameter	Type	Required	Description
id	string	false	If not specified - uuid v1 will be generated
item	string	true	Body of the Batch Item

Example

const batch: Batch = await client.saveItem({ id: 0, item: {}});

2. getReadyBatches()

Using provided emitter emit batches with status 'READY'

Example

const batches: Batch[] = await client.getReadyBatches();
  await Promise.all(batches.map(async (batch) => {
    try {
      
     // process batch implementation
      
      await client.updateBatchStatusById(batch.id, 'SUCCESS'); // batch was successfully processed
    } catch (e) {
      log.error('Error: %o', e);
      await client.updateBatchStatusById(batch.id, 'FAILED'); // batch processed with error
    }
  }));

Implementing your own batch storage

For reference implementation example take a look at MongoStore

1. Extend AbstractStore and override each abstract method.
2. Implementation must ensure that each operation is transactional

Environment variables

Name	Mandatory	Description	Values
LOG_OUTPUT_MODE	Yes	Log record view format	short long simple json bunyan
LOG_LEVEL	Yes	Log Level	fatal error warn info debug trace
MONGO_URL	No	URL of mongo db used during integration tests	url to mongo db
MONGO_USER	No	Username of mongo db user, used during integration tests	username
MONGO_PASSWORD	No	Password for connecting to mongo db, used during integration tests	password
MONGO_DB	No	Database of mongo db used during integration tests	database name
MONGO_DROP_DB	No	If 1 drop database each test integration test if 0 dont	1 0

Environment template file

Limitations

MongoStore:

1. Supported Mongo database versions: 4.0 and higher.

2. Retries do not implemented.

3. Emit rate not implemented.

4. Library is not guarantee sequence processing of batch item

MaesterStore:

1. Emit rate not implemented.