mongo-fetcher

MongoFetcher

It exposes a function mongoForeach() which, given a collection and a query, will perform an async task for every document matching the query. For example, let's assume we want to remove all our archives in a S3 bucket, which are referenced in mongo.

Our File collection contains documents which look like this :

{
  documentType: string,
  name: string,
  ... [other fields]
}

We could use our mongoForeach to upload them with a high throughput :

  const { mongoForeach } = require('@payfit/mongo-fetcher')
  const db = require('./db') // here we assume that ./db exposes a valid mongodb connection
  const s3service = require('./s3service')

  await mongoForeach(
    db.collection('File'),
    { documentType: 'archive' }, // we want only archives
    async file => { // we can use our db document and do any task we want with it
      try {
        await s3service.deleteFile(file.name)
        await db.collection('File').removeOne({ _id: file._id })
      } catch (err) {
        // handle error
      }
    },
    {
      batchSize: 150, // we fetch items from mongo 150 by 150
      concurrency: 20, // we want to have maximum 20 promises running in parallel that process the task
    }
  )