@firecode/admin

Firecode

A light, fast, and memory-efficient collection traversal library for Firestore and Node.js.

---

Firecode is a Node.js library that lets you efficiently traverse Firestore collections.

When you have millions of documents in a collection, you can't just get all of them at once as your program's memory usage will explode. Firecode's configurable traverser objects let you do this in a simple, intuitive and memory-efficient way using batching.

Firecode is an extremely light, well-typed, zero-dependency library that is useful in a variety of scenarios. You can use it in database migration scripts (e.g. when you need to add a new field to all docs) or a scheduled Cloud Function that needs to check every doc in a collection periodically or a locally run script that retrieves some data from a collection.

Read the introductory blog post ▸

View the full documentation (docs)▸

Overview

1. Installation
2. Core Concepts
3. Quick Start
4. More Examples
5. API
6. Upgrading
7. License

Installation

Firecode is designed to work with the Firebase Admin SDK so if you haven't already installed it, run

npm install firebase-admin

Then run

npm install @firecode/admin

Core Concepts

There are only 2 kinds of objects you need to be familiar with when using this library:

1. Traverser: An object that walks you through a collection of documents (or more generally a Traversable).

2. Migrator: A convenience object used for database migrations. It lets you easily write to the documents within a given traversable and uses a traverser to do that. You can easily write your own migration logic in the traverser callback if you don't want to use a migrator.

Quick Start

Suppose we have a users collection and we want to send an email to each user. This is how easy it is to do that efficiently with a Firecode traverser:

import { firestore } from 'firebase-admin';
import { createTraverser } from '@firecode/admin';

const usersCollection = firestore().collection('users');
const traverser = createTraverser(usersCollection);

const { batchCount, docCount } = await traverser.traverse(async (batchDocs, batchIndex) => {
  const batchSize = batchDocs.length;
  await Promise.all(
    batchDocs.map(async (snapshot) => {
      const { email, firstName } = snapshot.data();
      await sendEmail({ to: email, content: `Hello ${firstName}!` });
    })
  );
  console.log(`Batch ${batchIndex} done! We emailed ${batchSize} users in this batch.`);
});

console.log(`Traversal done! We emailed ${docCount} users in ${batchCount} batches!`);

We are doing 3 things here:

1. Create a reference to the users collection
2. Pass that reference to the createTraverser() function
3. Invoke .traverse() with an async callback that is called for each batch of document snapshots

This pretty much sums up the core functionality of this library! The .traverse() method returns a Promise that resolves when the entire traversal finishes, which can take a while if you have millions of docs. The Promise resolves with an object containing the traversal details e.g. the number of docs you touched.

More Examples

Use a fast traverser

const projectsColRef = firestore().collection('projects');
const traverser = createFastTraverser(projectsColRef, {
  batchSize: 500,
  // This means we are prepared to hold 500 * 20 = 10,000 docs in memory
  maxConcurrentBatchCount: 20,
});
const { docCount } = await traverser.traverse(async (_, batchIndex) => {
  console.log(`Gonna process batch ${batchIndex} now!`);
  // ...
});
console.log(`Traversed ${docCount} projects super-fast!`);

Add a new field using a migrator

const projectsColRef = firestore().collection('projects');
const migrator = createBatchMigrator(projectsColRef);
const { migratedDocCount } = await migrator.update('isCompleted', false);
console.log(`Updated ${migratedDocCount} projects!`);

Add a new field derived from the previous fields

type UserDoc = {
  firstName: string;
  lastName: string;
};
const usersColRef = firestore().collection('users') as firestore.CollectionReference<UserDoc>;
const migrator = createBatchMigrator(usersColRef);
const { migratedDocCount } = await migrator.update((snap) => {
  const { firstName, lastName } = snap.data();
  return {
    fullName: `${firstName} ${lastName}`,
  };
});
console.log(`Updated ${migratedDocCount} users!`);

Use a fast migrator

const projectsColRef = firestore().collection('projects');
const fastTraverser = createFastTraverser(projectsColRef, { maxConcurrentBatchCount: 25 });
const fastMigrator = createBatchMigrator(fastTraverser);
const { migratedDocCount } = await fastMigrator.update('isCompleted', false);
console.log(`Updated ${migratedDocCount} projects super-fast!`);

Change traversal config

const walletsWithNegativeBalance = firestore().collection('wallets').where('money', '<', 0);
const migrator = createBatchMigrator(walletsWithNegativeBalance, {
  // We want each batch to have 500 docs. The size of the very last batch may be less than 500
  batchSize: 500,
  // We want to wait before moving to the next batch
  sleepBetweenBatches: true,
  // We want to wait 500ms before moving to the next batch
  sleepTimeBetweenBatches: 500,
});
// Wipe out their debts!
const { migratedDocCount } = await migrator.set({ money: 0 });
console.log(`Updated ${migratedDocCount} wallets!`);

Rename an optional field

type PostDoc = {
  text: string;
  postedAt?: firestore.Timestamp;
};
const postsColGroup = firestore().collectionGroup('posts') as firestore.CollectionGroup<PostDoc>;
const migrator = createBatchMigrator(postsColGroup);
const { migratedDocCount } = await migrator
  .withPredicate(
    // Ignore if it doesn't have a `postedAt` field
    (snap) => snap.data().postedAt !== undefined
  )
  .update((snap) => {
    const { postedAt } = snap.data();
    return {
      publishedAt: postedAt!, // Safe to assert now
      postedAt: firestore.FieldValue.delete(),
    };
  });
console.log(`Updated ${migratedDocCount} posts!`);

API

You can find the full API reference for @firecode/admin here. We maintain detailed docs for every version! Here are some of the core functions that this library provides.

createTraverser

Creates a traverser that facilitates Firestore collection traversals. When traversing the collection, this traverser invokes a specified async callback for each batch of document snapshots and waits for the callback Promise to resolve before moving to the next batch.

Complexity:

• Time complexity: O((N / batchSize) * (Q(batchSize) + C))
• Space complexity: O(batchSize * D + S)
• Billing: max(1, N) reads

where:

• N: number of docs in the traversable
• Q(batchSize): average batch query time
• D: average document size
• C: average callback processing time
• S: average extra space used by the callback

createFastTraverser

Creates a fast traverser that facilitates Firestore collection traversals. When traversing the collection, this traverser invokes a specified async callback for each batch of document snapshots and immediately moves to the next batch. It does not wait for the callback Promise to resolve before moving to the next batch so there is no guarantee that any given batch will finish processing before a later batch. This traverser uses more memory but is significantly faster than the default traverser.

Complexity:

• Time complexity: O(C + (N / batchSize) * Q(batchSize))
• Space complexity: O(maxConcurrentBatchCount * (batchSize * D + S))
• Billing: max(1, N) reads

where:

• N: number of docs in the traversable
• Q(batchSize): average batch query time
• D: average document size
• C: average callback processing time
• S: average extra space used by the callback

createMigrator

Creates a migrator that facilitates database migrations. The migrator accepts a custom traverser to traverse the collection. Otherwise it will create a default traverser with your desired traversal config. This migrator does not use atomic batch writes so it is possible that when a write fails other writes go through.

Complexity:

• Time complexity: TC(traverser) where C = W(batchSize)
• Space complexity: SC(traverser) where S = O(batchSize)
• Billing: max(1, N) reads, K writes

where:

• N: number of docs in the traversable
• K: number of docs that passed the migration predicate (K<=N)
• W(batchSize): average batch write time
• TC(traverser): time complexity of the underlying traverser
• SC(traverser): space complexity of the underlying traverser

createBatchMigrator

Creates a migrator that facilitates database migrations. The migrator accepts a custom traverser to traverse the collection. Otherwise it will create a default traverser with your desired traversal config. This migrator uses atomic batch writes so the entire operation will fail if a single write isn't successful.

Complexity:

• Time complexity: TC(traverser) where C = W(batchSize)
• Space complexity: SC(traverser) where S = O(batchSize)
• Billing: max(1, N) reads, K writes

where:

• N: number of docs in the traversable
• K: number of docs that passed the migration predicate (K<=N)
• W(batchSize): average batch write time
• TC(traverser): time complexity of the underlying traverser
• SC(traverser): space complexity of the underlying traverser

Upgrading

This project is still very new and we have a lot to work on. We will be moving fast and until we release v1, there may be breaking changes between minor versions (e.g. when upgrading from 0.4 to 0.5). However, all breaking changes will be documented and you can always use our Releases page as a changelog.

License

This project is made available under the MIT License.