@contentstack/datasync-manager

Contentstack is a headless CMS with an API-first approach. It is a CMS that developers can use to build powerful cross-platform applications in their favorite languages. Build your application frontend, and Contentstack will take care of the rest. Read More.

Contentstack DataSync Manager

Contentstack DataSync lets you sync your Contentstack data with your database, enabling you to save data locally and serve content directly from your database. It is a combination of four powerful modules that is DataSync Webhook Listener, DataSync Manager, DataSync Asset Store Filesystem, DataSync Content Store — Filesystem and MongoDB.

The DataSync Manager is one of the four important components of Contentstack DataSync. When any publish, unpublish, or delete operations are performed on assets or content, the DataSync Manager fetches the data and sends it to Content Store. It uses Contentstack's Sync APIs to sync data from Contentstack with your preferred database — Filesystem and MongoDB in our case.

Prerequisite

• nodejs v20+

Usage

This is how DataSync Manager works with DataSync boilerplate:

const assetStore = require('@contentstack/datasync-asset-store-filesystem')
const contentStore = require('@contentstack/datasync-content-store-filesystem')
const listener = require('@contentstack/webhook-listener')
const syncManager = require('@contentstack/datasync-manager') // <--
const config = require('./config')

// Set asset, content store, listener and config to Datasync Manager
syncManager.setAssetStore(assetStore)
syncManager.setContentStore(contentStore)
syncManager.setListener(listener)
syncManager.setConfig(config)

// start DataSync manager
syncManager.start()
  .then(() => {
    console.log('Contentstack sync started successfully!')
  })
  .catch(console.error)

You can replace @contentstack/datasync-content-store-filesystem used above, with @contentstack/datasync-content-store-mongodb and switch content store databases.

Configuration

• Here's a list of contentstack's configuration keys

Key Name	Default	Description
apiKey		Required. Your stack's API key
deliveryToken		Required. Your environment's delivery token
sync_token		Token from where you'd like to start the process
pagination_token		Token from where you'd like to start the process
MAX_RETRY_LIMIT	6	Number of times the API call would retry, if the server fails

• Here's a list of configuration keys for contentstack datasync-manager:

Property	Default	Description
cooloff	3000	Number of ms the app would wait before making a subsequent Sync API request
enableAssetReferences	true	This is not implemented, you can help: #33
enableContentReferences	true	This is not implemented, you can help: #34
limit	100	Number of items fetched in each Sync API request made.
maxsize	2097152	Maximum file size of files (ex: .ledger file)
queue.pause_threshold	10000	The min-max internal queue size of contentstack-sync-manager. The DataSync manager would pause making Sync API requests once the internal queue size exceeds pause_threshold count.
queue.resume_threshold	5000	The min-max internal queue size of contentstack-sync-manager. The DataSync manager will resume Sync API requests once the queue size goes below resume_threshold count.
saveFilteredItems	true	When enabled, the app will log items that have been filtered out. Items can be filtered out due to the following: Missing required keys Language filters applied via config
filters	content_type_uid locale action	Filters allow you to narrow down the items that'd be returned by the SYNC API response. Ex: {filters: {content_type_uid: ['blogs']}, locale: ['en-us'], action: ['publish']}

And here's an example to get you started:

{
  // exmaple to override default values
  contentstack: {
      apiKey: '',
      deliveryToken: '',
      sync_token: '',
      MAX_RETRY_LIMIT: 5
    }
  },
  syncManager: {
      cooloff: 2000,
      limit: 80,
      filters: {
        content_type_uid: ['blogs'],
        locale: ['es-es', 'en-us'],
        action: ['publish', 'unpublish']
      }
    }
  }
}

Environment Variables

The following environment variables can be used to customize the behavior of Contentstack DataSync Manager:

Variable	Description	Default
TOKEN_PATH	Path to the directory where token/checkpoint/ledger files are stored.	Project root directory
PLUGIN_PATH	Path to the directory where plugins are loaded from.	Project root directory
NODE_ENV	Node.js environment (affects config/environment selection).	development
SYNC_ENV	Overrides the environment used for sync operations.	Value of NODE_ENV
KILLDURATION	Time (in ms) before the process is forcefully killed (overrides config).	Value from config

Note:

• TOKEN_PATH is especially useful for storing token data in a custom directory (e.g., for selective re-syncing based on timestamps).
• If a relative path is provided, it is resolved from the project root.
• These variables can be set in your shell or in your process manager configuration.

Further Reading

• Getting started with Contentstack DataSync
• Contentstack DataSync doc lists the configuration for different modules

Support and Feature requests

If you have any issues working with the library, please file an issue here at Github. You can send us an e-mail at [e-mail] if you have any support or feature requests. Our support team is available 24/7 on the intercom. You can always get in touch and give us an opportunity to serve you better!

Licence

This repository is published under the MIT license.