contentful-sync-cli
Command line program to sync Contentful data to local files on disk. Data is stored in JSON format.
Install
npm install -g contentful-sync-cli
Command Usage
Use of the Contentful Sync API requires an access token from Contentful. It should be stored in the CONTENTFUL_ACCESS_TOKEN
environment variable.
CONTENTFUL_ACCESS_TOKEN=e5e8d4c5c122cf28fc1af3ff77d28bef78a3952957f15067bbc29f2f0dde0b50
mkdir /tmp/contentful-data-cfexampleapi
cd /tmp/contentful-data-cfexampleapi
contentful-sync fetch cfexampleapi
The current working directory will be populated with the assets and entries from the specified space grouped by content type. Each asset and entry will occupy one file. Each entry will be saved in JSON.
The environment variable containing the Contentful Acces Token may be added to your shell profile.
Multiple Spaces
If you need to sync from multiple Contentful Spaces, then you'll need an access token for each space. Then add the Space ID as a suffix to the environment variable name. For example:
CONTENTFUL_ACCESS_TOKEN_cfexampleapi=e5e8d4c5c122cf28fc1af3ff77d28bef78a3952957f15067bbc29f2f0dde0b50
Run contentful-sync --help
for more details.
Usage: contentful-sync [options] [command]
Commands:
fetch <space> [destination] Fetch content from a space with the Contentful Sync API
Options:
-h, --help output usage information
-V, --version output the version number
-q, --quiet Progress is not reported to the standard error stream.
-v, --verbose Be verbose.
-a, --access-token <token> Contentful Access Token
-i, --initial Fetch everything instead of fetching only what has changed.
-r, --resolve-links <boolean> Resolve links to other entries and assets.
-t, --type <type> What to sync: all (default), Asset, Entry, Deletion, DeletedAsset, or DeletedEntry
-c, --content-type <content-type> Limit sync to entries of specified content type. Implies --type Entry
--host <api host> Sync from Content Preview API or Content Delivery API
Library Usage
const contentfulSync = require('contentful-sync-cli');
let options = {
"accessToken": process.env.CONTENTFUL_ACCESS_TOKEN,
"space": "cfexampleapi",
"destination": "/tmp/contentful-data-cfexampleapi"
};
contentfulSync.fetch(options)
.then(() => {
console.log("Saved data to %s", options.destination);
})
.catch((err) => {
console.log(err);
});
Contentful Client API Usage with local storage
Once the Contentful data is fetched, then it may be accessed with the contentful-local
library. This library mirrors the official client API with the following differences:
By default, you get all entries, instead of 100, with no upper limit.
const contentful = require("contentful-sync-cli/contentful-local");
const client = contentful.createClient({
space: "cfexampleapi",
localPath: "/tmp/contentful-data-cfexampleapi"
});
client.getEntries()
.then(function (entries) {
entries.items.forEach(function (entry) {
if(entry.fields.productName) {
console.log(entry.fields.productName)
}
});
});
Or, if you would like to limit the number of entries you get:
client.getEntries({
skip: 100,
limit: 200,
order: 'sys.createdAt'
})
.then(function (entries) {
console.log(entries.items.length)
});
Usage with a Proxy
To proxy HTTP(S) requests, then set the appropriate npm config variables.
npm config set https-proxy http://proxy.example.com:3128/
npm config set proxy http://proxy.example.com:3128/
If the npm config variables are not found, then these environment variables will be used.
HTTPS_PROXY=http://proxy.example.com:3128/
HTTP_PROXY=http://proxy.example.com:3128/
Options
{
"space": null,
"destination": null,
"accessToken": null,
"initial": null,
"resolveLinks": true,
"type": "all",
"content_type": null,
"host": null,
}
Todo
- test for deleted records
- a progress indicator
- implement more search parameters
- currently implemented:
content_type, order, reverse order, order with multiple parameters
- need to implement:
select, equality, inequality, array equality/inequality, array with multiple values, inclusion, exclusion, ranges, full-text search, full-text search on a field, location proximity search, locations in a bounding object, limit, skip, filtering assets by MIME type, search on references
Change Log
October 21, 2019 – v3.0.9
- updated all deps to latest
- resolved security issue with older version of eslint
October 21, 2019 – v3.0.8
- updated dependencies to resolve security issues
- use Object.prototype.hasOwnProperties to prevent possible misuse of obj.hasOwnProperties
February 19, 2019 – v3.0.7
- updated dependencies to resolve security issues
January 8, 2019 – v3.0.6
- added additional error handling for malformed JSON data
November 6, 2018 – v3.0.5
- updated deps
- updated tests to work with contentful 7.0
- added test case to test sync through HTTP proxy
- fixed bug where proxy settings did not get applied
June 19, 2018 – v3.0.4
- updated deps
- updated test to work with contentful 6.0
January 5, 2018 – v3.0.3
- fixed bugs with 'reference, many' field types
October 17, 2017 – v3.0.2
- fixed bug when sync'ing deleted assets/entries
October 16, 2017 – v3.0.1
- return undefined when entry cannot be found
June 5, 2017 – v3.0.0
- refactored getEntry to get entry by ID without glob
- required saving/deleting entries to a common folder in addition to the existing folders for each content type
- replace glob.sync with async glob when deleting entries
June 2, 2017 – v2.2.1
- switched to graceful-fs to prevent EMFILE errors
June 1, 2017 – v2.2.0
- check for access tokens in env var with space id
- fixed bug preventing logger from logging
- improved logging
June 1, 2017 – v2.1.4
- speed up including linked entries by assuming the field name matches the content type for references
- refactored getEntry() to use async globbing when the content_type is unknown
May 19, 2017 – v2.1.3
- fixed ordering of multiple attributes
May 19, 2017 – v2.1.2
- fixed bug when ordering sets that included undefined values
- fixed bug when including whitespace between order attributes in query
April 6, 2017 – v2.1.1
- fixed unhandled promise rejection warning when auth token is not passed
March 15, 2017 – v2.1.0
- added test cases for query ordering search parameter
- implemented support for query ordering
March 14, 2017 – v2.0.2
- fixed bug when calling unWrap with a collection
March 14, 2017 – v2.0.1
- fixed bug when running fetch sub-command without --host argument
- fixed minor bug when logging results of command
March 14, 2017 – v2.0.0
- fetches from Content Preview API by default
- unless NODE_ENV === production
- API host may now be specified in the
host
option for #fetch() - changed file format to include all locales
- added library that mirrors the official Contentful Client API
- pulls data from local storage
- uses default locale of current Space
- added
locale
option to request a specific locale - resolves one level of linked entries or assets
- supports
include
option for additional levels
- saves asset records to disk
- removes deleted asset records from disk
- added test for initial sync
- added test for continued sync
- added #unWrap() and #unWrapCollection() to return only fields
February 13, 2017 – v1.1.0
- added support to optionally proxy HTTP requests
January 19, 2017 – v1.0.3
- fixed bug when an initial sync is run with a specified content type
- it's an upstream bug so an error is thrown
January 19, 2017 – v1.0.2
- fixed bug when syncing a deleted entry
October 26, 2016 – v1.0.1
- fixed bug when saving updates to specified path
October 20, 2016 – v1.0.0
- call sync api
- save sync token
- load sync token and continue the sync on subsequent requests
- create readme
- turn index.js into a module
- create bin script and require index.js module
- create
fetch
sub-command - parse command line options and merge with environment variables
- save entries to disk
- delete deletedEntries from disk