@constructor-io/catalog-ingestor

Constructor.io Catalog Ingestor

Constructor.io is an e-commerce first product discovery service that optimizes results using artificial intelligence (including natural language processing, re-ranking to optimize for business metrics, and end user personalization).

This package is a Node.js library for easily ingesting product catalogs into Constructor.io using a strict type system. It is intended for use with partner connections - that is, integrating other systems such as e-commerce platforms or product information management platforms with Constructor.io

If you're looking for a package to consume our API, please use @constructor-io/constructorio-node instead. Alternatively, if you want a JavaScript client for client side (i.e. front end) integrations please use @constructor-io/constructorio-client-javascript.

⚠️ Deprecation warning

This package has been deprecated as of February 2024 and will no longer be maintained.

We recommend using our Node.js client instead.

Documentation

Full API documentation is available on Github Pages

1. Review the Requirements

Before you begin, note that this package is intended for use with partner connections. Some credentials are provided by partner connectors while calling this package, so ideally you should be working with a specific partner integration to use this package.

The following credentials are required to use this package:

• apiToken - Your Constructor.io API token.
• apiKey - Your Constructor.io API key.

If you need a more general solution, check out our Node.js client.

2. Install

This package can be installed via npm: npm i @constructor-io/catalog-ingestor. Once installed, simply import or require the package into your repository.

Important: this library should only be used in a server-side context.

3. Retrieve an API key and token

You can find this in your Constructor.io dashboard. Contact sales if you'd like to sign up, or support if you believe your company already has an account.

4. Implement the ingestor

Once imported, an instance of the ingestor can be created as follows:

import { CatalogIngestor } from "@constructor-io/catalog-ingestor";

const catalogIngestor = new CatalogIngestor({
  apiToken: "YOUR API TOKEN",
  apiKey: "YOUR API KEY",
  type: "INGESTION_TYPE"

  // Optional parameters
  notificationEmail: "[e-mail]",
  force: false,
});

Parameter	Required	Type	Description
apiToken	true	string	Your Constructor.io API Token. Please refer to the documentation on how to obtain it.
apiKey	true	string	Your Constructor.io API Key. Please refer to the documentation on how to obtain it.
type	true	CatalogIngestionType	The type of ingestion to be performed. You can find more details in th section bellow.
notificationEmail	false	string	An email to send notifications in case the ingestion fails.
force	false	boolean	If true, will force the ingestion to complete even if it invalidates a large amount of records. Defaults to false.
section	false	string	The index's section to ingest the catalog into. Defaults to "Products".
retainCsvFile	false	boolean	If true, will not delete the resulting CSV file after sending it to constructor - Useful for debugging

Check out our API docs for more information.

5. Ingest your catalog

Ingestion types

We support full, delta and patch ingestions.

This is handled by the type option passed through the constructor.

Full ingestions

Full ingestions replace the entire catalog with the data provided. This is useful for initial catalog ingestion, or when you want to completely replace your catalog with new data.

import { CatalogIngestionType } from "@constructor-io/catalog-ingestor";

const catalogIngestor = new CatalogIngestor({
  type: CatalogIngestionType.FULL,
  // ...other options
});

Delta ingestions

Delta ingestions update the catalog with the data provided. This adds new catalog data, or replaces existing data.

import { CatalogIngestionType } from "@constructor-io/catalog-ingestor";

const catalogIngestor = new CatalogIngestor({
  type: CatalogIngestionType.DELTA,
  // ...other options
});

Patch delta ingestions

Patch delta ingestions update the catalog with the data provided. This strictly updates already existing data. Different from the other two types, it support ingesting partial data.

import { CatalogIngestionType } from "@constructor-io/catalog-ingestor";

const catalogIngestor = new CatalogIngestor({
  type: CatalogIngestionType.PATCH_DELTA_FAIL,
  // ...other options
});

There are three different patch delta ingestion types, and each one handles missing catalog entities with specific rules:

A missing catalog entity is one that is referenced by the ingestion but is not found on the catalog.

Ingestion Type	If there are any missing entities
PATCH_DELTA_FAIL	The whole catalog ingestion will be aborted and it's moved into an error state.
PATCH_DELTA_CREATE	They will be created, and existing entities will have any additional fields appended. This type also needs all required fields present.
PATCH_DELTA_IGNORE	They will be ignored.

Fetching & transforming data

This package is completely agnostic on how to fetch and transform your data. The only requirement is that you must provide a function that returns a Promise that resolves to the ingestion payload.

It's recommended to fetch & transform your data inside this promise. This way, we're able to identify and report any issues that may have occurred during ingestion.

Finally, to ingest your catalog data you simply need to call the ingest function and pass this promise:

import {
  CatalogIngestionPayload,
  CatalogIngestionType,
} from "@constructor-io/catalog-ingestor";

const catalogIngestor = new CatalogIngestor({
  type: CatalogIngestionType.FULL,
  // ...other options
});

async function fetchData(): Promise<ExternalData> {
  // TODO: Implement your logic to fetch data here.

  return {};
}

function transformData(data: ExternalData): CatalogIngestionPayload {
  // TODO: Implement your logic to transform data here.
  // As an example, we're just returning static data.

  return {
    data: {
      groups: [
        {
          parent_id: null,
          name: "Shoes",
          id: "shoes",
        },
      ],
      items: [
        {
          id: "nike-shoes-brown",
          item_name: "Nike Shoes Brown",
          image_url: "https://images.nike.com/shoes-brown.jpg",
          url: "https://www.nike.com/shoes-brown",
          description: "Best shoes",
          group_ids: ["shoes"],
          active: true,
          metadata: [],
          keywords: [],
          facets: [
            {
              key: "Color",
              value: "Brown",
            },
            {
              key: "Size",
              value: ["M", "L", "XL"],
            },
          ],
        },
      ],
      variations: [],
    },
  };
}


await catalogIngestor.ingest(async () => {
  const data = await fetchData();
  return transformData(data);
});

Development

Here's some useful npm scripts:

npm run lint            # run lint on source code and tests
npm run test            # run tests
npm run test:cov        # run tests with coverage report
npm run generate-docs   # output documentation to `./docs` directory

Running locally

First, install dependencies:

npm i

Then, set up your local environment and fill .env with your credentials:

cp .env.example .env

Don't forget to fill in some test data on src/dev.ts:

 data: {
  // insert your test data here
  groups: [],
  items: [],
  variations: [],
},

All done! You can now run the ingestor locally:

npm run dev

# Or you can test it using streams
npm run dev:stream