@jupiterone/integration-sdk-core - npm Package Versions

5.6.0 - 2021-01-14

Added

• Whitespace trimming of integration config values.
• Support for skipping writing graph object files when running CLI commands

Changed

• Skip logging warn message when a CredentialsError is received in graph object uploads.

5.5.0 - 2021-01-02

Added

• Support for specifying the upload batch size via the uploadBatchSize in createPersisterApiStepGraphObjectDataUploader.

5.4.0 - 2020-12-31

Added

• JobState.hasKey() as an efficient means of determining whether an entity or relationship has been added. This allows integrations to avoid doing any of their own key tracking.

Fixed

• A createIntegrationEntity bug in transferring source properties into an entity caused a memory leak.

5.3.0 - 2020-12-27

Added

• Support for omitting specific graph objects from file storage. Some entities and relationships do not need to be stored on the file system at all. We only need to store graph objects on the file system if we later intend to fetch the data from the job state or iterate over the _type.

  Usage in an integration step:

  {
    id: 'my-step',
    name: 'My step',
    entities: [
      {
        resourceName: 'The Record',
        _type: 'my_record',
        _class: ['Record'],
        indexMetadata: {
          // This _type will not be written to the file system
          enabled: false,
        },
      },
    ],
    relationships: [],
    async exeutionHandler() {
      ...
    }
  }
  

  See PR #404

5.2.1 - 2020-12-23

Fixed

• Fixed issue where the SDK could upload empty entity and relationship graph object files.

5.2.0 - 2020-12-18

Changed

• OPTIMIZATION: Buffer entities and relationships in memory and allow for fast lookups. This change allows us to skip flushing to disk anytime there is a call to findEntity, iterateEntities or iterateRelationships.

  See PR #395

• OPTIMIZATION: Allow FileSystemGraphObjectStore to specify graphObjectBufferThreshold, which defines the maximum number of graph objects that the graph object store can buffer into memory before flushing to disk. Machines with more memory should consider increasing this value as the default is 500.

  See PR #395

• OPTIMIZATION: Continuously upload integration data during collection phase. Previously, our integrations had two primary phases. The first phase was the collection phase, and the second phase was the upload phase. The integration SDK now mixes these two phases and the integrations will upload the collected data to JupiterOne during the integration execution step that it has been collected in.

  See PR #396

• OPTIMIZATION: Reduce the size of the graph object files that are stored on disk by default. Previously, all graph object files written to disk while running the integration locally and running the integration in the managed JupiterOne runtime, were created with whitespace. The file whitespace is now only created when running the integration locally via the CLI commands.

  See PR #399

Removed

• Remove BucketMap as it's no longer used in the FileSystemGraphObjectStore. BucketMap was technically exposed externally, but should not have been used externally.

5.1.0 - 2020-12-08

Added

• Added registerEventHandlers and unregisterEventHandlers exports from the runtime package. These functions will register a common set of event handlers that handle out-of-band errors that can be thrown in integration steps, such as unhandledRejection and multipleResolves. These handlers should be added to any deployment projects that invoke executeIntegrationInstance. Developers should instrument these event handlers as early as possible in the node process.
• Added registerIntegrationLoggerEventHandlers and unregisterIntegrationLoggerEventHandlers as convenience functions, since JupiterOne infrastructure will handle these event emitter failures by calling logger.error().
• Added registerEventHandlers to executeIntegrationLocally, so that events are caught when running yarn j1-integration collect.
• Made registerIntegrationEventHandlers call the integration logger onFailure function.

Changed

• Increase concurrency on integration data uploads
• Implement retry logic around integration data uploads

5.0.0 - 2020-11-24

Added

• ExecutionContext.executionHistory is always provided to integrations
• ExecutionHistory.current: Execution to provide information about the current execution.

Changed

• ExecutionHistory properties have been renamed as part of adding current: Execution, to avoid duplication in the naming: lastExecution -> previous, lastSuccessfulExecution -> lastSuccessful.
• BREAKING: executeIntegrationInstance now requires an ExecutionHistory argument containing the current: Execution.
• BREAKING: executeWithContext(context: ExecutionContext, ...) now requires that ExecutionContext.executionHistory is provided.

4.3.0 - 2020-11-24

Added

• Log whether compression is enabled or not at beginning of execution

4.2.0 - 2020-11-21

Added

• Specifying the INTEGRATION_FILE_COMPRESSION_ENABLED environment variable will now compress local integration graph object files with Brotli compression on writes and decompress on reads.

Changed

• Stop handling IntegrationDuplicateKeyError as fatal. Previously, this type of error would terminate the integration completely and no data would be ingested. Now, the step that raises this error will not complete, but all other steps will complete and partial datasets will be processed.

• Publish integration disk size event on interval